Hi all, How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often. With dmesg I find this kind of messages several times a day: [Sun Mar 18 16:10:10 2018] addbiblio.pl invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [Sun Mar 18 16:10:10 2018] addbiblio.pl cpuset=/ mems_allowed=0 And then MySQL crashes and restarts: Sun Mar 18 16:10:10 2018] Out of memory: Kill process 6539 (mysqld) score 222 or sacrifice child [Sun Mar 18 16:10:10 2018] Killed process 6539 (mysqld) total-vm:1538124kB, anon-rss:224948kB, file-rss:0kB
From /var/log/syslog: Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Unit entered failed state. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Failed with result 'signal'. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Mar 18 16:10:11 biblioteko systemd[1]: Stopped MySQL Community Server. Mar 18 16:10:11 biblioteko systemd[1]: Starting MySQL Community Server...
Just yesterday the killed process was rebuild_zebra.p, which didn't restart, and Koha was retrieving no results for searches. The strange thing is that Digital Ocean's own monitoring tool never reports such memory spikes: usage is usually high but never more than 80%. Maybe the spikes are too short? What would you recommend for a more fine-grained monitoring of memory? Right now my memory is this: root@biblioteko:~# free total used free shared buff/cache available Mem: 1016056 555304 266328 14936 194424 292164 Swap: 0 0 0 On file /etc/mysql/mysql.conf.d/mysqld.cnf the value of "innodb_buffer_pool_size" is 385M, if that's of any help. Do I really need more physical memory on my server?
Hi Rubeno, We have less biblio records than you (about 5000) and initially we hosted Koha on a VPS with just 1Gb of memory, also from Digital Ocean, as it happens. With six staff users logged in (during training sessions) were getting a variety of software errors, though I don't think mysqld ever crashed. Memory usage was pretty high, and upgrading to a droplet with 2Gb of memory gave a big improvement. It's certainly worth a try. Running "top" may give you a more fine-grained picture of memory usage, particularly if you sort on the %MEM column. Hope this helps, Best Regards, Chris Brown On Mon, Mar 19, 2018 at 6:12 AM, Rubeno Fernández <enseikou@gmail.com> wrote:
Hi all,
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
With dmesg I find this kind of messages several times a day:
[Sun Mar 18 16:10:10 2018] addbiblio.pl invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [Sun Mar 18 16:10:10 2018] addbiblio.pl cpuset=/ mems_allowed=0 And then MySQL crashes and restarts: Sun Mar 18 16:10:10 2018] Out of memory: Kill process 6539 (mysqld) score 222 or sacrifice child [Sun Mar 18 16:10:10 2018] Killed process 6539 (mysqld) total-vm:1538124kB, anon-rss:224948kB, file-rss:0kB
From /var/log/syslog: Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Unit entered failed state. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Failed with result 'signal'. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Mar 18 16:10:11 biblioteko systemd[1]: Stopped MySQL Community Server. Mar 18 16:10:11 biblioteko systemd[1]: Starting MySQL Community Server...
Just yesterday the killed process was rebuild_zebra.p, which didn't restart, and Koha was retrieving no results for searches.
The strange thing is that Digital Ocean's own monitoring tool never reports such memory spikes: usage is usually high but never more than 80%. Maybe the spikes are too short? What would you recommend for a more fine-grained monitoring of memory?
Right now my memory is this: root@biblioteko:~# free total used free shared buff/cache available Mem: 1016056 555304 266328 14936 194424 292164 Swap: 0 0 0
On file /etc/mysql/mysql.conf.d/mysqld.cnf the value of "innodb_buffer_pool_size" is 385M, if that's of any help.
Do I really need more physical memory on my server? _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
I had a similar experience. I was running a test installation on a Digital Ocean server with 512MB RAM where I was the *only* user. The production site has much more and is a lot happier. Fred King Medical Librarian, MedStar Washington Hospital Center fred.king@medstar.net 202-877-6670 ORCID 0000-0001-5266-0279 MedStar Authors Catalog: http://medstarauthors.org Sometimes I wake up from a dream and then go back to it in disguise so I can find out what they’re saying about me behind my back. --Bill in Portland -----Original Message----- From: Koha [mailto:koha-bounces@lists.katipo.co.nz] On Behalf Of Chris Brown Sent: Monday, March 19, 2018 3:28 AM To: Rubeno Fernández Cc: koha Subject: Re: [Koha] Out of memory Hi Rubeno, We have less biblio records than you (about 5000) and initially we hosted Koha on a VPS with just 1Gb of memory, also from Digital Ocean, as it happens. With six staff users logged in (during training sessions) were getting a variety of software errors, though I don't think mysqld ever crashed. Memory usage was pretty high, and upgrading to a droplet with 2Gb of memory gave a big improvement. It's certainly worth a try. Running "top" may give you a more fine-grained picture of memory usage, particularly if you sort on the %MEM column. Hope this helps, Best Regards, Chris Brown On Mon, Mar 19, 2018 at 6:12 AM, Rubeno Fernández <enseikou@gmail.com> wrote:
Hi all,
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
With dmesg I find this kind of messages several times a day:
[Sun Mar 18 16:10:10 2018] https://urldefense.proofpoint.com/v2/url?u=http-3A__addbiblio.pl&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVxj8-bJ04M&r=vKh6XwOmjyC51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5aiwJyQiQqtxn5XZudWCc-7c9qWutA44&s=Pr97cNpNijeEtvP2zz_V-3RB5qcZ1D5NoouUxkbV6CA&e= invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [Sun Mar 18 16:10:10 2018] https://urldefense.proofpoint.com/v2/url?u=http-3A__addbiblio.pl&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVxj8-bJ04M&r=vKh6XwOmjyC51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5aiwJyQiQqtxn5XZudWCc-7c9qWutA44&s=Pr97cNpNijeEtvP2zz_V-3RB5qcZ1D5NoouUxkbV6CA&e= cpuset=/ mems_allowed=0 And then MySQL crashes and restarts: Sun Mar 18 16:10:10 2018] Out of memory: Kill process 6539 (mysqld) score 222 or sacrifice child [Sun Mar 18 16:10:10 2018] Killed process 6539 (mysqld) total-vm:1538124kB, anon-rss:224948kB, file-rss:0kB
From /var/log/syslog: Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Unit entered failed state. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Failed with result 'signal'. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Mar 18 16:10:11 biblioteko systemd[1]: Stopped MySQL Community Server. Mar 18 16:10:11 biblioteko systemd[1]: Starting MySQL Community Server...
Just yesterday the killed process was rebuild_zebra.p, which didn't restart, and Koha was retrieving no results for searches.
The strange thing is that Digital Ocean's own monitoring tool never reports such memory spikes: usage is usually high but never more than 80%. Maybe the spikes are too short? What would you recommend for a more fine-grained monitoring of memory?
Right now my memory is this: root@biblioteko:~# free total used free shared buff/cache available Mem: 1016056 555304 266328 14936 194424 292164 Swap: 0 0 0
On file /etc/mysql/mysql.conf.d/mysqld.cnf the value of "innodb_buffer_pool_size" is 385M, if that's of any help.
Do I really need more physical memory on my server? _______________________________________________ Koha mailing list https://urldefense.proofpoint.com/v2/url?u=http-3A__koha-2Dcommunity.o rg&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVxj8-bJ04M&r=vKh6XwOmjy C51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5aiwJyQiQqtxn5XZudWCc-7 c9qWutA44&s=tnQspFRCDDxEBJblf-_FcsZhDDeWhlcF6WhHYxtZDgo&e= Koha@lists.katipo.co.nz https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.katipo.co.n z_mailman_listinfo_koha&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVx j8-bJ04M&r=vKh6XwOmjyC51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5a iwJyQiQqtxn5XZudWCc-7c9qWutA44&s=x97u-EzVBbonN7yFavnLDjNXL8LhF7E3kf82U Nc9pUg&e=
_______________________________________________ Koha mailing list https://urldefense.proofpoint.com/v2/url?u=http-3A__koha-2Dcommunity.org&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVxj8-bJ04M&r=vKh6XwOmjyC51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5aiwJyQiQqtxn5XZudWCc-7c9qWutA44&s=tnQspFRCDDxEBJblf-_FcsZhDDeWhlcF6WhHYxtZDgo&e= Koha@lists.katipo.co.nz https://urldefense.proofpoint.com/v2/url?u=https-3A__lists.katipo.co.nz_mailman_listinfo_koha&d=DwIGaQ&c=RvBXVp2Kc-itN3g6r3sN0QK_zL4whPpndVxj8-bJ04M&r=vKh6XwOmjyC51IkP1OfsdjQZoWT2vpi6VZl8El8EPRI&m=_CAkvYy8a_k5aiwJyQiQqtxn5XZudWCc-7c9qWutA44&s=x97u-EzVBbonN7yFavnLDjNXL8LhF7E3kf82UNc9pUg&e=
Excerpts from Rubeno Fernández's message of 2018-03-19 07:12:29 +0100:
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
We're running Koha 17.11.02 on Linode (Debian 8) with 2 GB of RAM, and our library has about 25000 item records. So far we haven't had any out of memory problems.
How much swap space do you have allocated? There are probably a lot of libraries loaded where Koha is only using a function or two, and I believe that MySQL is very aggressive about loading data into memory, not all of which is used. Conventional wisdom is that swap will slow down the system because you have to page things back into memory... but conventional wisdom doesn't take into account the fact that there are a lot of things that get loaded into memory but not used. Swap is good, folks! --Barton On Mon, Mar 19, 2018 at 10:04 AM, Mark Alexander <marka@pobox.com> wrote:
Excerpts from Rubeno Fernández's message of 2018-03-19 07:12:29 +0100:
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
We're running Koha 17.11.02 on Linode (Debian 8) with 2 GB of RAM, and our library has about 25000 item records. So far we haven't had any out of memory problems. _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Barton et al, The output from the "free" command in Rubeno's original post shows that he has no swap space. Chris Brown On Mon, Mar 19, 2018 at 2:55 PM, Barton Chittenden < barton@bywatersolutions.com> wrote:
How much swap space do you have allocated?
There are probably a lot of libraries loaded where Koha is only using a function or two, and I believe that MySQL is very aggressive about loading data into memory, not all of which is used.
Conventional wisdom is that swap will slow down the system because you have to page things back into memory... but conventional wisdom doesn't take into account the fact that there are a lot of things that get loaded into memory but not used. Swap is good, folks!
--Barton
On Mon, Mar 19, 2018 at 10:04 AM, Mark Alexander <marka@pobox.com> wrote:
Excerpts from Rubeno Fernández's message of 2018-03-19 07:12:29 +0100:
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
We're running Koha 17.11.02 on Linode (Debian 8) with 2 GB of RAM, and our library has about 25000 item records. So far we haven't had any out of memory problems. _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Heh. You're so right. https://www.youtube.com/watch?v=RVnrpBROMkQ&feature=youtu.be&t=69 On Mon, Mar 19, 2018 at 11:15 AM, Chris Brown <chris@stayawake.co.uk> wrote:
Barton et al,
The output from the "free" command in Rubeno's original post shows that he has no swap space.
Chris Brown
On Mon, Mar 19, 2018 at 2:55 PM, Barton Chittenden < barton@bywatersolutions.com> wrote:
How much swap space do you have allocated?
There are probably a lot of libraries loaded where Koha is only using a function or two, and I believe that MySQL is very aggressive about loading data into memory, not all of which is used.
Conventional wisdom is that swap will slow down the system because you have to page things back into memory... but conventional wisdom doesn't take into account the fact that there are a lot of things that get loaded into memory but not used. Swap is good, folks!
--Barton
On Mon, Mar 19, 2018 at 10:04 AM, Mark Alexander <marka@pobox.com> wrote:
Excerpts from Rubeno Fernández's message of 2018-03-19 07:12:29 +0100:
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often.
We're running Koha 17.11.02 on Linode (Debian 8) with 2 GB of RAM, and our library has about 25000 item records. So far we haven't had any out of memory problems. _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
It's true, by default Digital Ocean's droplets are created without any swap space. Actually, they recommend not to enable it on their solid state disks: "Although swap is generally recommended for systems utilizing traditional spinning hard drives, using swap with SSDs can cause issues with hardware degradation over time. Due to this consideration, we do not recommend enabling swap on DigitalOcean or any other provider that utilizes SSD storage. Doing so can impact the reliability of the underlying hardware for you and your neighbors. If you need to improve the performance of your server, we recommend upgrading your Droplet. This will lead to better results in general and will decrease the likelihood of contributing to hardware issues that can affect your service." (from https://www.digitalocean.com/community/tutorials/how-to-configure-virtual-me...) I guess it's my choice between being cheap or being a good customer. El dilluns, 19 de març de 2018, a les 16:15:10 CET, Chris Brown va escriure:
Barton et al,
The output from the "free" command in Rubeno's original post shows that he has no swap space.
Chris Brown
On Mon, Mar 19, 2018 at 2:55 PM, Barton Chittenden <
barton@bywatersolutions.com> wrote:
How much swap space do you have allocated?
There are probably a lot of libraries loaded where Koha is only using a function or two, and I believe that MySQL is very aggressive about loading data into memory, not all of which is used.
Conventional wisdom is that swap will slow down the system because you have to page things back into memory... but conventional wisdom doesn't take into account the fact that there are a lot of things that get loaded into memory but not used. Swap is good, folks!
--Barton
On Mon, Mar 19, 2018 at 10:04 AM, Mark Alexander <marka@pobox.com> wrote:
Excerpts from Rubeno Fernández's message of 2018-03-19 07:12:29 +0100:
How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on
Digital
Ocean
with 1GB memory and very low usage, but the server is crashing very
often.
We're running Koha 17.11.02 on Linode (Debian 8) with 2 GB of RAM, and our library has about 25000 item records. So far we haven't had any out of memory problems. _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
Greetings, If you are not running plack, then you are probably experiencing 1MB per query leak with a default apache configuration with 100MB per query used. Digital Ocean has a nice 3GB droplet for $15/month. As someone else also mentioned, MySQL loves to cache which leads to hogging memory. If you have implemented the auto-increment fix (https://wiki.koha-community.org/wiki/DBMS_auto_increment_fix), then a weekly reboot isn't a bad idea. Also, use MariaDB over MySQL. :) We were experiencing problems with our Koha, then I discovered my colleague had set up pingdom to check on the status of the OPAC every minute from multiple locations (the equivalent of really high use). We changed it to 15 minutes from one location, implemented the auto increment fix, cleaned up the data manually, rebooted on Sundays, and upgraded to 3GB. We have not had a problem since with our over 120K items. However, we have a relatively low circulation, and small to moderate use compared to the number of items we have. As for how much memory you should have, take a look at /var/spool/koha/{instance name} (I think) and see the size of the two files for any given day. Take that size and multiply it by at least 3. This will leave you room to run a full reindex, have MySQL cache things, and still have space for a several page loads per minute. If you have much higher use and/or circulation, you'll need to account for that. Imagine 100 students simultaneously accessing OPAC (okay, everyone press enter on your search in 3,2,1... now), you probably want to have 10GB (100MB*100/1024 ~= 9.75GB) in addition to amount you calculated before. Imagine 5 librarians cataloguing simultaneously (okay, everyone click New Record in 3,2,1... now), you probably want another 1GB (100MB*5/1024 ~= 488MB always ceiling round up) in addition to amount you calculated before. Currently, I do not recommend less than 2GB. Though, like Fred, I have had a test system hobble with 512MB. That is not recommended at all. Ideal Size = 2 GB + Backup Size in GB * 3 + (Number of Simultaneous Users * 100 / 1024) GB. Make sure to ceiling up to nearest GB size available. That's my attempt at this question. :) GPML, Mark Tompsett -----Original Message----- From: RubenoFernández Sent: Monday, March 19, 2018 2:12 AM To: koha@lists.katipo.co.nz Subject: [Koha] Out of memory Hi all, How much memory do you need to run Koha? I run a database with about 22000 records on a virtual server on Digital Ocean with 1GB memory and very low usage, but the server is crashing very often. With dmesg I find this kind of messages several times a day: [Sun Mar 18 16:10:10 2018] addbiblio.pl invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0 [Sun Mar 18 16:10:10 2018] addbiblio.pl cpuset=/ mems_allowed=0 And then MySQL crashes and restarts: Sun Mar 18 16:10:10 2018] Out of memory: Kill process 6539 (mysqld) score 222 or sacrifice child [Sun Mar 18 16:10:10 2018] Killed process 6539 (mysqld) total-vm:1538124kB, anon-rss:224948kB, file-rss:0kB From /var/log/syslog: Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Main process exited, code=killed, status=9/KILL Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Unit entered failed state. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Failed with result 'signal'. Mar 18 16:10:11 biblioteko systemd[1]: mysql.service: Service hold-off time over, scheduling restart. Mar 18 16:10:11 biblioteko systemd[1]: Stopped MySQL Community Server. Mar 18 16:10:11 biblioteko systemd[1]: Starting MySQL Community Server... Just yesterday the killed process was rebuild_zebra.p, which didn't restart, and Koha was retrieving no results for searches. The strange thing is that Digital Ocean's own monitoring tool never reports such memory spikes: usage is usually high but never more than 80%. Maybe the spikes are too short? What would you recommend for a more fine-grained monitoring of memory? Right now my memory is this: root@biblioteko:~# free total used free shared buff/cache available Mem: 1016056 555304 266328 14936 194424 292164 Swap: 0 0 0 On file /etc/mysql/mysql.conf.d/mysqld.cnf the value of "innodb_buffer_pool_size" is 385M, if that's of any help. Do I really need more physical memory on my server? _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
participants (6)
-
Barton Chittenden -
Chris Brown -
King, Fred -
Mark Alexander -
Mark Tompsett -
Rubeno Fernández