Out of memory when Koha starts due to opac-search.pl and 500.pl
Hi all I'm having serious problems with my Koha instance. It serves the OPAC for the Australian Speleological Federation. We are currently on Koha 23.11 on a Debian 10.13. The system has been running fine for ages. I was getting errors from the OOM killer: oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB opac-search.pl invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 opac-search.pl cpuset=/ mems_allowed=0 So I shutdown Koha (took a while as I was out of memory) systemctl stop koha-common.service Rebooted the machine and when i bought Koha up: systemctl start koha-common.service Now I'm still getting 96 processes & errors taking all CPU and memory: 3620 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl 3622 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3624 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3625 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3627 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3629 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3630 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3633 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl Actually its 96 x opac-search.pl + 57 x 500.pl A reboot does not help. Every time I start Koha those processes appear and take all cores and memory. I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 database server." Attempts to start it: systemctl start mariadb.service give that error probably because I'm out of memory does to the 100 perl processes running. A "systemctl stop koha-common.service" does not stop or end those opac-search.pl or 500.pl processes. The /var/log/koha/opac/opac-error.log says: [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script output before headers: 500.pl [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script output before headers: opac-search.pl Something is borked :-( Help most welcome. -- Mike Sys Admin Australian Speleological Federation
Hi Mike, it might be that you are hit by a bad crawler/bot and need to block access for them in your firewall. There are some that ignore the robots.txt and they can bring down a Koha server. I you look at the Apache access logs you might see that all those requests come from the same IP address. Hope this helps, Katrin On 10.07.24 13:02, Mike Lake wrote:
Hi all
I'm having serious problems with my Koha instance. It serves the OPAC for the Australian Speleological Federation. We are currently on Koha 23.11 on a Debian 10.13. The system has been running fine for ages.
I was getting errors from the OOM killer:
oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB opac-search.pl invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 opac-search.pl cpuset=/ mems_allowed=0
So I shutdown Koha (took a while as I was out of memory) systemctl stop koha-common.service
Rebooted the machine and when i bought Koha up: systemctl start koha-common.service Now I'm still getting 96 processes & errors taking all CPU and memory:
3620 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl 3622 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3624 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3625 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3627 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3629 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3630 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3633 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
Actually its 96 x opac-search.pl + 57 x 500.pl
A reboot does not help. Every time I start Koha those processes appear and take all cores and memory.
I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 database server." Attempts to start it: systemctl start mariadb.service give that error probably because I'm out of memory does to the 100 perl processes running.
A "systemctl stop koha-common.service" does not stop or end those opac-search.pl or 500.pl processes.
The /var/log/koha/opac/opac-error.log says:
[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script output before headers: 500.pl [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script output before headers: opac-search.pl
Something is borked :-( Help most welcome.
Hi Katrin suggested:
it might be that you are hit by a bad crawler/bot
Thanks Katrin. That *may* have been the cause. The system is working OK at present. I did a complete shutdown and reboot. I did notice in the opac-error.log, which is now over 10 MB, a recurring query (see below) that was being made every 30 seconds. Exact same query, clearly automated. That seems to have ended now. cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available I was also getting these errors which were filling up the logs: [Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client 190.92.203.86:51260] AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha There is no such perl script $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/' so I just created one to return "hello". Now our Koha instance is back up again and our VM is coping with the load. https://opac.caves.org.au Thanks for the reply. I'll make another separate post on another current opac-error.log error line, if it still persists, after I upgrade from 23.11.05 Mike ASF Sys Admin On 2024-07-13 7:34 pm, Katrin Fischer wrote:
Hi Mike,
it might be that you are hit by a bad crawler/bot and need to block access for them in your firewall. There are some that ignore the robots.txt and they can bring down a Koha server.
I you look at the Apache access logs you might see that all those requests come from the same IP address.
Hope this helps,
Katrin
On 10.07.24 13:02, Mike Lake wrote:
Hi all
I'm having serious problems with my Koha instance. It serves the OPAC for the Australian Speleological Federation. We are currently on Koha 23.11 on a Debian 10.13. The system has been running fine for ages.
I was getting errors from the OOM killer:
oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB opac-search.pl invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 opac-search.pl cpuset=/ mems_allowed=0
So I shutdown Koha (took a while as I was out of memory) systemctl stop koha-common.service
Rebooted the machine and when i bought Koha up: systemctl start koha-common.service Now I'm still getting 96 processes & errors taking all CPU and memory:
3620 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl 3622 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3624 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3625 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3627 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3629 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3630 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3633 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
Actually its 96 x opac-search.pl + 57 x 500.pl
A reboot does not help. Every time I start Koha those processes appear and take all cores and memory.
I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 database server." Attempts to start it: systemctl start mariadb.service give that error probably because I'm out of memory does to the 100 perl processes running.
A "systemctl stop koha-common.service" does not stop or end those opac-search.pl or 500.pl processes.
The /var/log/koha/opac/opac-error.log says:
[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script output before headers: 500.pl [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script output before headers: opac-search.pl
Something is borked :-( Help most welcome.
_______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
-- Mike
participants (2)
-
Katrin Fischer -
Mike Lake