Hi Katrin suggested:

it might be that you are hit by a bad crawler/bot

Thanks Katrin. That *may* have been the cause. The system is working OK at present. I did a complete shutdown and reboot. I did notice in the opac-error.log, which is now over 10 MB, a recurring query (see below) that was being made every 30 seconds. Exact same query, clearly automated. That seems to have ended now. cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available I was also getting these errors which were filling up the logs: [Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client 190.92.203.86:51260] AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha There is no such perl script $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/' so I just created one to return "hello". Now our Koha instance is back up again and our VM is coping with the load. https://opac.caves.org.au Thanks for the reply. I'll make another separate post on another current opac-error.log error line, if it still persists, after I upgrade from 23.11.05 Mike ASF Sys Admin On 2024-07-13 7:34 pm, Katrin Fischer wrote:

Hi Mike,

it might be that you are hit by a bad crawler/bot and need to block access for them in your firewall. There are some that ignore the robots.txt and they can bring down a Koha server.

I you look at the Apache access logs you might see that all those requests come from the same IP address.

Hope this helps,

Katrin

On 10.07.24 13:02, Mike Lake wrote:

...

Hi all

I'm having serious problems with my Koha instance. It serves the OPAC for the Australian Speleological Federation.  We are currently on Koha 23.11 on a Debian 10.13. The system has been running fine for ages.

I was getting errors from the OOM killer:

oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB opac-search.pl invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 opac-search.pl cpuset=/ mems_allowed=0

So I shutdown Koha (took a while as I was out of memory) systemctl stop koha-common.service

Rebooted the machine and when i bought Koha up: systemctl start koha-common.service Now I'm still getting 96 processes & errors taking all CPU and memory:

3620  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl 3622  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3624  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3625  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3627  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3629  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3630  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3633  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl

Actually its 96 x opac-search.pl + 57 x 500.pl

A reboot does not help. Every time I start Koha those processes appear and take all cores and memory.

I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 database server." Attempts to start it: systemctl start mariadb.service give that error probably because I'm out of memory does to the 100 perl processes running.

A "systemctl stop koha-common.service" does not stop or end those opac-search.pl or 500.pl processes.

The /var/log/koha/opac/opac-error.log says:

[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script output before headers: 500.pl [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script output before headers: opac-search.pl

Something is borked :-(   Help most welcome.

_______________________________________________

Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha

-- Mike