[Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl

[Mike Lake]

Hi

Katrin suggested:
> it might be that you are hit by a bad crawler/bot

Thanks Katrin. That *may* have been the cause. The system is working OK 
at present. I did a complete shutdown and reboot.

I did notice in the opac-error.log, which is now over 10 MB, a recurring
query (see below) that was being made every 30 seconds. Exact same 
query, clearly automated. That seems to have ended now.

cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available

I was also getting these errors which were filling up the logs:

[Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client 
190.92.203.86:51260]
AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha

There is no such perl script
$ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
so I just created one to return "hello".

Now our Koha instance is back up again and our VM is coping with the 
load.  https://opac.caves.org.au

Thanks for the reply.
I'll make another separate post on another current opac-error.log error 
line, if it still persists, after I upgrade from 23.11.05

Mike
ASF Sys Admin

On 2024-07-13 7:34 pm, Katrin Fischer wrote:
> Hi Mike,
> 
> it might be that you are hit by a bad crawler/bot and need to block
> access for them in your firewall. There are some that ignore the
> robots.txt and they can bring down a Koha server.
> 
> I you look at the Apache access logs you might see that all those
> requests come from the same IP address.
> 
> Hope this helps,
> 
> Katrin
> 
> On 10.07.24 13:02, Mike Lake wrote:
>> Hi all
>> 
>> I'm having serious problems with my Koha instance. It serves the OPAC
>> for the Australian Speleological Federation.  We are currently on Koha
>> 23.11 on a Debian 10.13. The system has been running fine for ages.
>> 
>> I was getting errors from the OOM killer:
>> 
>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
>> file-rss:0kB, shmem-rss:0kB
>> opac-search.pl invoked oom-killer:
>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
>> oom_score_adj=0
>> opac-search.pl cpuset=/ mems_allowed=0
>> 
>> So I shutdown Koha (took a while as I was out of memory)
>> systemctl stop koha-common.service
>> 
>> Rebooted the machine and when i bought Koha up:
>> systemctl start koha-common.service
>> Now I'm still getting 96 processes & errors taking all CPU and memory:
>> 
>> 3620  R  /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
>> 3622  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3624  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3625  D  /usr/bin/perl 
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3627  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3629  D  /usr/bin/perl 
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3630  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3633  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 
>> Actually its 96 x opac-search.pl + 57 x 500.pl
>> 
>> A reboot does not help. Every time I start Koha those processes appear
>> and take all cores and memory.
>> 
>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
>> database server."
>> Attempts to start it: systemctl start mariadb.service
>> give that error probably because I'm out of memory does to the 100
>> perl processes running.
>> 
>> A "systemctl stop koha-common.service" does not stop or end those
>> opac-search.pl or 500.pl processes.
>> 
>> The /var/log/koha/opac/opac-error.log says:
>> 
>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
>> output before headers: 500.pl
>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
>> output before headers: opac-search.pl
>> 
>> Something is borked :-(   Help most welcome.
>> 
> _______________________________________________
> 
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha

-- 
Mike