[Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl

Mon Jul 15 11:49:03 NZST 2024

Hi Mike,

It certainly sounds like a crawler/bot getting stuck in a loop. In your log there, I see the client IP address 190.92.203.86, which belongs to Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei Cloud Singapore hitting Australian Koha sites over the last 6 months or so. 

That 'AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha' error is interesting. If you look at /etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the crawler sent any HTTP requests using your IP address and not the hostname, they'd be caught by that directive instead of your name-based virtual host. Could be some other explanations for why the virtual host wasn't used, but overall that would explain that message.

Anyway, it's not necessarily a Koha-specific issue. If you're not already using it, I'd suggest you look at installing and setting up something like fail2ban. That said, I have noticed the bots out of Huawei Cloud Singapore tend to cycle through a lot of different IP addresses, which does make things tricky. Sometimes, it'll just use one IP address that is easy to detect and block, but sometimes it might just do 1-2 hits per IP address (from a variety of different IP ranges). 

Let me know if you'd like to chat more about it.

David Cook
Senior Software Engineer
Prosentient Systems
Suite 7.03
6a Glen St
Milsons Point NSW 2061
Australia

Office: 02 9212 0899
Online: 02 8005 0595

-----Original Message-----

Date: Sat, 13 Jul 2024 21:10:36 +1000
From: Mike Lake <mikel at speleonics.com.au>
To: koha at lists.katipo.co.nz
Subject: Re: [Koha] Out of memory when Koha starts due to
	opac-search.pl and 500.pl
Message-ID: <f034d85a454901421773c0f4df4a045f at speleonics.com.au>
Content-Type: text/plain; charset=UTF-8; format=flowed

Hi

Katrin suggested:
> it might be that you are hit by a bad crawler/bot

Thanks Katrin. That *may* have been the cause. The system is working OK 
at present. I did a complete shutdown and reboot.

I did notice in the opac-error.log, which is now over 10 MB, a recurring
query (see below) that was being made every 30 seconds. Exact same 
query, clearly automated. That seems to have ended now.

cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available

I was also getting these errors which were filling up the logs:

[Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client 
190.92.203.86:51260]
AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha

There is no such perl script
$ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
so I just created one to return "hello".

Now our Koha instance is back up again and our VM is coping with the 
load.  https://opac.caves.org.au

Thanks for the reply.
I'll make another separate post on another current opac-error.log error 
line, if it still persists, after I upgrade from 23.11.05

Mike
ASF Sys Admin

On 2024-07-13 7:34 pm, Katrin Fischer wrote:
> Hi Mike,
> 
> it might be that you are hit by a bad crawler/bot and need to block
> access for them in your firewall. There are some that ignore the
> robots.txt and they can bring down a Koha server.
> 
> I you look at the Apache access logs you might see that all those
> requests come from the same IP address.
> 
> Hope this helps,
> 
> Katrin
> 
> On 10.07.24 13:02, Mike Lake wrote:
>> Hi all
>> 
>> I'm having serious problems with my Koha instance. It serves the OPAC
>> for the Australian Speleological Federation.  We are currently on Koha
>> 23.11 on a Debian 10.13. The system has been running fine for ages.
>> 
>> I was getting errors from the OOM killer:
>> 
>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
>> file-rss:0kB, shmem-rss:0kB
>> opac-search.pl invoked oom-killer:
>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
>> oom_score_adj=0
>> opac-search.pl cpuset=/ mems_allowed=0
>> 
>> So I shutdown Koha (took a while as I was out of memory)
>> systemctl stop koha-common.service
>> 
>> Rebooted the machine and when i bought Koha up:
>> systemctl start koha-common.service
>> Now I'm still getting 96 processes & errors taking all CPU and memory:
>> 
>> 3620  R  /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
>> 3622  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3624  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3625  D  /usr/bin/perl 
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3627  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3629  D  /usr/bin/perl 
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 3630  R  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 3633  D  /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>> 
>> Actually its 96 x opac-search.pl + 57 x 500.pl
>> 
>> A reboot does not help. Every time I start Koha those processes appear
>> and take all cores and memory.
>> 
>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
>> database server."
>> Attempts to start it: systemctl start mariadb.service
>> give that error probably because I'm out of memory does to the 100
>> perl processes running.
>> 
>> A "systemctl stop koha-common.service" does not stop or end those
>> opac-search.pl or 500.pl processes.
>> 
>> The /var/log/koha/opac/opac-error.log says:
>> 
>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
>> output before headers: 500.pl
>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
>> output before headers: opac-search.pl
>> 
>> Something is borked :-(   Help most welcome.
>> 
> _______________________________________________
> 
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha

-- 
Mike

------------------------------

Subject: Digest Footer

_______________________________________________
Koha mailing list
Koha at lists.katipo.co.nz
https://lists.katipo.co.nz/mailman/listinfo/koha

------------------------------

End of Koha Digest, Vol 225, Issue 8
************************************