Hi Davis and all Ah :-) Some very good help there. Yes I did some whois queries and many are from Singapore. Also it had not realised that there is an alias "ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/" as I had never looked at serve-cgi-bin.conf And yes why would anyone use an IP address to make a Koha query. I didn't realise that would hit that script alias then. I'm using fail2ban but up till now just for SSH. So tonight I have been looking at a regex for Apache to match some of the errors in the Koha logs. I'll get back with how I go. Regexes :-( Thanks :-) Mike Lake On 2024-07-15 9:49 am, David Cook wrote:
Hi Mike,
It certainly sounds like a crawler/bot getting stuck in a loop. In your log there, I see the client IP address 190.92.203.86, which belongs to Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei Cloud Singapore hitting Australian Koha sites over the last 6 months or so.
That 'AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha' error is interesting. If you look at /etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the crawler sent any HTTP requests using your IP address and not the hostname, they'd be caught by that directive instead of your name-based virtual host. Could be some other explanations for why the virtual host wasn't used, but overall that would explain that message.
Anyway, it's not necessarily a Koha-specific issue. If you're not already using it, I'd suggest you look at installing and setting up something like fail2ban. That said, I have noticed the bots out of Huawei Cloud Singapore tend to cycle through a lot of different IP addresses, which does make things tricky. Sometimes, it'll just use one IP address that is easy to detect and block, but sometimes it might just do 1-2 hits per IP address (from a variety of different IP ranges).
Let me know if you'd like to chat more about it.
David Cook Senior Software Engineer Prosentient Systems Suite 7.03 6a Glen St Milsons Point NSW 2061 Australia
Office: 02 9212 0899 Online: 02 8005 0595
-----Original Message-----
Date: Sat, 13 Jul 2024 21:10:36 +1000 From: Mike Lake <mikel@speleonics.com.au> To: koha@lists.katipo.co.nz Subject: Re: [Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl Message-ID: <f034d85a454901421773c0f4df4a045f@speleonics.com.au> Content-Type: text/plain; charset=UTF-8; format=flowed
Hi
Katrin suggested:
it might be that you are hit by a bad crawler/bot
Thanks Katrin. That *may* have been the cause. The system is working OK at present. I did a complete shutdown and reboot.
I did notice in the opac-error.log, which is now over 10 MB, a recurring query (see below) that was being made every 30 seconds. Exact same query, clearly automated. That seems to have ended now.
cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available
I was also getting these errors which were filling up the logs:
[Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client 190.92.203.86:51260] AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha
There is no such perl script $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/' so I just created one to return "hello".
Now our Koha instance is back up again and our VM is coping with the load. https://opac.caves.org.au
Thanks for the reply. I'll make another separate post on another current opac-error.log error line, if it still persists, after I upgrade from 23.11.05
Mike ASF Sys Admin
On 2024-07-13 7:34 pm, Katrin Fischer wrote:
Hi Mike,
it might be that you are hit by a bad crawler/bot and need to block access for them in your firewall. There are some that ignore the robots.txt and they can bring down a Koha server.
I you look at the Apache access logs you might see that all those requests come from the same IP address.
Hope this helps,
Katrin
On 10.07.24 13:02, Mike Lake wrote:
Hi all
I'm having serious problems with my Koha instance. It serves the OPAC for the Australian Speleological Federation. We are currently on Koha 23.11 on a Debian 10.13. The system has been running fine for ages.
I was getting errors from the OOM killer:
oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB, file-rss:0kB, shmem-rss:0kB opac-search.pl invoked oom-killer: gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0, oom_score_adj=0 opac-search.pl cpuset=/ mems_allowed=0
So I shutdown Koha (took a while as I was out of memory) systemctl stop koha-common.service
Rebooted the machine and when i bought Koha up: systemctl start koha-common.service Now I'm still getting 96 processes & errors taking all CPU and memory:
3620 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl 3622 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3624 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3625 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3627 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3629 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 3630 R /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl 3633 D /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
Actually its 96 x opac-search.pl + 57 x 500.pl
A reboot does not help. Every time I start Koha those processes appear and take all cores and memory.
I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39 database server." Attempts to start it: systemctl start mariadb.service give that error probably because I'm out of memory does to the 100 perl processes running.
A "systemctl stop koha-common.service" does not stop or end those opac-search.pl or 500.pl processes.
The /var/log/koha/opac/opac-error.log says:
[cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script output before headers: 500.pl [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script output before headers: opac-search.pl
Something is borked :-( Help most welcome.
_______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
-- Mike
------------------------------
Subject: Digest Footer
_______________________________________________ Koha mailing list Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha
------------------------------
End of Koha Digest, Vol 225, Issue 8 ************************************
-- Mike