[Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl
Mike Lake
mikel at speleonics.com.au
Tue Jul 16 01:10:11 NZST 2024
Hi Davis and all
Ah :-) Some very good help there. Yes I did some whois queries and many
are from Singapore.
Also it had not realised that there is an alias "ScriptAlias /cgi-bin/
/usr/lib/cgi-bin/" as I had never looked at serve-cgi-bin.conf
And yes why would anyone use an IP address to make a Koha query. I
didn't realise that would hit that script alias then.
I'm using fail2ban but up till now just for SSH. So tonight I have been
looking at a regex for Apache to match some of the errors in the Koha
logs.
I'll get back with how I go. Regexes :-(
Thanks :-)
Mike Lake
On 2024-07-15 9:49 am, David Cook wrote:
> Hi Mike,
>
> It certainly sounds like a crawler/bot getting stuck in a loop. In your
> log there, I see the client IP address 190.92.203.86, which belongs to
> Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei
> Cloud Singapore hitting Australian Koha sites over the last 6 months or
> so.
>
> That 'AH02811: script not found or unable to stat:
> /usr/lib/cgi-bin/koha' error is interesting. If you look at
> /etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a
> global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the
> crawler sent any HTTP requests using your IP address and not the
> hostname, they'd be caught by that directive instead of your name-based
> virtual host. Could be some other explanations for why the virtual host
> wasn't used, but overall that would explain that message.
>
> Anyway, it's not necessarily a Koha-specific issue. If you're not
> already using it, I'd suggest you look at installing and setting up
> something like fail2ban. That said, I have noticed the bots out of
> Huawei Cloud Singapore tend to cycle through a lot of different IP
> addresses, which does make things tricky. Sometimes, it'll just use one
> IP address that is easy to detect and block, but sometimes it might
> just do 1-2 hits per IP address (from a variety of different IP
> ranges).
>
> Let me know if you'd like to chat more about it.
>
> David Cook
> Senior Software Engineer
> Prosentient Systems
> Suite 7.03
> 6a Glen St
> Milsons Point NSW 2061
> Australia
>
> Office: 02 9212 0899
> Online: 02 8005 0595
>
> -----Original Message-----
>
> Date: Sat, 13 Jul 2024 21:10:36 +1000
> From: Mike Lake <mikel at speleonics.com.au>
> To: koha at lists.katipo.co.nz
> Subject: Re: [Koha] Out of memory when Koha starts due to
> opac-search.pl and 500.pl
> Message-ID: <f034d85a454901421773c0f4df4a045f at speleonics.com.au>
> Content-Type: text/plain; charset=UTF-8; format=flowed
>
> Hi
>
> Katrin suggested:
>> it might be that you are hit by a bad crawler/bot
>
> Thanks Katrin. That *may* have been the cause. The system is working OK
> at present. I did a complete shutdown and reboot.
>
> I did notice in the opac-error.log, which is now over 10 MB, a
> recurring
> query (see below) that was being made every 30 seconds. Exact same
> query, clearly automated. That seems to have ended now.
>
> cgi-bin/koha/opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available
>
> I was also getting these errors which were filling up the logs:
>
> [Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client
> 190.92.203.86:51260]
> AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha
>
> There is no such perl script
> $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
> so I just created one to return "hello".
>
> Now our Koha instance is back up again and our VM is coping with the
> load. https://opac.caves.org.au
>
> Thanks for the reply.
> I'll make another separate post on another current opac-error.log error
> line, if it still persists, after I upgrade from 23.11.05
>
> Mike
> ASF Sys Admin
>
> On 2024-07-13 7:34 pm, Katrin Fischer wrote:
>> Hi Mike,
>>
>> it might be that you are hit by a bad crawler/bot and need to block
>> access for them in your firewall. There are some that ignore the
>> robots.txt and they can bring down a Koha server.
>>
>> I you look at the Apache access logs you might see that all those
>> requests come from the same IP address.
>>
>> Hope this helps,
>>
>> Katrin
>>
>> On 10.07.24 13:02, Mike Lake wrote:
>>> Hi all
>>>
>>> I'm having serious problems with my Koha instance. It serves the OPAC
>>> for the Australian Speleological Federation. We are currently on
>>> Koha
>>> 23.11 on a Debian 10.13. The system has been running fine for ages.
>>>
>>> I was getting errors from the OOM killer:
>>>
>>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
>>> file-rss:0kB, shmem-rss:0kB
>>> opac-search.pl invoked oom-killer:
>>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
>>> oom_score_adj=0
>>> opac-search.pl cpuset=/ mems_allowed=0
>>>
>>> So I shutdown Koha (took a while as I was out of memory)
>>> systemctl stop koha-common.service
>>>
>>> Rebooted the machine and when i bought Koha up:
>>> systemctl start koha-common.service
>>> Now I'm still getting 96 processes & errors taking all CPU and
>>> memory:
>>>
>>> 3620 R /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
>>> 3622 R /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>> 3624 R /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>> 3625 D /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>>> 3627 R /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>> 3629 D /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>>> 3630 R /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>> 3633 D /usr/bin/perl
>>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
>>>
>>> Actually its 96 x opac-search.pl + 57 x 500.pl
>>>
>>> A reboot does not help. Every time I start Koha those processes
>>> appear
>>> and take all cores and memory.
>>>
>>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
>>> database server."
>>> Attempts to start it: systemctl start mariadb.service
>>> give that error probably because I'm out of memory does to the 100
>>> perl processes running.
>>>
>>> A "systemctl stop koha-common.service" does not stop or end those
>>> opac-search.pl or 500.pl processes.
>>>
>>> The /var/log/koha/opac/opac-error.log says:
>>>
>>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
>>> output before headers: 500.pl
>>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
>>> output before headers: opac-search.pl
>>>
>>> Something is borked :-( Help most welcome.
>>>
>> _______________________________________________
>>
>> Koha mailing list http://koha-community.org
>> Koha at lists.katipo.co.nz
>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>
> --
> Mike
>
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Koha mailing list
> Koha at lists.katipo.co.nz
> https://lists.katipo.co.nz/mailman/listinfo/koha
>
>
> ------------------------------
>
> End of Koha Digest, Vol 225, Issue 8
> ************************************
--
Mike
More information about the Koha
mailing list