[Koha] Out of memory when Koha starts due to opac-search.pl and 500.pl

Fri Aug 2 22:06:11 NZST 2024

Jumping in a bit late. But very recently I also saw quite a bit of traffic
from Singapore (Huawei). Using about ten different ip ranges (x.x.0.0/16)
and lots of different IPs. So blocking is hard.
If you use nginx, rate limiting might be a good option to explore. I added
a rate limit too on the x.x of the IP address.

Op ma 15 jul 2024 om 15:22 schreef Mike Lake <mikel at speleonics.com.au>:

> Hi Davis and all
>
> Ah :-)  Some very good help there. Yes I did some whois queries and many
> are from Singapore.
> Also it had not realised that there is an alias "ScriptAlias /cgi-bin/
> /usr/lib/cgi-bin/" as I had never looked at serve-cgi-bin.conf
> And yes why would anyone use an IP address to make a Koha query. I
> didn't realise that would hit that script alias then.
>
> I'm using fail2ban but up till now just for SSH. So tonight I have been
> looking at a regex for Apache to match some of the errors in the Koha
> logs.
>
> I'll get back with how I go. Regexes :-(
>
> Thanks :-)
> Mike Lake
>
>
> On 2024-07-15 9:49 am, David Cook wrote:
> > Hi Mike,
> >
> > It certainly sounds like a crawler/bot getting stuck in a loop. In your
> > log there, I see the client IP address 190.92.203.86, which belongs to
> > Huawei Cloud Singapore. I've seen a lot of bots/crawlers from Huawei
> > Cloud Singapore hitting Australian Koha sites over the last 6 months or
> > so.
> >
> > That 'AH02811: script not found or unable to stat:
> > /usr/lib/cgi-bin/koha' error is interesting. If you look at
> > /etc/apache2/conf-enabled/serve-cgi-bin.conf, you'll see that at a
> > global level /cgi-bin/ is aliased to /usr/lib/cgi-bin. So if the
> > crawler sent any HTTP requests using your IP address and not the
> > hostname, they'd be caught by that directive instead of your name-based
> > virtual host. Could be some other explanations for why the virtual host
> > wasn't used, but overall that would explain that message.
> >
> > Anyway, it's not necessarily a Koha-specific issue. If you're not
> > already using it, I'd suggest you look at installing and setting up
> > something like fail2ban. That said, I have noticed the bots out of
> > Huawei Cloud Singapore tend to cycle through a lot of different IP
> > addresses, which does make things tricky. Sometimes, it'll just use one
> > IP address that is easy to detect and block, but sometimes it might
> > just do 1-2 hits per IP address (from a variety of different IP
> > ranges).
> >
> > Let me know if you'd like to chat more about it.
> >
> > David Cook
> > Senior Software Engineer
> > Prosentient Systems
> > Suite 7.03
> > 6a Glen St
> > Milsons Point NSW 2061
> > Australia
> >
> > Office: 02 9212 0899
> > Online: 02 8005 0595
> >
> > -----Original Message-----
> >
> > Date: Sat, 13 Jul 2024 21:10:36 +1000
> > From: Mike Lake <mikel at speleonics.com.au>
> > To: koha at lists.katipo.co.nz
> > Subject: Re: [Koha] Out of memory when Koha starts due to
> >       opac-search.pl and 500.pl
> > Message-ID: <f034d85a454901421773c0f4df4a045f at speleonics.com.au>
> > Content-Type: text/plain; charset=UTF-8; format=flowed
> >
> > Hi
> >
> > Katrin suggested:
> >> it might be that you are hit by a bad crawler/bot
> >
> > Thanks Katrin. That *may* have been the cause. The system is working OK
> > at present. I did a complete shutdown and reboot.
> >
> > I did notice in the opac-error.log, which is now over 10 MB, a
> > recurring
> > query (see below) that was being made every 30 seconds. Exact same
> > query, clearly automated. That seems to have ended now.
> >
> > cgi-bin/koha/
> opac-search.pl?q=ccl%3Dau%3A%22James%2C%20Julia%20M.%22%20and%20itype%3ABK%20and%20su-to%3ACaves%20and%20ccode%3AArX%20and%20ccode%3AArX%20and%20holdingbranch%3AASFLIB%20and%20au%3AMartin%2C%20D.J.%20and%20au%3AWelch%2C%20Bruce%20R.%20and%20location%3AARC&sort_by=relevance_dsc&limit=available
> >
> > I was also getting these errors which were filling up the logs:
> >
> > [Fri Jul 12 21:24:30.948916 2024] [cgi:error] [pid 17819] [client
> > 190.92.203.86:51260]
> > AH02811: script not found or unable to stat: /usr/lib/cgi-bin/koha
> >
> > There is no such perl script
> > $ dpkg -L koha-common | grep '/usr/lib/cgi-bin/'
> > so I just created one to return "hello".
> >
> > Now our Koha instance is back up again and our VM is coping with the
> > load.  https://opac.caves.org.au
> >
> > Thanks for the reply.
> > I'll make another separate post on another current opac-error.log error
> > line, if it still persists, after I upgrade from 23.11.05
> >
> > Mike
> > ASF Sys Admin
> >
> > On 2024-07-13 7:34 pm, Katrin Fischer wrote:
> >> Hi Mike,
> >>
> >> it might be that you are hit by a bad crawler/bot and need to block
> >> access for them in your firewall. There are some that ignore the
> >> robots.txt and they can bring down a Koha server.
> >>
> >> I you look at the Apache access logs you might see that all those
> >> requests come from the same IP address.
> >>
> >> Hope this helps,
> >>
> >> Katrin
> >>
> >> On 10.07.24 13:02, Mike Lake wrote:
> >>> Hi all
> >>>
> >>> I'm having serious problems with my Koha instance. It serves the OPAC
> >>> for the Australian Speleological Federation.  We are currently on
> >>> Koha
> >>> 23.11 on a Debian 10.13. The system has been running fine for ages.
> >>>
> >>> I was getting errors from the OOM killer:
> >>>
> >>> oom_reaper: reaped process 1554 (opac-search.pl), now anon-rss:0kB,
> >>> file-rss:0kB, shmem-rss:0kB
> >>> opac-search.pl invoked oom-killer:
> >>> gfp_mask=0x6200ca(GFP_HIGHUSER_MOVABLE), nodemask=(null), order=0,
> >>> oom_score_adj=0
> >>> opac-search.pl cpuset=/ mems_allowed=0
> >>>
> >>> So I shutdown Koha (took a while as I was out of memory)
> >>> systemctl stop koha-common.service
> >>>
> >>> Rebooted the machine and when i bought Koha up:
> >>> systemctl start koha-common.service
> >>> Now I'm still getting 96 processes & errors taking all CPU and
> >>> memory:
> >>>
> >>> 3620  R  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/opac-imageviewer.pl
> >>> 3622  R  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
> >>> 3624  R  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
> >>> 3625  D  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
> >>> 3627  R  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
> >>> 3629  D  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
> >>> 3630  R  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
> >>> 3633  D  /usr/bin/perl
> >>> /usr/share/koha/opac/cgi-bin/opac/errors/500.pl
> >>>
> >>> Actually its 96 x opac-search.pl + 57 x 500.pl
> >>>
> >>> A reboot does not help. Every time I start Koha those processes
> >>> appear
> >>> and take all cores and memory.
> >>>
> >>> I do see that the MariaDB is down. "Failed to start MariaDB 10.3.39
> >>> database server."
> >>> Attempts to start it: systemctl start mariadb.service
> >>> give that error probably because I'm out of memory does to the 100
> >>> perl processes running.
> >>>
> >>> A "systemctl stop koha-common.service" does not stop or end those
> >>> opac-search.pl or 500.pl processes.
> >>>
> >>> The /var/log/koha/opac/opac-error.log says:
> >>>
> >>> [cgi:error] [pid 6911] [client 124.243.148.74:47310] End of script
> >>> output before headers: 500.pl
> >>> [cgi:error] [pid 6959] [client 190.92.216.104:41580] End of script
> >>> output before headers: opac-search.pl
> >>>
> >>> Something is borked :-(   Help most welcome.
> >>>
> >> _______________________________________________
> >>
> >> Koha mailing list  http://koha-community.org
> >> Koha at lists.katipo.co.nz
> >> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> > --
> > Mike
> >
> >
> > ------------------------------
> >
> > Subject: Digest Footer
> >
> > _______________________________________________
> > Koha mailing list
> > Koha at lists.katipo.co.nz
> > https://lists.katipo.co.nz/mailman/listinfo/koha
> >
> >
> > ------------------------------
> >
> > End of Koha Digest, Vol 225, Issue 8
> > ************************************
>
> --
> Mike
> _______________________________________________
>
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>