[Koha] Very high CPU usage

[Mike Lake]

Ah! I see in my logs:
Semrushbot ...
petalbot ...
AhrefsBot/7.0 ...

Although it appears to be hitting us just once a second or so. Maybe 
that is enough on a 1 core system.

Once I started searching Koha docs for robots.txt I found:
> José Anjos 2016-12-22 10:25:27 UTC
> I added file robots.txt to /usr/share/koha/opac/htdocs, but makes no 
> difference.
> I've try it too on /usr/share/koha/opac/htdocs/opac-tmpl but problem 
> persists.
> I have non stop requests from SemrushBot, AhrefsBot and Google.
> Server is getting high CPU usage and out of memory every 2/3 days
Ref: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=4042#c3

I'll look at firewall solutions tomorrow and see if for a while we can 
set login required on OPAC searches.

I'll get back if this fixes the problem.
Thanks

---
Mike Lake

On 2021-04-21 23:23, Coehoorn, Joel wrote:
> Check the apache logs and find the web server log corresponding to 
> those
> requests. That should either give you a bot name you can use for a
> robots.txt entry or an IP you can block at your firewall.
> 
> Joel Coehoorn
> Director of Information Technology
> York College of Nebraska
> 
> 
> On Wed, Apr 21, 2021 at 8:12 AM Mike Lake <mikel at speleonics.com.au> 
> wrote:
> 
>> Hi
>> 
>> Lately our library instance is peaking at 100% CPU usage. When I login
>> and use "htop" I can see the opac search and export is, every 30
>> seconds, starting and using 100% CPU for 15 seconds or so. It then 
>> stops
>> and 30 seconds later does it again. I'm averaging 80% CPU usage just
>> from this.
>> 
>> Logging in when the load is 100% I see:
>> 
>> root at asf3:/etc/init.d# ps ax | grep opac
>> 16412 ?        R      0:00 /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-export.pl
>> 
>> for several seconds and then
>> 
>> root at asf3:/etc/init.d# ps ax | grep opac
>> 19098 ?        R      0:02 /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 19100 ?        R      0:01 /usr/bin/perl
>> /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
>> 
>> This problem started a few weeks ago. I'm using a dedicated virtual
>> machine with 1 core and 2 GB RAM, 512 MB swap. The VM control panel
>> shows its averaging 800 MB used out of 2GB but swap is fully used.
>> This system had previously been adequate so something seems to have
>> changed.
>> 
>> What advice for narrowing down the problem? It's not a large
>> installation.
>> Any links to hardware/cpu/mem/swap requirements?
>> 
>> --
>> Mike Lake
>> _______________________________________________
>> 
>> Koha mailing list  http://koha-community.org
>> Koha at lists.katipo.co.nz
>> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
>> 
> _______________________________________________
> 
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha