Ah! I see in my logs: Semrushbot ... petalbot ... AhrefsBot/7.0 ... Although it appears to be hitting us just once a second or so. Maybe that is enough on a 1 core system. Once I started searching Koha docs for robots.txt I found:
José Anjos 2016-12-22 10:25:27 UTC I added file robots.txt to /usr/share/koha/opac/htdocs, but makes no difference. I've try it too on /usr/share/koha/opac/htdocs/opac-tmpl but problem persists. I have non stop requests from SemrushBot, AhrefsBot and Google. Server is getting high CPU usage and out of memory every 2/3 days Ref: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=4042#c3
I'll look at firewall solutions tomorrow and see if for a while we can set login required on OPAC searches. I'll get back if this fixes the problem. Thanks --- Mike Lake On 2021-04-21 23:23, Coehoorn, Joel wrote:
Check the apache logs and find the web server log corresponding to those requests. That should either give you a bot name you can use for a robots.txt entry or an IP you can block at your firewall.
Joel Coehoorn Director of Information Technology York College of Nebraska
On Wed, Apr 21, 2021 at 8:12 AM Mike Lake <mikel@speleonics.com.au> wrote:
Hi
Lately our library instance is peaking at 100% CPU usage. When I login and use "htop" I can see the opac search and export is, every 30 seconds, starting and using 100% CPU for 15 seconds or so. It then stops and 30 seconds later does it again. I'm averaging 80% CPU usage just from this.
Logging in when the load is 100% I see:
root@asf3:/etc/init.d# ps ax | grep opac 16412 ? R 0:00 /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-export.pl
for several seconds and then
root@asf3:/etc/init.d# ps ax | grep opac 19098 ? R 0:02 /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl 19100 ? R 0:01 /usr/bin/perl /usr/share/koha/opac/cgi-bin/opac/opac-search.pl
This problem started a few weeks ago. I'm using a dedicated virtual machine with 1 core and 2 GB RAM, 512 MB swap. The VM control panel shows its averaging 800 MB used out of 2GB but swap is fully used. This system had previously been adequate so something seems to have changed.
What advice for narrowing down the problem? It's not a large installation. Any links to hardware/cpu/mem/swap requirements?
-- Mike Lake _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
_______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha