Block web crawlers Bots
My server memory usage is increasing by some web search engine bots by access of OPAC. I tried put *robots.txt in * on /usr/share/koha/opac/htdocs As *User-agent: ** *Disallow: /* But still, my server is getting loaded by bots. The apache log file shows bots names like petalbot, Googlebot, AhrefsBot etc. How can I block all these bots' access to my OPAC pages? With Regards Jithin N *System Information* Koha 21.05 Deian 10
Hi, We use syntax : User-agent: * Disallow: / I don't know if it has same results. Note it make take some days to get out of bots scope. If you have an Nginx reverse proxy, you may use https://github.com/mitchellkrogza/nginx-ultimate-bad-bot-blocker Best regards, Le 27/01/2022 à 19:04, JITHIN N a écrit :
My server memory usage is increasing by some web search engine bots by access of OPAC. I tried put
*robots.txt in *
on /usr/share/koha/opac/htdocs
As
*User-agent: **
*Disallow: /*
But still, my server is getting loaded by bots. The apache log file shows bots names like petalbot, Googlebot, AhrefsBot etc. How can I block all these bots' access to my OPAC pages?
With Regards
Jithin N
*System Information*
Koha 21.05
Deian 10 _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
-- Fridolin SOMERS <fridolin.somers@biblibre.com> Software and system maintainer 🦄 BibLibre, France
Hi, well, the robots.txt is just an information given to bots that you don't wish your web server to be crawled. It is not a technical block, so bots can choose to ignore the robots.txt, and at least those not compliant with rules and standards will do so. You would need a technical solution to exclude these IPs, e.g. via your institution's firewall. You could also configure your Apache to exclude them. However, this needs of course constant maintenance to be up to date. A better way would be to exclude IPs according to certain parameters, e.g. if they contact your web server > n times within a given timeslot. You would have to look out for solutions for "ratelimiting". There seem to be Apache modules available for this, which might be of use. In another - not Koha - context, mod_qos was recommended. However, I wouldn't know which solution is best with Koha. I am unfortunately not so much into this, so please discuss this with a web server techie. Hope this is of use anyway, Regards, Anke -- Anke Bruns M.A. (LIS) Arbeitsgruppe "Anwendungs- und Informationssysteme" E-Mail: anke.bruns@gwdg.de --------------------------------------- Achtung! Neue Kontaktdaten! Gesellschaft für wissenschaftliche Datenverarbeitung mbH Göttingen (GWDG) Burckhardtweg 4, 37077 Göttingen, URL: https://gwdg.de Support: Tel.: +49 551 39-30000, URL: https://gwdg.de/support Sekretariat: Tel.: +49 551 39-30001, E-Mail: gwdg@gwdg.de Geschäftsführer: Prof. Dr. Ramin Yahyapour Aufsichtsratsvorsitzender: Prof. Dr. Norbert Lossau Sitz der Gesellschaft: Göttingen Registergericht: Göttingen, Handelsregister-Nr. B 598 --------------------------------------- Zertifiziert nach ISO 9001 ---------------------------------------
-----Ursprüngliche Nachricht----- Von: Koha <koha-bounces@lists.katipo.co.nz> Im Auftrag von JITHIN N Gesendet: Freitag, 28. Januar 2022 06:05 An: koha@lists.katipo.co.nz Betreff: [Koha] Block web crawlers Bots
My server memory usage is increasing by some web search engine bots by access of OPAC. I tried put
*robots.txt in *
on /usr/share/koha/opac/htdocs
As
*User-agent: **
*Disallow: /*
But still, my server is getting loaded by bots. The apache log file shows bots names like petalbot, Googlebot, AhrefsBot etc. How can I block all these bots' access to my OPAC pages?
With Regards
Jithin N
*System Information*
Koha 21.05
Deian 10 _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
Hi Instead of blocking webcrawlers and bots which want to index your records (which is generally a good thing) I recommend you implement a sitemap. * https://koha-community.org/manual/21.11/en/html/cron_jobs.html#sitemap * https://lists.katipo.co.nz/public/koha/2020-November/055401.html * https://github.com/Koha-Community/Koha/blob/master/misc/cronjobs/sitemap.pl Like this the webcrawlers will just access your sitemap instead of every single record in your database. This will decrease memory usage immensely. Blocking webcrawler by firewall can get very hairy because many bots do not have just one IP address where they come from. Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch Am 28.01.22 um 06:04 schrieb JITHIN N:
My server memory usage is increasing by some web search engine bots by access of OPAC. I tried put
*robots.txt in *
on /usr/share/koha/opac/htdocs
As
*User-agent: **
*Disallow: /*
But still, my server is getting loaded by bots. The apache log file shows bots names like petalbot, Googlebot, AhrefsBot etc. How can I block all these bots' access to my OPAC pages?
With Regards
Jithin N
*System Information*
Koha 21.05
Deian 10 _______________________________________________
Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha
participants (4)
-
Bruns, Anke -
Fridolin SOMERS -
JITHIN N -
Michael Kuhn