[Koha] Problems with the facebook web crawler

[Michael Kuhn]

Hi Nigel

In such a case I would advise to create a sitemap - unfortunately this 
Koha feature seems not so well documented, but the following may give 
you a start:

* https://lists.katipo.co.nz/public/koha/2020-November/055401.html

* 
https://wiki.koha-community.org/wiki/Commands_provided_by_the_Debian_packages#koha-sitemap

* https://koha-community.org/manual/24.05/en/html/cron_jobs.html#sitemap

Best wishes: Michael
-- 
Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis
Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz
T 0041 (0)61 261 55 61 · E mik at adminkuhn.ch · W www.adminkuhn.ch



Am 25.07.24 um 13:27 schrieb Nigel Titley:
> Is anyone else getting problems with the facebook web crawler hammering 
> their OPAC search function?
> 
> This has been happening on and off for a couple of months but set in 
> with a vengeance a couple of days ago. The crawler is hitting us with 
> many OPAC search queries, beyond the capacity of our system to respond.
> 
> robots.txt is being ignored
> 
> I started by blocking facebook's entire IPv6 range as the queries were 
> all coming in over IPv6. They responded by switching to IPv4 and because 
> they have a number of blocks it wasn't practical to block each and every 
> one of them.
> 
> I've temporarily switched off OPAC entirely and the system has returned 
> to normal and I can at least perform intranet functions but this is 
> obviously non-ideal.
> 
> Does anyone have any thoughts on this?
> 
> I'm running 22.05.13.000 on Ubuntu.
> 
> Thanks
> 
> Nigel
> _______________________________________________
> 
> Koha mailing list  http://koha-community.org
> Koha at lists.katipo.co.nz
> Unsubscribe: https://lists.katipo.co.nz/mailman/listinfo/koha