Hi Hugo
You 're not the only one who has suffered this from Google, but Baidu is worse and some others as well, giving you telegram answers to your points...
Yes I have also suffered a lot from crawlers, and I have spend a lot of hours trying to adjut firewalls, robots....
What version of Koha you're using? moderns one have a command koha-sitemap (If I am not wrong)
I'm on Koha 16.11.04, and yes, there is a command "koha-sitemap". Unfortunately I couldn't find anything about it in the the Koha 16.11 manual. The only sources I found was: * https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11190 * http://search.cpan.org/~fredericd/Koha-Contrib-Tamil-0.011/bin/koha-sitemap Since I couldn't find the command in https://wiki.koha-community.org/wiki/Commands_provided_by_the_Debian_package... I added it there in the new section "Bot-related". But I think it'd be good to propagate this better - or even activate it by default because who would want such behavior?
Google Webmaster warns you that it has not inmediate effect, you should wait a little more...
In summary you have done all the expected work, now it is just time to ajust it and wait for the results
With the combinations of robots.txt, koha-sitemap & firewall I have been happy for a long time... but you're not save from this never
:( I am sorry..
I'm getting more patient now since I was at least able to cure the symptoms... Best wishes & thanks again: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch