Hi Yes this is annoying issue with boots, this is google but there are plenty of them... You should use robots.txt propertly, but If I am not wrong with Google it is more effective go to google webmaster web and modify the googleboot behaviour with your koha installarion You should also use a koha-sitemap.. depending on the version is out of the box functionality Perhaps you may think on use ufw or even ufw + fail2ban Some times bots are nightmare 2017-05-03 13:49 GMT+02:00 Mark Alexander <marka@pobox.com>:

...

When I searched for who is 66.249.64.32 I saw this IP addresse belongs to Google.

This does seem to be the Google indexer:

% nslookup 66.249.64.32 ... 32.64.249.66.in-addr.arpa name = crawl-66-249-64-32.googlebot.com.

I haven't seen this problem (yet), but perhaps that is because I have a /usr/share/koha/opac/htdocs/robots.txt containing this:

Crawl-delay: 60

User-agent: * Disallow: /

User-agent: Googlebot Disallow: /cgi-bin/koha/opac-search.pl Disallow: /cgi-bin/koha/opac-showmarc.pl Disallow: /cgi-bin/koha/opac-detailprint.pl Disallow: /cgi-bin/koha/opac-ISBDdetail.pl Disallow: /cgi-bin/koha/opac-MARCdetail.pl Disallow: /cgi-bin/koha/opac-reserve.pl Disallow: /cgi-bin/koha/opac-export.pl Disallow: /cgi-bin/koha/opac-detail.pl Disallow: /cgi-bin/koha/opac-authoritiesdetail.pl _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha

-- *Hugo Agud - Orex Digital * *www.orex.es <http://www.orex.es>* <http://www.orex.es/> [image: www.orex.es/koha] <http://www.orex.es/koha> [image: www.orex.es/vufind] <http://www.orex.es/vufind> <http://www.orex.es/omeka> Director Calle Sant Joaquin,117, 2º-3ª · 08922 Santa Coloma de Gramanet - Tel: 933 856 138 hagud@orex.es · http://www.orex.es/ No imprima este mensaje a no ser que sea necesario. Una tonelada de papel implica la tala de 15 árboles y el consumo de 250.000 litros de agua. Aviso de confidencialidad Este mensaje contiene información que puede ser CONFIDENCIAL y/o de USO RESTRINGIDO. Si usted no es el receptor deseado del mensaje (ni está autorizado a recibirlo por el remitente), no está autorizado a copiar, reenviar o divulgar el mensaje o su contenido. Si ha recibido este mensaje por error, por favor, notifíquenoslo inmediatamente y bórrelo de su sistema.

Hi You 're not the only one who has suffered this from Google, but Baidu is worse and some others as well, giving you telegram answers to your points... Yes I have also suffered a lot from crawlers, and I have spend a lot of hours trying to adjut firewalls, robots.... What version of Koha you're using? moderns one have a command koha-sitemap (If I am not wrong) Google Webmaster warns you that it has not inmediate effect, you should wait a little more... In summary you have done all the expected work, now it is just time to ajust it and wait for the results With the combinations of robots.txt, koha-sitemap & firewall I have been happy for a long time... but you're not save from this never :( I am sorry.. 2017-05-03 16:14 GMT+02:00 Michael Kuhn <mik@adminkuhn.ch>:

Hi Mark and Hugo

Many thanks for your hints! I have now done the following.

1. I created a file "/usr/share/koha/opac/htdocs/robots.txt" containing this:

Sitemap: sitemapindex.xml User-agent: * Disallow: /cgi-bin/

2. I generated a Koha sitemap using the seemingly undocumented Perl script "sitemap.pl" (according to https://bugs.koha-community.or g/bugzilla3/show_bug.cgi?id=11190) which created the file "/usr/share/koha/opac/htdocs/sitemapindex.xml" and the file "/usr/share/koha/opac/htdocs/sitemap0001.xml" containing the URLs.

3. Even after a complete reboot of the host the "opac-search.pl" processes were still there, appearing immediately after the reboot!

4. I went to Google Webmaster Tools where I downloaded the HTML confirmation file "googleb56bd3db2af352b1.html" and placed it in "/usr/share/koha/opac/htdocs" as well. I also followed the steps given on the Wemaster Tools page, i. e. I called the URL and I confirmed the download.

5. Even after a complete reboot of the host the "opac-search.pl" processes were still there, appearing immediately after the reboot!

6. I then installed the Uncomplicated Firewall / UFW where I applied the following rules and enabled it:

# ufw status Status: active

To Action From -- ------ ---- 22/tcp ALLOW Anywhere 80/tcp ALLOW Anywhere 8080/tcp ALLOW Anywhere Anywhere DENY 66.249.64.32

But however this is possible, still Googlebot is crawling and eating CPU! This can be seen in the log file "plack.log" where hundreds and thousands of lines like the following can be seen:

66.249.64.32 - - [03/May/2017:15:48:28 +0200] "GET /opac/ opac-authoritiesdetail.pl?authid=12872 HTTP/1.1" 200 17703 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

And I also found another bot:

62.138.14.218 - - [03/May/2017:15:48:29 +0200] "GET /opac/ opac-search.pl?q=se,phr:%22Zeitreise%22 HTTP/1.1" 200 54672 "-" "Linguee Bot (http://www.linguee.com/bot; bot@linguee.com)"

Now what I don't understand is how Googlebot (66.249.64.32) can access the webserver even if it is blocked by UFW?!

9. Already quite desperate I finally executed the following line to drop all packets from 66.249.64.32.

# iptables -I INPUT -s 66.249.64.32 -j DROP # iptables -I INPUT -s 62.138.14.218 -j DROP

And yes - this actually stopped these harassing bots.

But of course, next was this:

66.249.64.35 - - [03/May/2017:15:59:21 +0200] "GET /opac/ opac-authoritiesdetail.pl?authid=16429 HTTP/1.1" 200 17661 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"

I also dropped this IP address and now - finally! - the OPAC search for the normal user works as fast as expected.

In fact I can't believe I should be the only one experiencing this behavior (especially since the stuff about creating "sitemap.pl" is quite hidden and however undocumented in the Koha manual).

The other thing is people usually say it's a good thing to be indexed by Google. Today however, I won't agree. Maybe tomorrow, I will then try to delete the rule which drops the Google packets and I really hope Google will then do what it is told to do in "robots.txt", using the Koha sitemap.

So all this just for the record - maybe it will help someone in the future.

Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch

-- *Hugo Agud - Orex Digital * *www.orex.es <http://www.orex.es>* <http://www.orex.es/> [image: www.orex.es/koha] <http://www.orex.es/koha> [image: www.orex.es/vufind] <http://www.orex.es/vufind> <http://www.orex.es/omeka> Director Calle Sant Joaquin,117, 2º-3ª · 08922 Santa Coloma de Gramanet - Tel: 933 856 138 hagud@orex.es · http://www.orex.es/ No imprima este mensaje a no ser que sea necesario. Una tonelada de papel implica la tala de 15 árboles y el consumo de 250.000 litros de agua. Aviso de confidencialidad Este mensaje contiene información que puede ser CONFIDENCIAL y/o de USO RESTRINGIDO. Si usted no es el receptor deseado del mensaje (ni está autorizado a recibirlo por el remitente), no está autorizado a copiar, reenviar o divulgar el mensaje o su contenido. Si ha recibido este mensaje por error, por favor, notifíquenoslo inmediatamente y bórrelo de su sistema.

Excerpts from Michael Kuhn's message of 2017-05-03 16:14:55 +0200:

# ufw status Status: active

To Action From -- ------ ---- 22/tcp ALLOW Anywhere 80/tcp ALLOW Anywhere 8080/tcp ALLOW Anywhere Anywhere DENY 66.249.64.32

But however this is possible, still Googlebot is crawling and eating CPU!

I haven't used UFW, but I'm looking at the documentation here: https://help.ubuntu.com/community/UFW and it seems that the order of the rules is important. Quote: Once a rule is matched the others will not be evaluated (see manual below) so you must put the specific rules first. As rules change you may need to delete old rules to ensure that new rules are put in the proper order. and from the man page: Rule ordering is important and the first match wins. Therefore when adding rules, add the more specific rules first with more general rules later.

The sitemapper tool is baked in Koha. The packages have a handy koha-sitemap script. Regards. El mié., 3 may. 2017 a las 11:45, Michael Kuhn (<mik@adminkuhn.ch>) escribió:

Hi Mark

...

...

# ufw status Status: active

To Action From -- ------ ---- 22/tcp ALLOW Anywhere 80/tcp ALLOW Anywhere 8080/tcp ALLOW Anywhere Anywhere DENY 66.249.64.32

But however this is possible, still Googlebot is crawling and eating CPU!

I haven't used UFW, but I'm looking at the documentation here:

https://help.ubuntu.com/community/UFW

and it seems that the order of the rules is important. Quote:

Once a rule is matched the others will not be evaluated (see manual below) so you must put the specific rules first. As rules change you may need to delete old rules to ensure that new rules are put in the proper order.

and from the man page:

Rule ordering is important and the first match wins. Therefore when adding rules, add the more specific rules first with more general rules later.

Many thanks for the clarification! Yes, this makes sense.

So I would have to delete all rules and write them again in the correct order? Very "uncomplicated" indeed ;-)

However, I now deletd the rule for 66.249.64.32 in UFW since the rule in iptables succeeded (without giving any special order).

Best wishes & thanks again: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch _______________________________________________ Koha mailing list http://koha-community.org Koha@lists.katipo.co.nz https://lists.katipo.co.nz/mailman/listinfo/koha

-- Tomás Cohen Arazi Theke Solutions (https://theke.io <http://theke.io/>) ✆ +54 9351 3513384 GPG: B2F3C15F

Hi Magnus

...

The sitemapper tool is baked in Koha. The packages have a handy koha-sitemap script.

And the documentation for it is available if you do this on the command line:

$ man koha-sitemap

Yes - but before, of course, the world needs to know there IS such a command. That's why I wrote in my e-mail from 3rd May 2017 16:38 the following: Unfortunately I couldn't find anything about it in the the Koha 16.11 manual. The only sources I found was: * https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11190 * http://search.cpan.org/~fredericd/Koha-Contrib-Tamil-0.011/bin/koha-sitemap Since I couldn't find the command in https://wiki.koha-community.org/wiki/Commands_provided_by_the_Debian_package... I have now added "koha-sitemap" there in the new section "Bot-related". I still think it'd be good to propagate this better (for example mention it in the Koha manual) - or even activate it by default because who would want such behavior? Best wishes: Michael -- Geschäftsführer · Diplombibliothekar BBS, Informatiker eidg. Fachausweis Admin Kuhn GmbH · Pappelstrasse 20 · 4123 Allschwil · Schweiz T 0041 (0)61 261 55 61 · E mik@adminkuhn.ch · W www.adminkuhn.ch