4 Jun
2015
4 Jun
'15
1:46 a.m.
Paul A schreef op wo 03-06-2015 om 09:50 [-0400]:
This was specifically for Google, so I had a look at the wiki article and did some more digging. It seems that at least Bing and Yahoo (neither of whom appear to abuse our servers) support the Crawl-Delay
It seems you're right. That's a bit annoying of them. I apparently had it in my head that all the major search engines did. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF