3 Jun
2015
3 Jun
'15
3:13 a.m.
Paul A schreef op di 02-06-2015 om 19:48 [-0400]:
How do you "slow it down"? To the best of my knowledge it's an on/off choice. See <https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt>.
https://en.wikipedia.org/wiki/Robots_exclusion_standard#Crawl-delay_directiv... PS: no need to CC me in replies, I'm on the mailing list. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF