3 Jun
2015
3 Jun
'15
1:30 a.m.
Paul A schreef op di 02-06-2015 om 09:27 [-0400]:
And I must admit that "google is getting my goat." Woodpeckering may be a fact of modern life, but at several million entries per day, there's got to be a limit.
Use robots.txt to slow down its crawl rate. Some crawlers don't obey that, but google does. -- Robin Sheat Catalyst IT Ltd. ✆ +64 4 803 2204 GPG: 5FA7 4B49 1E4D CAA4 4C38 8505 77F5 B724 F871 3BDF