3 Jun
2015
3 Jun
'15
1:48 a.m.
At 11:30 AM 6/3/2015 +1200, Robin Sheat wrote:
Paul A schreef op di 02-06-2015 om 09:27 [-0400]:
And I must admit that "google is getting my goat." Woodpeckering may be a fact of modern life, but at several million entries per day, there's got to be a limit.
Use robots.txt to slow down its crawl rate. Some crawlers don't obey that, but google does.
How do you "slow it down"? To the best of my knowledge it's an on/off choice. See <https://developers.google.com/webmasters/control-crawl-index/docs/robots_txt>. tnx and br -- p.