2 Nov
2009
2 Nov
'09
4:59 p.m.
MJ Ray (2009/11/02 15:14 +0000):
It could contain any of http://www.robotstxt.org/robotstxt.html
I didn't know this link, thanks.
It should contain whatever will make robots behave as you want.
That's what I don't know -- how I want them to behave. My guess is that everything shold be disallowed because no page has a meaning without arguments... What I would like to know is whether this guess is correct or not.
I don't think our libraries currently use one but I've not checked specifically for this.
I apache error logs and noticed some clients look for this file so I thought I could create one so that the error log does not contain this not too interesting entry. Cheers, Sébastien.