Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
sbierwagen
on May 10, 2011
|
parent
|
context
|
favorite
| on:
Robots.txt is a suicide note
AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.
chalst
on May 11, 2011
[–]
Do all of them ignore robots.txt?
sbierwagen
on May 11, 2011
|
parent
[–]
Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:
User-agent: * Disallow: /
Does that count?
1:
http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/
Consider applying for YC's Winter 2027 batch!
Applications
are open till November 2.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search:
