*the Archive Team are bloody-minded about preserving information* They won't do ...

sbierwagen · on May 10, 2011

AT, the organization, doesn't own any machines. Operations are all done by members, the majority of whom use consumer ISPs.

chalst · on May 11, 2011

Do all of them ignore robots.txt?

sbierwagen · on May 11, 2011

Well, when I iterated through the everything2.com namespace and downloaded the majority of their content[1], I respected their robots.txt. After I did that, they changed it to:

  User-agent: * 
  Disallow: /

Does that count?

1: http://bbot.org/blog/archives/2011/01/17/more_fun_with_wget/