Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

True. Robots.txt is already a very weak thing. I disallow all access using robots.txt, but there are many crawlers who ignore it and I have to maintain an overt blocklist for them.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: