*shrug* Things like /search?q= are in my robots.txt because crawling those pages... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

eli on May 10, 2011 | parent | context | favorite | on: Robots.txt is a suicide note

shrug

Things like /search?q= are in my robots.txt because crawling those pages is just a waste of everyone's time and resources.

Consider applying for YC's Winter 2027 batch! Applications are open till November 2.
Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact