Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Which begs the question, are there regions of the web that just simply aren't indexed?


Absolutely there are, despite our efforts. See the bmtcinfo.com example mentioned elsewhere on this page. But normally there's a good reason if we don't have a page, e.g. bmtcinfo.com returns an error unless you add a "www" in front of it.


Those that blacklist googlebot on their robots.txt


...allegedly.




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: