Hacker News
new
|
past
|
comments
|
ask
|
show
|
jobs
|
submit
login
_drimzy
on June 12, 2020
|
parent
|
context
|
favorite
| on:
A Facebook crawler was making 7M requests per day ...
Why would a crawler want to crawl the same page every second (let alone 71 times a second). Even if I were Google, wouldn't crawling any page every few mins (or say every minute) be way more than enough for freshness?
Consider applying for YC's Winter 2027 batch!
Applications
are open till November 2.
Guidelines
|
FAQ
|
Lists
|
API
|
Security
|
Legal
|
Apply to YC
|
Contact
Search: