I usually encounter rate limiting when trying to scrape some websites. I was eve... | Hacker News

Hacker Newsnew | past | comments | ask | show | jobs | submit

DeathArrow on May 17, 2024 | parent | context | favorite | on: Visualizing algorithms for rate limiting

I usually encounter rate limiting when trying to scrape some websites. I was even rate limited when manually browsing a website which considered I am a bot.

aabbcc1241 on May 19, 2024 [–]

You can rate limit your scrapper, to match the designed limit of the target website.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact