Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Also, tor + privoxy as a rule-of-thumb when your scraper IP is getting banned.

And - I know, it is (now) so old - memcached as a good place to store things.



When your scraper IP is getting banned, that's typically a sign that you should stop, talk to the site owner, and/or reconsider what you're doing.


That's not always the case.

While government or private contractors do not allow to gather that data by any citizen (there are examples of cities allowing it), I find myself in the right to do so, and also, in the right to redistribute that data freely so other developers can play / investigate / learn with it.

Why? Well, for starters:

1) Their own apps suck (or are inexistent)

2) They don't want to help their own users.

3) It's fun

4) Their services are paid with public money.

5) It raises awareness on the need of public data legislation.

There's a lot more to talk on the subject.

If you want to check it, http://citybik.es

I am helping projects and visualizations like: http://bikes.oobrien.com/ or my own http://citybik.es/realtime/




Consider applying for YC's Winter 2027 batch! Applications are open till November 2.

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: