Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

There hardly are any "illegitimate" uses. The web is meant to be machine-readable (we wouldn't have Google or anything nearly as convenient in the first place if it wasn't). Whatever have been published is public and should not come with artificial limitations on how do you read and process it. Blocking crawling should be outlawed as it clearly is a monopolistic practice. E.g. I want to build my own crawler to index and categorize the web subset I choose for me. I believe this is a perfectly legitimate use. But they will probably try to stop me.


> Blocking crawling should be outlawed

That's overly broad. But maybe it should be illegal to have exceptions only for major monopolies.


Turn it around at least for a few minutes. Does a website operator have to handle whatever arbitrary traffic you want to throw at them from your crawler?

They’re the ones choosing to use tech that’s blocking you. Proposing to make it illegal for them to make that choice or to speak to you differently than they speak to other users of their site may give you some idea of the resistance you’re likely to face to this proposal.


i think there is a line somewhere along with beeing accountable in a business sense.

ie. if your internet host just hands out information, you are free to block/throttle as you please.

as soon as you are taking money (operate as a business), you are accountable and must not discriminate.

so: > Does a website operator have to handle whatever arbitrary traffic you want to throw at them

absolutely, yes!




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: