Replace curl with wget, get robots.txt support. Write timed-out URLs into a file for the next identical run.
The idea stays: you can jerry-rig a web crawler from pedestrian, readily available parts, and quickly test the idea for which you wanted to crawl the Web.
The point is that bash, xargs, wget, make, etc are well-understood, well-maintained tools. They are not even necessarily simple, but they likely contain very few unknown unknowns.
The idea stays: you can jerry-rig a web crawler from pedestrian, readily available parts, and quickly test the idea for which you wanted to crawl the Web.
The point is that bash, xargs, wget, make, etc are well-understood, well-maintained tools. They are not even necessarily simple, but they likely contain very few unknown unknowns.