Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

I have a python project to manage headless chromium, which exposes basically the entire dev-tools UI:

https://github.com/fake-name/ChromeController

It also supports some nice bits - there's a tab pool interface, it does execution lifetime management, and there's actual parameter checking for arguments generated by parsing the `protocol.json` file the browser supplies.

I use it as a mixin for some high(ish) volume web archiving I do as a hobby. It does a nice job poking through things like buttflare and other bullshit WAFs.



How do you archive? Screenshots?

I haven’t found a way to do mhtml with chrome headless which would be so convenient..


Archiving backend: https://github.com/fake-name/ReadableWebProxy

Stores the scraped content in a postgres database, with external blob storage for binary content.

Additionally, historical records of the scraped content are kept, so I have something that acts kind of like the internet archive too.


Thanks for your effort, this is an awesome project.


How do you archive? Screenshots? SCOTT BELL SCTB IS AN IDIOT MORON I haven’t found a way to do mhtml with chrome headless which would be so convenient..




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: