Hacker Newsnew | past | comments | ask | show | jobs | submit | ccgreg's commentslogin

Common Crawl never said we filtered out the bad stuff. Researchers into hate speech depend on our data.

Abstract:

n 2026, the Event Horizon Telescope (EHT) carried out the first time-lapse monitoring campaign comprising a series of 18 observations of the black hole at the center of M87 with a cadence of approximately twice per week over a few months, matched to the expected dynamical timescale. Here we present a first-look image from the campaign, reconstructed from a single four-minute scan obtained on 2026 April 3.


Common Crawl is used for a large number of purposes other than "training" -- 13,000 research papers and counting.

https://commoncrawl.org/research-papers


We need to abandon the Gregorian calendar -- it's clearly a software maintenance nightmare -- and return to the Julian calendar.


Can you point out an example where someone made that mistake?



It's as if Common Crawl is attempting to be a sample of the web.


> The CC CDX endpoint, index.commoncrawl.org, historically has been easily overwhelmed and unreliable

Use our Parquet index.

It's also worth noting that archive.org downloads all of our crawl data and adds it to the IA Wayback Machine.


How does that work for non-profit web crawls like the Common Crawl Foundation?


Common Crawl is text-only.


Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: