The lack of cameras is absolutely not what's stopping police from doing their job, its the complete lack of accountability or responsibility.

Someone breaks into a car. There is no camera, no witnesses immediately came forward.

Please do tell me what you want the police department to do here.

Someone breaks into a car. There is a camera. The cops don't care and the owner doesn't know the guy.

Please tell me what you think the police would do here?

The world would have been better without you.

ICE and the FBI are pretty formal institutions.

Yeah, but I'm not afraid of them, I'm afraid of the hate mob.

Because it's the only possible way out for them.

It's almost like all the work you've put into silencing any criticism of the current regime and associated oligarchs was for nothing!

Isn't that just how oauth is done when a local app wants to be notified after login?

A huge amount of the web is only crawlable with a googlebot user-agent and specific source IPs.

> And given you-know-what, the battle to establish a new search crawler will be harder than ever. Crawlers are now presumed guilty of scraping for AI services until proven innocent.

I have always wondered but how does wayback machine work, is there no way that we can use wayback archive and then run a index on top of every wayback archive somehow?

You can read https://hackernoon.com/the-long-now-of-the-web-inside-the-in... it was a nice look into their infra structure. One could theoretically build it. A few things stand out:

1. IIUC depends a lot on "Save Page Now" democratization, which could work, but its not like a crawler.

2. In absence of alexa they depend quite heavily on common crawl, which is quite crazy because there literally is no other place to go. I don't think they can use google's syndicated API, cause they would then start showing ads in their database, which is garbage that would strain their tiny storage budget.

3. Minor from a software engineering perspective but important for survival of the company: since they are an artifact of record storage, to convert that to an index would need a good legal team to battle google to argue. They do that the DoJ's recent ruling in their favor.

I do not know a lot about this subject, but couldn’t you make a pretty decent index off of common crawl? It seems to me the bar is so low you wouldn’t have to have everything. Especially if your goal was not monetization with ads.

I think someone had commented on another thread about SerpAPI the other day that common crawl is quite small. It would be a start, I think the key to a good index people will use is freshness of the results. You need good recall for a search engine, precision tuning/re-ranking is not going to help otherwise.

Are these websites not serving public content? If there's some legal concerns just create a separate scraping LLC that fakes user agent and uses residential IPs or VPN or something. I can't imagine that the companies would follow through with some sort of lawsuit against a scraper that's trying to index their site to get them more visitors, if they allow GoogleBot.

Isnt that what SerpAPI was doing?

If a crawler offered enough money they could be allowed too. It's not like Google has exclusive crawling rights.

There is a logistics problem here - even if you had enough money to pay, how would you get in touch with every single site to even let them know you're happy to pay? It's not like site operators routinely scan their error logs to see your failed crawling attempts and your offer in the user-agent.

Even if they see it, it's a classic chicken & egg problem: it's not worth the time of the site operator to engage with your offer until your search engine popular enough to matter, but your search engine will never become popular enough to matter if it doesn't have a critical mass of sites to begin with.

Realistically you don't need every single site on board before you index becomes valuable. You can get in touch with sites via social media, email, discord, or even visiting them face to face.

You really do need every single site, as search is a long tail problem. All the interesting stuff is in the fringes, if you only have a few big sites you'll have a search engine of spam.

I think that is only needed for a small subset of queries. Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media. Ranking quality is much more important than coverage over the whole internet.

> Seriously think of the last time you did a search and went to a fringe site as opposed to a well known brand or social media.

Oh, almost never. That's exactly why search sucks now.

io is actually a country code tld for a territory that no longer exists.

I was aware of that, there was some controversy regarding that TLD a while back.

I consider it an exotic one though, just like .tv (which is also a country TLD) or .ai (not a country TLD).

Nobody is coming to help.

"All of the people I know who were friends with this sociopathic child-trafficking pedophile told me he was reformed now" is certainly something to put out there.