I think all the great minds at google have a better chance than I do. Plus you were the one saying how unfair it would be for them to not invade people’s privacy in order to cash in. My point is that if they can’t operate without violating our human rights then they shouldn’t operate.
Yep. People don’t pay for search anymore. They haven’t since Google took over the world a decade and a half ago. That’s not to say free is the best option, but people don’t want to pay when there’s a free solution that works just as well.
And when effectively no users opt in to advertising, how does the search engine stay in business?
A few years ago my response to this would be that I'd gladly pay Google $10/year for access to Search without adverts, but over the past couple of years that's changed. Google have become so utterly terrible at handling customers that I no longer would. I'll only use Google services if they're free. The idea that Google might just kill my account without notice leaving me with no access to search would be catastrophic.
I would gladly pay a different search engine company for access to a good search service though.
Do you have exact numbers? The estimates I've seen are about $60 per user per annum, so about $5 per month. That's for all services, not just search.
So for less than a Netflix subscription you have Youtube, Gmail, Search, Drive, Translate, Calendar, Maps, Photos, Hangouts and more. Services that many people use regularly.
It wouldn't be hard to argue that any of these products is worth a lot more than $10 a year, but convincing users at large to go down that route will be a challenge, undoubtedly.
It is less than a Netflix subscription but essentially everyone on the planet subscribes and can't cancel.
(Also, many users would agree on the value but balk at redirecting funds from an existing allocation to continue service on something they've already gotten for free.)
The ad industry's biggest accomplishment is making most users believe that they can have access to state of the art search engine, video streaming, messenging etc... for "free". It means that it's incredibly hard to compete with a different business model.
Yes I would. Based on the really, really crappy results that google gaves me out lately (I have actually quit using it), $12 a year seems more than reasonable (you have market of millions of users, charging $10 / month is ripoff). Under condition that it is not some ads company under disguise (like google), that they eliminate all SEO crap (i dont want those results anywhere near the 100th page of results) and ads from my results and serve me relevant content.
Instead of google I am aggregating multiple search engines and joining the results eliminating all that are not on most of them. I am losing anything special (like there is something "special" today) but at least I dont get bunch of stuff on google that just wastes my time.
Nextgrid: yes, exactly that. I want old results, I want old content, I want content that matches with my search words 100% instead of getting what google crappy algorithms think that I want. I dont want hipster crap that propagates next silver bullet (that is just a copy what we already had packed into SaaS).
> Based on the really, really crappy results that google gaves me out lately
People on Hacker News seem to think Google's results have gotten worse over time. Maybe they have for the narrow set of interests and requirements of this crowd, but I'm very confident they've gotten better for the vast majority of other users (users who I bet are less likely to use an ad blocker too).
> That they eliminate all SEO crap (i dont want those results anywhere near the 100th page of results) and ads from my results and serve me relevant content.
This is an extremely difficult engineering problem. Google doesn't make a conscious decision to include low/middling quality SEO'd content. That content is optimized to appear relevant and high quality. Google does a better job than say Bing at telling the difference, but it's still obviously an unsolved problem.
It's a good question. I'd flip it around a bit and say that if you're a search engine that:
- Hosts commercial advertising content (not the entire web; only product and service offerings)
- Charges for hosting
- Has a highly qualified audience that genuinely wants to receive relevant advertising (based on requests being user-initiated, not software-initiated)
- Claims credit for revenue generated as a result of providing relevant advertising
... then you're likely to be able to cover the costs of developing and maintaining the system.
- This is a bit like eBay or Amazon, arguably; except that the user profile and context is optional for the user to provide
- Addictive products could still be a loophole that bad advertisers could exploit to monetize users without providing them overall benefit; fortunately local regulations tend to cover that
I'm not sure splitting off the crawler would have the effect you want it to. Crawling is hard, but actually one of the easier parts of building a search engine. There are less complete, but still pretty impressive public crawls available already, like Common Crawl.
The harder problem is what you do with the result of the crawl once you have it. You need to index it. This is one of Google's deepest moats. Being able to serve any of hundreds of billions of documents in milliseconds requires an enormous amount of infrastructure and hundreds of careers worth of difficult software engineering.
I don't see a good way to split indexing out as a separate company from the rest of the search engine. It's tightly integrated with ranking. Generally a new ranking technique requires some new information to be associated with each document in the index, e.g. some precomputed scores that describe the document's quality or fitness for different kinds of queries. At query time, you're often just weighting and comparing these scores across documents.
If you had Google's index, you'd be a long way towards being able to replicate Google's results, but I don't think that would lead to the kind of innovation you're hoping for.
Not even close. I've used Common Crawl, it has nowhere near the breadth that Google has in their index.
Problem no. 1 is that website owners block non Google crawlers because of limited bandwidth. So you can't reliably even get started on creating your own search engine. Once we have the data available, we can figure out the other bits and what they require. But to get to that door, you need access to the crawl.
You don't need to have everything in the Google index to beat Google.
You need to be better than Google in the search niche that you'll serve.
And for that, at the bare minimum, you need access to all the crawl data that Google has to get started, then throw away the bits that you don't need.
Google's crawler gets preferential treatment on many sites. Not necessarily because of private deals, but because it drives traffic from search results.
Quote via [2]
--- start quote ---
When Mr. Maril started researching how sites treated Google’s crawler, he downloaded 17 million so-called robots.txt files — essentially rules of the road posted by nearly every website laying out where crawlers can go — and found many examples where Google had greater access than competitors.
ScienceDirect, a site for peer-reviewed papers, permits only Google’s crawler to have access to links containing PDF documents. Only Google’s computers get access to listings on PBS Kids. On Alibaba.com, the U.S. site of the Chinese e-commerce giant Alibaba, only Google’s crawler is given access to pages that list products.
So it sounds like the regulators should focus on forcing every web property to give equal access to all crawlers. Why steal Google's investment and hard work and innovation?
>I'm not sure splitting off the crawler would have the effect you want it to. Crawling is hard, but actually one of the easier parts of building a search engine. There are less complete, but still pretty impressive public crawls available already, like Common Crawl.
When they saw crawler they don't mean Googlebot or something.
They mean the whole thing you've described, the search engine as a backend.
What the law proposes is that other companies should be able to just buy the search engine service from a new company split from Google, of which Google which will just be just another client.
>I don't see a good way to split indexing out as a separate company from the rest of the search engine. It's tightly integrated with ranking.
The idea is to split the whole search part from Google. Let them have the ad business, Android and so on, and just rent the search from the spin-off company...
There is so much irony in the fact that the web loves Google crawler, optimizes for it, makes sure that Google robots have special, first class treatment.
And at the same time, on the same website I need to enter Google-provided captcha. To fight robots.
Well, I wasn't taking the OP's idea to mean only splitting off the crawler. Why not an API where you send it the search query and then it returns the results? It could have a variety of return options including using a ranking of the API publisher's choosing or you could ask for something raw, and innovate with an even better or more suitable ranking based on your specific needs.
In the end, the point would be to make Google closer to what it was at the beginning -- just input and output. No ads on top of it, no bundling with other offerings etc. And if the publisher's ranking remains as good as it should, then surely some will be willing to pay not only for the raw data, but the ranking as well.
This is exactly how I envisioned it. A crawl+index company that gives equal access to all competitors.
The search front companies would use the index to retrieve pages, then they would rank these pages with their own criteria and filter with their own white/black lists, then present the result in their own UI. Each one would have their own ads and tracking (or lack thereof). The index would not be able to know the identities of the users.
In the end they all boil down to an "attention economy", it's about us and our eyeballs, we should have more say on what we are consuming.
The part about ranking, filtering, tracking and UI is true for FaceBook and Twitter as well, even though they are a different kind of service. We could have alternative views on these systems, where we have more control on the way the system manipulates the information for us.
One reason to care which is bigger is the effect they each will have on the norms of governance around the world. Superpowers tend to nudge the rest of the world's governments to be more like them (see Britain and the US over the last 300 years). Do you want the rest of the world to be more like China or more like the West?
China has shown the world what it considers to be acceptable norms of governance: unrepentant, ongoing genocide against the Uigers, snuffing out liberal democracy in Hong Kong, and denying the sovereignty of a liberal neighbor in Taiwan.
I'm fully expecting plenty of whataboutism and false equivalencies in response to this comment and others making a similar point.
Believe me, I'm disappointed in the US and the rest of the West's recent slide away from liberal values, and I hope we can start doing better soon, but I encourage onlookers to ask themselves: do you really believe the there's no meaningful difference between China's vision for the future and the West's? No difference between the future each will create for you?
>unrepentant, ongoing genocide against the Uigers, snuffing out liberal democracy in Hong Kong, and denying the sovereignty of a liberal neighbor in Taiwan.
You're quoting Britain there as someone to look to, and at the same time mentioning genocide...
> I'm fully expecting plenty of whataboutism and false equivalencies in response to this comment and others making a similar point.
Please don't poison the well. If you believe I've made such an argument, please politely point that out about a specific statement and I'll try to understand my biases and correct it.
> do you really believe the there's no meaningful difference between China's vision for the future and the West's?
If we were comparing to Mao, Stalin, or Hitler, then, yes; but, modern states, no.
The U.S. imprisons millions of people for petty drug crimes and kills millions of people in imperial wars. Is that so different than the treatment of the Uigers, Taiwanese, or those in Hong Kong?
The social credit system and Great Firewall are being implemented through different means: social ostracism and social media oligopolies. I've personally experienced it.
> No difference between the future each will create for you?
I'm actually generally optimistic. If we take the evidence of the past few hundred years, we are likely to innovate and grow despite such excessive governance. It's the innovators that will make my future better. As far as the government goes, the practical difference seems minimal: I will meticulously pay my taxes and follow the laws so as not to receive the glare of either of the Saurons.
> The U.S. imprisons millions of people for petty drug crimes and kills millions of people in imperial wars. Is that so different than the treatment of the Uigers, Taiwanese, or those in Hong Kong?
Yes, because in the US, we can actually have a conversation about changing those policies. In China, they can't.
Although there are counter-examples of Snowden, Assange, and others, I do generally agree with you that there are fewer political crimes in the U.S.; however, words are cheap, it's actions that matter. My point is about the results: the government continues to grow in depth and breadth. So, yes, if you're on the socialist end of the spectrum, you may feel very free to accomplish your goals. Those on the other end do not.
As an example, how did all of our free speech help in stopping the war in Iraq based on a lie about WMDs that killed millions of people? Is anyone even arguing that China has killed millions of Uyghurs?
To be clear, I think both sides are wrong. I'm just saying they're not qualitatively different by a magnitude.
Many games still aren't available on Mac, and things like Wine are not nearly as clean of an experience as just installing games in native Windows. I'd guess switching to ARM will set back the progress made in Mac gaming as well.
It runs a machine learning model in your browser to convert the text into points in a high dimensional space, and then it projects those points down to 3D.
Right now you can tell it to visualize post titles or comments from any subreddit or tweets from any Twitter user. I find it especially interesting to explore the news with it since every article is naturally presented alongside other articles that are about similar topics, often giving useful context.
Only works in desktop Chrome right now unfortunately. I was hoping to make it Firefox-first, but I need an API Firefox hasn't implemented yet.
Since the model runs on your machine, the "running model" stage will be slower or faster depending on your local GPU. If you have a decent GPU, I recommend bumping up to 512 points.
Would love any feedback! Here, on Twitter (@gradientassent), or in the Discord channel linked inside.
I'm working on a project right now where the centerpiece is a Three.js scene. It's about data visualization, so the surrounding code covers much of the same ground as a typical single-page web app: fetching data, UI for manipulating the dataset, etc.
I decided not to use my beloved Clojure and re-frame because it seemed like I was going to end up wrestling with Three's imperative API.
Others inclined toward projects like the OP and the Clojure ecosystem more generally: how have you approached similar projects before? Where there's a very non-Clojure-native library that will be critical?
I ended up with TypeScript, React, and Redux instead of re-frame.
> how have you approached similar projects before? Where there's a very non-Clojure-native library that will be critical?
Started playing around with Arcadia which is Clojure-CLR + Unity3D tooling/library. Basically used for make some interactions with the Unity3D framework easier.
While most of the code is using interop, having the repl flow is something that for me, after using Clojure professionally for a while, is really really hard to give up. I've tried getting into game development before, but the slow iteration cycles on "make change -> save -> do thing to test" just sucks all the fun out of it.
If I were to use Three.js in a project, I'd approach it the same way. As I try to stay far away from any other languages, I'd write a small layer between my CLJS code and the Three.js API, the smaller the better, and then use that.
Because giving up on being able to eval whatever and having access to a repl is not worth it in the projects I work with.
Basically, any hurdle (like basing my project around a non-CLJS library) is easier to overcome than having to deal with something like TypeScript again.
It does actually cost money to operate a good search engine.