The problem isn't that Google doesn't employ these people or invest in their activities.
It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.
If Google:
- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).
- Allowed you to create your own custom search engines, like "Programmable Search Engine".
That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.
The biggest perverse incentive for Google is that making better search results can mean less clicks to ads (clicking an ad because results are crap, going thru more pages of results means more ads). Clicks are revenue which is much easier to optimise for.
Internal Search owners can push for better algos, but what if the algo causes revenue to fall? Are there strong forces strong enough within the organisation to ensure that search quality prevails?
If this is the case, the problem is existential. It can only be arrested at the very top
The biggest perverse incentive for Google is that making better search results can mean less clicks to ads
This gets close to the real root of the issue -- attention is monetizable independently of the quality of content. There would be much less incentive to create SEO spam if search engines negatively weighted pages with ads and affiliate links, and if manufacturers were barred (e.g. by the FTC) from owning or imitating reviewers.
> The biggest perverse incentive for Google is that making better search results can mean less clicks to ads
This is also something that Google can control if competitors come along.
i.e. If a reasonable competitor comes along that is willing to sacrifice ad revenue for better search result quality than Google, google can just adjust their search quality upwards to knock them out (and then adjust it back once the competitive threat is gone).
Perverse incentives from Google are all over the place - Searching for the delivery business "Just Eat" in the UK for instance returns an ad for their competitor Deliveroo above the legitimate organic search result for me - and I can also see that JustEat are trying to pay for their own brand name just to compete - and IMO this sort of behaviour is anti-competitive, borderline extortion considering Google is the de-facto way of searching for a business, and shocking from a search-quality perspective (where the wrong result is intentionally shown at the top because they paid more money).
If I had to pay $10/month for good search results, I absolutely would. I think most people would. Get rid of the ads and spam, and you have a service worth a premium. The solution is to make it user-centric instead of advertiser(spammer)-centric.
The custom search engine is harder than you'd think.
Google's search algorithm is tuned up for searching the whole web. It turns out the heuristics you need are very different depending on the size of the collection.
When Gerard Salton was doing IR experiments with punched cards he was working with collections of as little as 70 documents and in that case you are going to be very concerned about recall and not precision. Maybe there is 1 relevant document and if you miss it you failed.
If you had 70 billion documents you might have 10,000 relevant documents and if you lost 60% of them you still have 4,000 documents. The end user gets more results than they can sift through.
Thus I always groan when I see a site is using "Google Site Search" because the relevance is usually worse than you'd get with the alternatives.
Connected with that is the tuning work: Google has sufficient data to tune up a big model for everybody but true personalized search eludes them because they don't have enough data from you to tune up a model for you.
I agree with you that "true personalized search eludes them because they don't have enough data from you to tune up a model for you". That's what Larry Page said as well "Google doesn't know what you know". His ultimate goal is Answer Machine powered by AI but that's not happening anytime soon. I think internet search engines that we are using today are primitive compared to what we will have in the future.
The problem with all of this is it would help us greatly, but it would be useless to the 99% that the internet is increasingly being designed for. Modern UI trends are becoming obsessed with removing as many options and features as possible so the dumbest humans bordering on smartest vegetables can still use the service.
It does not if there are common interests and characteristics among users. Let's say for example I'm a young African-American girl who wants to learn how to code and I query "how can African-American girl learn coding?" and Google shows me Black Girls Code a non-profit organization that focuses on providing technology education for African-American girls. Considering that Google knows that I'm African-American girl and that I want to learn coding, how many other African-American girls want to learn coding? Probably many so caching doesn't break customization and personalization as long as Google knows my characteristics and interests and characteristics and interests of other people that are similar to mine.
It doesn’t have to actually, there are some pretty advanced caching mechanisms that allow you to combine cached elements together. For the web at least, you could do this back in the day with Server Side Include, and a place I worked at used it to cache logged in content.
> That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Why would they do this? Google's customers are the advertisers, not the end-users. And no one is going to pay for a search engine, it's been tried and has failed.
> And no one is going to pay for a search engine, it's been tried and has failed.
Always curious about things like this. I certainly would pay for this; it sounds like many other people here as well would. I'm curious if the constraint is that there aren't enough people to actually pay for the investment required for the service, or if there aren't enough people willing to pay to meet the standard VC notions of success. We seem to have a problem with building and supplying services for niche (read: "not expressable as an integer percent of the world's population") customer bases, and I'm never sure if that's a business problem or a cultural problem.
The people most able to pay for a service like this are the people that advertisers most want because they’re the people with enough discretional budget to spend on things like better Google search results. Allowing someone to buy something like this also reduces your attractiveness to your advertising clients.
I think you have to look at it more like Amazon Prime.
Nobody is going to pay for /just/ a search engine. But they might pay for, say, a /better/ search engine, plus additional features around gmail/gcal/gdrive.
Think of it more as subscribing "to google" and less as subscribing to "google search".
Regardless, the point isn't to "fix" google. It's to highlight a possible path for a new market entrant.
... If an existing player wanted to make a move here, I would say that both Mozilla and Apple are well positioned to add "personalized search" to a subscription service. Same with Microsoft. DDG could also make moves here if they expanded beyond search.
You only need ~1/10000th of Google's revenue to be a financially successful startup. 1/1000th and you'll have a great business, and at 1/100th you'll be somewhere between a unicorn and a decacorn.
I don’t know if that follows. Google has been maximizing revenue at the expense of the search result quality.
An “objectively superior” search engine, from an end user’s perspective, might have to make engineering choices that come at the expense of ad revenue.
But we’re all just talking hypothesis, it’d be cool to see someone launch a startup to get some answers.
if you think about it, Google provides advertisers a customized search engine to find customers. So it is not you searching the web, it is web's advertisers searching leads
there is a smaller niche market. SEMrush is a tool used in the digital marketing industry that is now public and has a multi-billion dollar market cap. It originally started as a search engine. When they didnt gain traction they used the tech to monitor Google and interface it for customers who are tracking their performance in search results (and much more).
How do you fight brigading, the organization of groups elsewhere to collectively vote on something? Eg white supremacist groups get together and vote down everything by people of color, and vote up their pages about how great they are?
Randomly select votes that are actually recorded. Then add in metavoting that votes on the votes with random sampling. At Google's scale with a sufficiently random sampling you'd be extremely hard pressed to successfully brigade or spam the voting.
Google could easily use its current fingerprinting to constrain (to an extent) multiple votes. Even knowing only a portion of the population will participate in the voting they can use a Wilson confidence interval[0] or similar to properly weight votes.
Random sampling works here since you're not guaranteed one vote per user per page and the outcome in binomial, seen and downvoted or seen and not downvoted.
easy, voting blocs, you assign yourself to the results of people who vote similarly to you. additionally there'd be local and regional blocs too. I can't think of a reason that the naive everyone sees everything everyone else is doing would work in the long run. That's Twitter, and it's garbage.
This is a great point. I would think Google could rank users from low quality to high quality in terms of the quality of the websites which they recommend or downvote. Tricky business and could be difficult to control, but basically the same thing they currently do for websites, but extended to humans.
> - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
They're going the OPPOSITE DIRECTION from this!! They recently removed all downvote visibility of YouTube videos from the user, so now downvotes only feed into their algorithm. So in the last line of defense of me ending up watching a shitty video, one of the most valuable tools has been removed by my betters. It's preposterous that people think that Google is doing a good job. They're actively getting worse, and ignoring everyone saying so.
They're doing a great job. I'm so happy dislike visibility have been removed. It removes the effectiveness of pile on coordinated harassment which many youtubers have fallen victim too.
Except the ratio is visible for the video creator, so they know nonetheless. And there's the possibility of disabling the the like/dislike for each video.
All the big channels I follow are mad at this change, and there's a coordinated effort to bring it back.
you mean the dislike counter they just disabled to force people to sit through more low quality content and pre-roll ads to claim increase in platform engagement and viewership?
The only thing matters is revenue and Google had increases in acquisition costs in prior revenue reports. Expect to see the data points for the latter metrics to be highlighted on the earnings announcement, and a record quarter for YT coming out of the change.
> Allowed you to block certain domains from your search results
I would love for Google to build this in. Until they do, there is a WebExtension that does this: https://addons.mozilla.org/en-US/firefox/addon/hohser/ ("Block or Highlight Search Engine Results"). I use it to block stuff like W3Schools so when I search for something, MDN is always #1. Saves me a lot of time having to add "MDN" to the end of every query.
Those shitty SEO spam sites exist only to serve ads, and Google has a monopoly on internet ads. So there is no real incentive for them to solve the problem.
Google has 28.9% share, Facebook 25.2%, and Amazon has 10% and growing fast. Not a monopoly, and the incentive is there: if search results are consistently bad, people will stop searching as much, and revenue and market share decline.
Real review sites serve ads too. I don't think Google has any incentive to make things worse, and they still want people to google reviews instead of just asking friends or people on reddit for reviews.
Things have changed in the past few years, now that Google has developed advanced transformer models, but for a long time Google's question answering facility has been: "let spammers make 10^8 pages where the title is the question and the answer is in the page".
The trouble is that there's a fine line between "answer is in the page" and "word salad!"
> - Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
Not convinced this would help. The spammers would just hire people to dislike competitors
> - Allowed you to block certain domains from your search results
This I would use. Never show me results form collider, watchmojo, ranker,
> - Allowed you to create your own custom search engines, like "Programmable Search Engine".
I think this would lead to people writing highly polarized engines. The Red Pill engine for example and we'd have a new problem, the proliferation of popular highly biased results. Of course that's not to say Google's results aren't already biased but they certainly are trying to cover everyone.
Not really that simple, I see a lot of potential for abuse - using bots and brigading to mass downvote your competitors or political opponents.
Couple of positions up or down in google results for somewhat popular and valuable keywords can mean the difference in thousands of dollars per day of ad or affiliate revenue. I suspect it would get pretty wild if google launched something like this. There already are black-hat seo methods and services, but something so simple and direct would turn it up to 11.
They already have the tech to fight this on YT. They, in theory, are supposed to be doing the same thing to detect inauthentic behavior on ad placement and click abuse.
It's that Google has destroyed their own search results in order to continue to expand their revenue opportunities.
If Google:
- Enabled downvoting on results, like YT videos. (Has its own spam problems, just like YT)
- Allowed you to block certain domains from your search results, like YT videos. (If they added some kind of "coordinated network detection" and down-ranked domains coordinating with ones you've blocked, that'd be pretty cool).
- Allowed you to create your own custom search engines, like "Programmable Search Engine".
That would be incredibly valuable. They already have most of the tech. They could even create a subscription service around custom search engines if they really wanted. Plenty of people would find something like that incredibly valuable.
Anyhow, buried in there is your startup idea. Remember: your startup doesn't have to generate the same revenue or profit as the incumbent on day one to be successful.