I think somewhere along the way work on search just devolved into handling millions of queries at a time. It could have been so much better.
If you ask a "search expert" today what's he is trying to fix, he will say something related to scaling.
If you asked an expert in the 80's or 90's they would talk about query complexity and NLP i.e. Who were the four semi-finalists of last years Wimbledon? And you would get back 4 names.
Today you will get back a result saying 50 million pages were found in 1 second. The page may or may not contain the 4 names if you wade through 17 popups. Nobody questions how brainless this is.
Well, I hear a lot of people complaining that the results on DuckDuckGo are still worse than on Google, even though both search-engines produce results within a second. And these are people that really want to quit using Google for privacy reasons. I never hear people complaining that a search is slow. So I do think that search-quality is where the competition is happening.
Edit: But, I agree, we don't often see any good HN posts or papers about search-quality. Perhaps a case of trade secrets?
The quality/ranking code and algorithms are definitely one of the "holy grails" at Google. That code and documentation is locked down tight, compared to the 99.9% of the rest of the codebase that's open to any engineer.
You don't hear people complaining about speed, because there's constant optimization around that. If searches start taking 2-3 seconds with better search quality. The service might be much more useful, but it will feel very much worse. And how people feel is how they judge you.
Actually quite the opposite. While we do make sure performance is good, the bulk of our time is spent dealing with relevance improvements, content wrangling, language pipelines, and lots of other cool stuff.
FWIW I do see organizations building relevance/discovery teams staffed differently than their team for "scaling out a search engine". Usually the relevance team is tasked with anything in the product that matches users to content, whether it be search, recommendations, or other 'discovery' themed functionality.
If you ask a "search expert" today what's he is trying to fix, he will say something related to scaling.
If you asked an expert in the 80's or 90's they would talk about query complexity and NLP i.e. Who were the four semi-finalists of last years Wimbledon? And you would get back 4 names.
Today you will get back a result saying 50 million pages were found in 1 second. The page may or may not contain the 4 names if you wade through 17 popups. Nobody questions how brainless this is.