The Noisy Channel recently had a post linking to this article on how Google measures their search quality.
When Google tries new ranking models, these metrics sometimes move, sometimes not, and never by much. In fact Google does not use such real usage data to tune their search ranking algorithm. What they really use is a blast from the past. They employ armies of “raters” who rate search results for randomly selected “panels” of queries using different ranking algorithms. These manual ratings form the gold-standard against which ranking algorithms are measured — and eventually released into service.
Who knew? I suppose it makes sense. Yahoo! started out as human-managed lists of “quality” links, and the human brain is pretty sweet. It’s nice that the people can still sometimes kick machines in the butt.
