Matt Cutts – Google’s famed Head of Search Spam – post a video yesterday which answered how the internet giant evaluates which new search algorithms they use and which they throw away or adapt.

The question was posed by James Foster of Sydney, Australia who asked:

“What are some of the metrics that Google uses to evaluate whether one iteration of the ranking algorithm is delivering better quality results to users than another?”

Matt Cutts uses about three steps to break down the evaluation process:

(1) Cutts says that Google tests the new algorithms offline first – benchmarking how the results rank alongside the new algorithms and if the URLs (websites) are higher quality than what they were when the previous algorithms in place. This ‘quality’ is based on how the search quality ‘raters’ rate the URLs in previous instances. If the URLs were unrated, Google may ask these raters to rate the new URLs or compare the old search results to this new test set. Based off of those metrics, Google can decide whether to move onto the next phase or not.

(2) Now for the live tests. Google samples a subset of real live searchers and gives them the new results with the new set of test algorithms. If these new search results yields a higher click through rate, it could infer that the new results are better than the older ones. However, this isn’t always the case – especially with web spam – Cutts says. In general, the more clicks a specific search results page has, the better quality the results.

(3) Now it’s  decision time. Google’s Search Quality Launch Committee has the final say on whether the new algorithm goes live to the public or not.

They have a lot of sub-departments don’t they?

Matt Cutts says that Google has the process down to a “pretty good system” but every so often they need to refine, remove, change an adapt these process within their workflow. What we still don’t know is how often Google are doing this, and at what quantity? Size, also; how big can these projects get? Pretty big we assume. And how small? Are they implemented small changes every day? Week?

As always with Google and SEO in general – there’s still a lot of mystery!