A New Search Results Evaluation Model from Google

Sharing is caring!

A Search Results Evaluation Model Whitepaper

Search Results (SERPS) are no longer about showing pages that are ordered by rankings for a query term. A Google paper shows us a different way of thinking about them in our age of structured Snippets and featured snippets mixed with URL search results, with a Search Results Evaluation Model. The paper is:

Incorporating Clicks, Attention and Satisfaction into a
Search Engine Result Page Evaluation Model
by Aleksandr Chuklin, Google Research Europe & University of Amsterdam and Maarten de Rijke, University of Amsterdam

Search engine results have gone through some significant changes over the past couple of years. A paper from the CIKM’16 conference in October 24-28, 2016, recently published on the Research at Google pages describes some of the user behavior that may take place around search results. The benefit that the paper brings us is that it describes:

In this paper we propose a model of user behavior on a SERP that jointly captures click behavior, user attention and satisfaction, the CAS model, and demonstrate that it gives more accurate predictions of user actions and self-reported satisfaction than existing models based on clicks alone.

Good Abandonment

Sometimes people search and expect to find answers to their questions in search results. In these days of featured snippets, that may happen more frequently than when answers might just happen to appear in snippets for pages featured in results. So, sometimes a set of SERPS can answer questions without any clicks.

This paper doesn’t describe the idea of entity metrics, but it reminds me of how the Google patent about them presented SERPS in a way that was different than just a presentation of URLs that were ordered based upon IR and PageRank scores. It is a slightly different set of things to think about when doing Search Results Evaluation. It’s recommended reading:

Structured Data & The SERPs: What Google’s Patents Tell Us About Ranking In Universal Search

There are different reasons why some pages or entities show up in search results than just a rank, and they can have value, satisfy searchers, and even lead to clicks that may educate or entertain.

Back seven years ago, I wrote about a paper from Google and Yahoo researchers that talked about satisfaction with search results titled, Evaluating the Relevancy of Search Results Based upon Position. That one focused more on the importance of where results were ranking in search results. There are possibly other things to consider these days.

Like the idea that a searcher might find an answer to their question and not have to click through to a page to find their answer, which can save them time and effort in finding pages pointed out by search results. That would be a case of “good abandonment.”

Click Models

This new paper also talks about user clicks being a sign of possible satisfaction. Very different from the kind of biometric satisfaction that I wrote about in, Satisfaction a Future Ranking Signal in Google Search Results?.

This paper instead talks about a Clicks, Attention, and Satisfaction (CAS) model. It provides some information about each of those things, in the context of human raters that might interact with search results.

The paper tells us about some of the crowdsourcing approaches used with human evaluators, including a glimpse at some questions asked to determine the relevance of results.

Takeaways

Crowdsourcing Direct Relevance

The Search Results Evaluation Model doesn’t tell you about how to show up in search results or rank higher, but it does tell us about how search results are changing and evolving. Search engines are trying to better understand when results might answer the questions posed by searchers in search results. They also tell us that they are using human evaluators to better understand how relevant search results might be and how satisfied searchers might be with the search engine.

Human Evaluators might start judging how well question-answering snippets satisfy people looking for answers to their questions, which is good to see.

Sharing is caring!

16 thoughts on “A New Search Results Evaluation Model from Google”

  1. Hi Bill,

    Thanks for the post. I was reading a patent a few days ago on Google scoring websites by user clicks. For example a user can give a website up to 20 points in a single week and 100 points in a single month. It talked about giving SEO’s a penalty if they tried to use proxies etc to try and get false points.

    The downside was the patent did not talk in detail about how they worked out the clicks. I was hoping they were going to mention ctr or long clicks (seo terms). But it did not, hopefully this patent fills in the gaps about how Google uses clicks.

    So I look forward to reading this paper.

  2. Hi Bob,

    You will have to let me know how you felt about the “Click, Attention, and Satisfaction” model that this paper describes. It does appear that evaluation is being done by people hired to evaluate sites, so SEOs trying to game this system doesn’t seem like a potential issue.

  3. I always read your article. It was interesting to read and also informative. Thanks for your blog.

  4. Hi Bill.
    Another excellent post (and find) yet again.
    This paper backs up what I’ve been thinking in terms of search (pardon the pun!)

    You’ve quoted the following.
    “This paper we propose a model of user behavior on a SERP that jointly captures click behavior, user attention and satisfaction, the CAS model, and demonstrate that it gives more accurate predictions of user actions and self-reported satisfaction than existing models based on clicks alone.”
    A lot of the results that I’ve been seeing lately has reflected this. Most SEOs are calling it content marketing and improved user experience, but this is exactly what’s needed and I’m glad you’ve shared the data to back this up.
    Thank you

  5. Hi David,

    Thank you. I was excited to find this whitepaper from Google, because it does a nice job of explaining how they are evaluating search results that answer questions and how satisfied searchers might be with those results. It was good to see, I do like it when I manage to come across something like this. 🙂

  6. This is a really interesting.

    Am I right in assuming as this is a ‘proposed method’ at present ‘good abandonment’ is not taken into account in Google’s current search algorithm?

  7. Hi Stephen,

    I’m not sure it is safe to assume too much of anything these days. It’s possible that Google is using this evaluation method; but they might not be either. It looks like it would make sense for them to use human raters to use an evaluation method like the one described to check up on and review search results, especially if they want to check up upon featured snippets.

  8. Bill Slawski,

    A debt of gratitude is in order for the post. I was perusing a patent a couple days prior on Google scoring sites by client clicks. For instance a client can surrender a site to 20 focuses in a solitary week and 100 focuses in a solitary month. It discussed giving SEO’s a punishment in the event that they attempted to utilize intermediaries and so on to attempt and get false focuses.

  9. Hi Bill,
    superb article with informational stuff like click behavior, user attention and satisfaction, the CAS model, I really like it thanks for sharing.

Comments are closed.