Google Studies How Search Behavior Changes When Searchers Are Faced with Difficult Questions
A paper by Google researchers Anne Aula, Rehan M. Khan and Zhiwei Guan published last month asks the question How does Search Behavior Change as Search Becomes More Difficult? (pdf)
The paper describes two studies in which participants were given informational tasks to perform – a mix of hard and easy questions – to see if searchers adopted different strategies for searching when they were faced with questions where there were definite answers where answers to those questions might be difficult to find. An example of one of the difficult tasks (can you find the answer?):
You once heard that the Dave Matthews Band owns a studio in Virginia but you don’t know the name of it. The studio is located outside of Charlottesville and it’s in the mountains. What is the name of the studio?
The first study had 23 people performing searches, finding answers to questions like the one above, and examining the searches they performed and the pages they visited to see how they went about finding answers. The second study expanded to 179 searchers, and based some of the processes used on things they learned from the first experiment. A general conclusion from the second study:
When having difficulty in finding information, users start to formulate more diverse queries, they use advanced operators more, and they spend a longer time on the search result page as compared to the successful tasks.
Why Study Difficult Search Tasks?
Why would a search engine want to know what kinds of user behavior might signal that a searcher is having difficulties finding the answers to questions?
The short answer is that if Google could identify situations where someone was having a hard time finding something, it might attempt to offer useful suggestions.
The authors of the paper provide a section on related research, and I’ve included a list at the bottom of this post of all of the citations used in the paper that were directly available online for anyone interested in exploring search behavior in more depth. I left out citations to papers that were only available through a subscription service.
The paper also provides a detailed look at the methods used in the two studies, and a look at how a couple of different searchers refined the queries that they used in an attempt to complete the informational tasks given to them.
Search Strategies Adopted in Difficult Tasks
The whole study is worth spending some time with, but I’m going to skip ahead to one of the sets of conclusions. We’re told that in unsuccessful tasks, where the searchers involved in the study couldn’t find the answers, the strategies they may have used included:
- users formulated more question queries,
- they used advanced operators more often,
- they spent a longer time on the results page (both on average, and when looking at the maximum time in the search session),
- they formulate the longest query somewhere in the middle of the search session (in successful tasks, they are more likely to have the longest query towards the end of the search session), and
- they spent a larger proportion of the task time on the search results page.
The researchers also noticed that the searchers in their study, compared to searchers in previous studies, used longer queries, had more queries per search session, and spend slightly longer times on search results pages.
A couple of other signals that they noticed when searchers where having difficulties with a task included people scrolling up and down the results or a landing page in a seemingly random manner and also revisiting pages that they had already seen in the same query session.
These types of signals might help the search engine uncover when a searcher may be frustrated with a search and either leave unhappy or be in need of some help.
Speaking of that, if you can find the name of the studio outside of Charlottesville that the Dave Matthews Band owns, I would appreciate the answer. I found myself exhibiting a lot of the same behavior listed above for people who were unable to complete some difficult informational tasks.
Search Behavior References
Many of the papers referred to in the study are pretty interesting, and I’ve seen many of them before. There were a few articles cited in the paper that I couldn’t locate, but here are the ones that I could find online:
- Query Formulation in Web Information Search (pdf)
- Eyetracking reveals the personal styles for search result evaluation (pdf)
- Understanding the relationship between searchers’ queries and information goals(pdf)
- Coverage, relevance, and ranking: the impact of query operators on web search engine results (pdf)
- Designing for expert information finding strategies (pdf)
- Eye-tracking analysis of user behavior in WWW search (pdf)
- Employing log metrics to evaluate search behavior and success: case study BBC search engine (pdf)
- Web search behavior of internet experts and newbies
- How are we searching the world wide web? A comparison of nine search engine transaction logs (pdf)
- Real life information retrieval: a study of user queries on the web (pdf)
- Real life, real users, and real needs: A study and analysis of user queries on the web (pdf)
- Patterns of information seeking on the web: a qualitative study of domain expertise and web expertise (pdf)
- Computers and iPhones and mobile phones, oh my! A logs-based comparison of search users on different devices (pdf)
- Differences between novice and experienced users in searching information on the World Wide Web (pdf)
- Cognitive strategies in web searching
- Investigating behavioral variability in web search (pdf)
- Investigating the querying and browsing behavior of advanced search engine users (pdf)