A paper by Google researchers Anne Aula, Rehan M. Khan, and Zhiwei Guan published last month asks the question How does Search Behavior Change as Search Becomes More Difficult? (pdf)
The paper describes two studies in which participants were given informational tasks to perform – a mix of hard and easy questions – to see if searchers adopted different strategies for searching when they were faced with questions where there were definite answers where answers to those questions might be difficult to find. An example of one of the difficult tasks (can you find the answer?):
You once heard that the Dave Matthews Band owns a studio in Virginia but you don’t know the name of it. The studio is located outside of Charlottesville and it’s in the mountains. What is the name of the studio?
The first study had 23 people performing searches, finding answers to questions like the one above, and examining the searches they performed and the pages they visited to see how they went about finding answers. The second study expanded to 179 searchers and based some of the processes used on things they learned from the first experiment. A general conclusion from the second study:
When having difficulty in finding information, users start to formulate more diverse queries, they use advanced operators more, and they spend a long time on the search result page as compared to the successful tasks.
Why Study Difficult Search Tasks?
Why would a search engine want to know what kinds of user behavior might signal that a searcher is having difficulties finding the answers to questions?
The short answer is that if Google could identify situations where someone was having a hard time finding something, it might attempt to offer useful suggestions.
The authors of the paper provide a section on related research, and I’ve included a list at the bottom of this post of all of the citations used in the paper that were directly available online for anyone interested in exploring search behavior in more depth. I left out citations to papers that were only available through a subscription service.
The paper also provides a detailed look at the methods used in the two studies, and a look at how a couple of different searchers refined the queries that they used in an attempt to complete the informational tasks given to them.
Search Strategies Adopted in Difficult Tasks
The whole study is worth spending some time with, but I’m going to skip ahead to one of the sets of conclusions. We’re told that in unsuccessful tasks, where the searchers involved in the study couldn’t find the answers, the strategies they may have used include:
- users formulated more question queries,
- they used advanced operators more often,
- they spent a long time on the results page (both on average, and when looking at the maximum time in the search session),
- they formulate the longest query somewhere in the middle of the search session (in successful tasks, they are more likely to have the longest query towards the end of the search session), and
- they spent a larger proportion of the task time on the search results page.
The researchers also noticed that the searchers in their study, compared to searchers in previous studies, used longer queries, had more queries per search session, and spend slightly longer times on search results pages.
They also note that people seem to be using less advanced search operators (more here) in their queries than previously reported.
A couple of other signals that they noticed when searchers were having difficulties with a task included people scrolling up and down the results or a landing page in a seemingly random manner and also revisiting pages that they had already seen in the same query session.
These types of signals might help the search engine uncover when a searcher may be frustrated with a search and either leave unhappy or require some help.
Speaking of that, if you can find the name of the studio outside of Charlottesville that the Dave Matthews Band owns, I would appreciate the answer. I found myself exhibiting a lot of the same behavior listed above for people who were unable to complete some difficult informational tasks.
Search Behavior References
Many of the papers referred to in the study are pretty interesting, and I’ve seen many of them before. There were a few articles cited in the paper that I couldn’t locate, but here are the ones that I could find online:
- Query Formulation in Web Information Search (pdf)
- Eyetracking reveals the personal styles for search result evaluation (pdf)
- Understanding the relationship between searchers’ queries and information goals(pdf)
- Coverage, relevance, and ranking: the impact of query operators on web search engine results (pdf)
- Designing for expert information finding strategies (pdf)
- Eye-tracking analysis of user behavior in WWW search (pdf)
- Employing log metrics to evaluate search behavior and success: case study BBC search engine (pdf)
- Web search behavior of internet experts and newbies
- How are we searching the world wide web? A comparison of nine search engine transaction logs (pdf)
- Real life information retrieval: a study of user queries on the web (pdf)
- Real life, real users, and real needs: A study and analysis of user queries on the web (pdf)
- Patterns of information seeking on the web: a qualitative study of domain expertise and web expertise (pdf)
- Computers and iPhones and mobile phones, oh my! A logs-based comparison of search users on different devices (pdf)
- Differences between novice and experienced users in searching information on the World Wide Web (pdf)
- Cognitive strategies in web searching
- Investigating behavioral variability in web search (pdf)
- Investigating the querying and browsing behavior of advanced search engine users (pdf)
ahhh this is a great read. I love posts with search research papers. You can only read so many blog posts about title tags, links and duplicate content until you go crazy. Good to read some really insightful stuff.
Thank you for the papers that will keep me reading. I was experimenting with map based search for homes this weekend. I found that this is a great method for mobile, but due to the icons on the map, you had to be careful where you punched and map section you zoomed in on to obtain the most relevant information. I wonder how many people perform a search, and become convinced that they have obtained the result, yet may have incomplete data.
Found it.
Here are the searches I performed, in order:
Virginia Recording Studio
dave matthews cue recording
Chris Kress
dave matthews recording
The Lillywhite Sessions
The Lillywhite Sessions, studio
Dave Matthews Stand Up
Haunted Hollow Studio
And, over at Bing Maps:
Charlottesville, nc
I started by taking a look at the mountains by Charlottesville, to help me weed out any studios that were to far away. Then I looked for studios in the area, and checked some of those websites for the name Dave Matthews. I thought I had a lead in “Cue Recording Studios, and again in “Chris Kress”. That was all a huge waste of time.
I then went to Dave Matthew’s wikipedia page, did a ctrl-f for “Studio,” and found the story behind the studio but not the name. The story mentioned “The Lillywhite Sessions,” which was a dead end, as it was never an official album with credits. I re-read the Wikipedia entry, found out that the album “Stand Up” was recorded at this nameless studio, and read the album credits, also at Wikipedia. There it was:
Haunted Hollow Studio.
A great question from Google. I’d ask for more cases to solve, but I’ve wasted enough time already.
I would have liked to try and find the answer but Tielman already did. This will really be useful for students as well as any other researchers online. It is comforting to hear that Google is helping search engine users to find answers when it’s getting difficult.
Interesting post, I’ll have to come back to read some of the other papers when I have more time. I wonder what the demographics were of the people doing the test, as I’m sure an IT savvy, employed, 18 year old, American male would search very differently to a 70 year old, retired woman from Jamaica.
And thus Google begins its work trying to remap the human genome through search patterns. Sometimes their cleverness makes me sick with joy.
Hi Bill, thank you for sharing this very interesting topic. I have to agree with Steve, the methods used to solve a certain query will depend on who is performing the search.
@Andrew, yeah I always have this weird “comic book” fantasy that Google is slowly evolving and replicating how a human thinks and speaks. Let’s pray that Google wont change their name to Cyberdyne. 🙂
Interesting research. I guess it just goes to show how different people search for different things.
Although I do wonder how they’re going to implement this research work, I’m almost scared in fact.
Interesting info for sure. It’s cool to see how our search behavior changes as things get harder to find. It’s cool how Google can help us so much. I wonder how far this type of research will help itself. To find out our own general behaviors to enhance our searching experience. Overall a great thought provoker! thanks for the article!
Hi Tielman,
Thanks for finding the answer to that question, and for providing the details of how you found it. It is a relatively hard information task (and time consuming). I gave myself 10 minutes to find the answer, and didn’t get quite as far as you did (perhaps a little too much in a hurry to publish this post).
Hi Chris,
Thanks. Granted patents tend to be long in the tooth, often filed three or more years ago. Pending patents are usually not published until at least 14 months after they are filed. Both can be useful for giving use ideas and insights into what the search engines may have been doing, but search white papers more often give us a hint of something to come.
Hi Frank,
My post today included a large number of papers published in the last couple of years as well.
Interesting question on searches that provide incomplete data. I imagine that there are a fair amount of those, as well as results that provide incorrect data.
Then again, I remember when I was younger, going to the Library, and finding the book in the card catalog that I was positive had the answer to the question I was trying to answer, only to find the book checked out by someone. 🙂
Hi Andrew,
I agree with you.
A number of the papers that I linked to above try to understand the strategies used by people who are “expert searcher” to come up with ideas that can be used by people with less expertise at searching. This paper turned that idea on its side by exploring what “expert” searchers do when they are faced with difficult informational tasks. Somewhat like reference librarians might do when someone comes to them for help.
Hi Steve,
Interesting question. The paper did provide a very limited amount of demographic information about the people involved in the second test. The 179 users ranged from 18 – 54 years old. I would have liked to learn more about them as well.
Hi Andrew,
Is Google attempting to reshape us, or is it evolving because of the way we use it, or some combination of both? It looks like more and more like what they do involves attempting to understand how people use the search engine and the Web.
Hi Vic,
It’s true that the methods used may depend upon the searcher, but I think part of the idea behind the experiment was to identify different search strategies rather than trends in how different people search, so some diversity of backgrounds and education levels and experience may have been a good thing rather than something harmful.
Hi Tola,
One approach the search engine may take might be just to offer more query suggestions to a searcher when it seems like they might be having trouble finding an answer. Another might be to display search results differently, such as by categories, like Bing sometimes does.
Regardless, the search engines do collect a lot of data about how we search for things, and it’s available for them to view in the search engines’ log files. Chances are that they have a lot more data than they have any idea what to do with.
Hi Meesan,
Thank you. Search definitely should be as easy as possible, but sometimes it can be difficult to find the answers to some questions regardless of how much expertise you might have in searching.
That problem isn’t always caused by the search engines, but rather by a lack of information on a topic or task that we might be interested in. Regardless of that, I would say that we often likely blame the search engine that we are using when we can’t find the answer to something, rather than the pages that could provide us with that information but don’t.
Super collection of search behaviour references Bill, thanks for digging these up and sharing with everyone.
I always revert to advanced search operators on search engines when struggling to locate the information I am after, 9 times out of 10 I am successful in locating however many non tech savvy users would not even be aware of these.
Thank you, Geoff
How people search, and how they try to find things on the Web, whether through a search engine or elsewhere is fascinating.
There are a lot of papers on the topic from academics and from the search engines, and I appreciate that they’ve shared some of their experiments and findings with us.
I tend to use the minus search operator in a lot of my searches when looking for resources within search results, to filter out things that I don’t want that I see in those search results.
That’s an interesting study and makes me think of how I search for information when I’m trying to look up something that can’t be summed up in a few words. It’s kind of a fun experiment to try, because I’ve pulled up some off-the-wall information sometimes. I don’t let my kids type in things unless they know exactly what they’re looking for because they’ve pulled up a few adult sites that go straight to the pictures!
A Google title search with two Boolean expressions worked and reduced the number of results. intitle:”dave matthews band” AND studio AND charlottesville located the CBS News site with the snippet:
The Dave Matthews Band Opens Up – CBS Sunday Morning – CBS NewsMay 29, 2009 … The band retreated to Haunted Hollow, house on 140 acres in the Charlottesville countryside that they converted into a studio.
Thanks for the links to the very relevant and useful articles.
Great post I love these kind of studies. It really is amazing the lengths some people end up going to just to find something online. I gave up on finding the studio name, was taking to long.
Nice piece! I did not view any replies before stopping where you wrote “if you can find the name…” and took a shot at the challenge. Congrats to Tielman and any others who managed to track it down. For me it took around 5 minutes so it took longer to document the path taken than the actual search.
Searches:
1. “Dave Matthews Band”+studio owned
2. “haunted hollow”
3. haunted hollow recording studio
Answer: The 3rd result of the first seach leads to confirmation of the answer using this search phrase: “Dave Matthews Band”+studio owned
Comment #5 in this post on a forum http://www.gearslutz.com/board/so-much-gear-so-little-time/129035-dave-matthews-band-studio-schedule-productivity.html names the studio: Haunted Hollow with a link to pictures: http://www.antsmarching.org/images/news/studio/studiopictures.php.
Continue for independent confirmation and Google the studio name: “haunted hollow” in quote marks and the Google search dropdown offers a long list of options, so I skipped “haunted hollow” and chose: haunted hollow recording studio.
The first result is to the architect who did the redesign of the building. They give the project name as Haunted Hollow Recording Studio. They do not name the band only mentioning “Haunted Hollow studio is a state of the art recording facility and social gathering place for a very successful local band.” except there’s a strong clue in the page name: http://cdadesignstudio.com/CDA%20DMB%201.html with the initials DMB.
The second result is the general contractor overseeing the actual work that does connect and name the band. The second result to this link: http://www.martinhorn.com/projectsSpecial3.html provides evidence that links the band and names the studio.
Hi Andrea,
When you don’t know much about a topic that you’re trying to find information about, it can somethings be really hard to come up with a good query that might get you close to the answer. And there are some potential traps out there for the unwary.
Hi Paul,
Thanks. I appreciate that people are sharing their approaches to finding the answer to the Dave Matthews Band Question. I didn’t use the “intitle” operator, but it looks like it was helpful in your search. The CBS snippet provides an answer without even going to the page it describes.
Hi Iain,
It’s fun to get a little inside the heads of people from the search engines who conduct experiments like this one, and get a sense of how they conduct an experiment and analyze the results. I really enjoy these kinds of studies as well.
Hi Jim,
Thanks very much for sharing how you find the results of the Dave Matthews question as well. It is interesting to see how many different approaches people took to finding the answer. I like finding second sources like you do, and I’m glad that you did. It gave us a chance to actually see some pictures of the studio – looks like a great place to record an album.
It’s amazing to see how these search engines worked. I remember when I first started getting into the internet people would write their queries like “What is the name Dave Matthews Band’s studio?” rather than using keywords like “Dave Mathews Band studio, Charlottesville, Virginia, mountains”. But now it seems that search engines are getting wise to those who prefer to write out the questions, and are able to provide adequate results and suggestions. This will be revolutionary for students and other researchers in the future. Thanks for the great article!
Hi John,
Thanks. Some search engines even advised you to search in question like formats, such as askjeeves. I really like it when the search engines provide us with a window into the kinds of search behaviors that they are seeing.
Great article, Bill. I just happened across this, scrolling through your past posts, and found it very interesting. It’s intriguing to try to read between the lines, and figure out what the SEs are thinking/planning. Rarely productive, but still, fun.
Like Jim, as soon as I saw your last line, I set out to find it.
First, in the Google search engine, I entered (without quotes) “Dave Matthews Band + studio + mountains”. The #3 result was for the DMB website, but I found nothing there.
I then added “+ charlottesville” to the end of the previous search string. The #1 result took me to foodandwine.com, where I found mention of the studio, but without a name.
Finally, I modified my search to “Dave Matthews Band + studio + Virginia”, again drawing a blank. After that, I looked the band up on its Facebook page, and came up dry once more.
At that point, I had about ten minutes invested, and gave it up, as a possible hoax. Coming back here, I found Tielman’s comment, cursed, and left this.
Enlightening piece, and interesting exercise, Bill!
Hi Doc,
Thank you. I really like seeing this kind of whitepaper from one of the search engines, casting a critical analysis at how people search, and how effective that search engine might be in helping people with their searches. In this particular instance, trying to understand the strategies that people use in searches, and how those might fail may provide Google with some strategies of their own in the future. I did find myself wondering when reading Google’s paper, “Why are they sharing this with us?”
I gave up after around 10 minutes, too. I was surprised how quickly Tielman found the answer.
Human behavioral research means better results and a better algo. Love the deep info on customer behavior. Sweet post and insightful as well as thought provoking. If Google is doing conversion analysis (which this seems to confirm) we should all be doing it as well. I think these behaviors will change over time as more and more folks become digitally aware. Thanks for a great read.
Hi Dave,
Thank you. I would guess that the amount of data that Google has collected about how people search, and how people browse the Web is probably much greater than the data they’ve collected about pages found on the Web.
It makes sense that Google would be doing some level of conversion analysis, like many of us who have sites of our own as well. It’s one of the things that can help them get better at what they do.
Whatever they are doing, it appears to be working. Although my personal site seems to be falling back in search results, I do find that it is easier now than before to get the search results I am looking for. I would just be nice to have a little more insight into how they acutally work so I could improve my resulst. What is odd, is that in search engines other than Google, I often come up #1 in results, but am lucky to get the first page in Google.
Hi Jordy,
There are a lot of different things going on in how Google decides to rank pages for different queries. In addition to looking at how relevant a page might be for a search term, and looking at how “important” that page might be through something like PageRank, the search engine also takes those results and reranks them in a number of ways, including personalization, and indications of “quality,” and quite possibly based on a number of other factors.
The way Google ranks pages is a constantly evolving thing, I know from experience that relevancy is in the eye of the beholder and how G decides is as Bill Slawski states a combination of factors. The problem is getting into the mindset of a user and defining what they really meant, 9/10 they are spot on. It’s frustrating when a term has multiple meanings depending on the context it is used in and its the detection of the context that can be the tough but I guess this is lessoned by longer search terms.
Hi Steve,
The interesting thing about being able to see studies like this one is that it gives us a chance to learn more about how the search engines learn about searchers.
As site owners, many of us want to know more about how visitors use our pages, where they go, what they see, where they might click, and so on. So it’s nice to see Google doing some of the same things to explore how they learn about how people use their site.
Terms with multiple meanings likely are a considerable challenge when it comes to deciding how to display the information that a searcher probably wants to see. As a developer of sites, its interesting to watch as Google explores the strategies that people use to search with.
absolutely agreed with bill, google never work on thumb rule system rather it changes the behavior with passage of time ..
Hi Khalid,
I think one of the areas that Google is intently focused upon right now is in finding ways to deliver more recent results. Earlier this week they announced that they would start showing fresher results for 35% of the queries that they receive. I suspect that they realized that they had to take a step like that based upon their query logs,and the number of searches that they were receiving for events that were very recent that they didn’t have good results for yet.