How much can eye-tracking studies tell us about the ways that people read and react to search engine results pages? What is a search engine like Google doing when it comes to using eye-tracking?
I have to confess that I’m skeptical when I read about some of the studies that describe a golden triangle of Google and show nice heat maps which attempt to indicate how people view search results. Why the skepticism?
It began with one of the more interesting eye-tracking studies that I’d seen over the past few years, where the Poynter Institute used eye-tracking to test the way that people read news web sites. Excited by the results, I wanted to find out more, and started searching around.
I came across a blog post from Frank Spillers which looked closely some of the many issues involving eye-tracking, specifically within the context of that Poynter research – Eye-Tracking studies- Usability holy grail?
There are some great takeaway points from this blog post, and I think one of the most important is this one:
Bottom line: If you are using eye-tracking, to make it meaningful, you must:
…
3. Match what users are doing and feeling with the eye-tracking data reports. Data is just data unless it is meaningful and informative.
My understanding of the “golden triangle of Google” research I linked to above is limited to what I’ve seen about it described at a couple of Search Engine Strategies sessions and in a few blog posts. Yet I’m left wondering why there’s only one heat map talked about and shown to describe the way people look at search results. And, there’s no insight into what people are doing and feeling when that golden triangle is shown.
Even on a simple level, shouldn’t the way someone looks at search results pages be different if the type of search was for different purposes? Say, for instance, a navigational search as opposed to an informational one, or a transactional one? See Andrei Broder’s A taxonomy of web search (pdf) for more on the differences between those types of searches.
So, how is Google attempting to use eye-tracking to understand how people look at their pages?
A paper prepared by a couple of Googlers, to be presented at the CHI2006 in Montréal this past April, gives a little insight into the backgrounds of Google researchers, and their “initial experiences with incorporating eyetracking into user studies at Google.” Laura Granka and Kerry Rodden wrote and presented Incorporating Eyetracking into User Studies at Google, and I’m fairly comfortable with their use of eye-tracking after reading their description of their approach.
Here’s why:
1. Their statement that: “in general, existing eyetracking software lacks specialized features for analysis of studies where web pages are used as the stimuli, e.g., dealing with repeat visits to the same page, or page content that changes dynamically.”
2. Google has access to log file information to tell them what people were doing on pages, and eye-tracking information can be used to help understand that behavior instead of being used to define it.
3. A clear understanding and explanation of the importance of context, as shown in this statement:
While interpretation standards for the meaning of ocular indices are desired, another important issue to consider is that task significantly influences viewing behavior. Task differences most notably affect a user’s scan path, but they also lead to different interpretations of certain ocular indices. For instance, fixation duration can take on a different meaning based on whether a user is performing a reading task or visual search task. Furthermore, scan path and the number of fixations in specific areas of interest also differ based on the task a user was given.
4. The desire to supplement eye-tracking information with other information, such as click-throughs and mouse movements, and any correlations between them.
A post from the Google Research Blog back in April describes some of the other Google usability testing presented at the CHI conference.
Very, very useful post for anyone interested in usability studies.
Thanks 🙂
Thank you, Richard.
I’m glad that I came across that blog post from Frank Spillers. It put the usefulness and effectiveness of eye-tracking studies into a reasonable perspective for me.
I haven’t heard anything negative about the eye-tracking studies from Enquiro, yet I’ve never seen anyone really put that golden triangle into the context with the data that it is intended to impart. It seemed worth putting it up beside the Google paper because the Enquiro studies are so well known (and the paper from Google made such a small splash in forums and search related blogs).
Thanks, Richard.
I’m looking forward to it.
Some very good points, Steve.
Neil Patel, from Pronet Advertising has been suggesting to me that I left align titles on posts here. I’ve looked at the site a couple of times like that, and think it looks a little more cluttered when I do. But his suggestion fits in well with the inclination for people to read from left to right.
I’m hoping that the usability interface folks at Google share more information about some of the testing that they do.
I see we’re going to be reading you over at SEL 🙂
Well done and obviously deserved!
Bill,
I have to agree with the Pavlovian aspect of user behavior, especially when analyzing the ocular behavior of a user. This was dramatically influenced by my working with a focus group on a redeveloped site my team was nearing completion.
With no prejudices, the layout both image based and contextual base, their eyes moved where we wanted them. The normal inclination of an English speaking/reading user is to follow the same path that they read – left to right, top to bottom; however, you can do things to influence their attention.
I think this type of research is great for building sticky-ness to your site, but I think you can start training visitors on their first visit.
Cheers,
Steve
I think that we are in complete agreement on eye-tracking, Yuri. Thanks.
I’m not sure that eye-tracking should be the primary tool to use, or at least, the only tool. But it can be useful.
I’m going to have to explore left aligning those titles, some more. (Be tough to read upside down. 🙂 )
Cheers.
Testing testing.
Some good post. One thing about eye-tracking, though, is that it may not completely reveal how humans behave on websites. They only reveal how humans behave on the websites they watched while being studied. What if they had a completely different website, with non-standard navigation? Where would their eyes go?
That being said, eye tracking certainly is useful in determining the basic information architecture.
By the way, I’d figure left-aligned titles may be more human friendly, too. Up till now I haven’t paid much attention to inconvenience the right align causes, as your blog surely is worth reading even if you align the text from right to left and turn it upside down.
Never even noticed the titles 😀
Always left-aligned in my reader!
@Stephen – you make some very good points and I would love to hear some elaboration on what you found.
Rgds
Richard
I agree with you, Richard. Would love to hear a little more from Steve on what types of things that he was seeing with the eye-tracking that he was doing.
I’d imagine that the use of images, motion, whitespace, font sizes and styles, and navigation layout and consistency (or inconsistency) throughout a site can have an impact.
Mixing ads with polls, and other user interactive features might be the kind of thing that could get people to look in an area that they might otherwise skip past. That’s one of the reasons that I like the way search engines are using something like Google’s one-box – inserting images, news, local results, and other features in an area where people might have anticipated only advertising would appear in the past. Would love to see studies related to how people react to those.
Thanks, Manoj
I probably need to start looking at some more of the scientific literature associated with eyetracking before writing more on this topic. I’m not sure of how much value there is in comparing the amount of time spent on different search engine results pages. Some of the other results have me scratching my head.
Pogo sticking behavior on search results pages? I’d be more surprised if there wasn’t.
An new eye tracking study about Yahoo and MSN has also been recently released by Enquiro:
http://www.enquiro.com/eyetrackingreport.asp
Ahh..it’s always heartening to see your studies debated. Actually, a lot of what you are talking about is discussed, at length, in the two studies we’ve done. We looked at first versus repeat visits, we talked about intent and we are the first to say that the Golden Triangle image that shows up everywhere is not representative of every visit to a page. That said, it is a remarkable stubborn scan pattern. You would be amazed at how often it shows up.
In regards to the above quotes on the limitations of eye tracking as a methodology, I would hasten to add that eye tracking software does everything standard usability testing software does. It captures everything that log information would. You capture the entire session, including repeat visits to the same page, you capture clicks, you capture time spent on page. It’s an incredibly rich data set. While it shouldn’t be the be all and end all of research, I think it captures more dimensions of the user experience than one might think, given your initial post.
There are limitations, but they are due primarily to the limitations of using foveal focus to indicate what is actually being seen on the page. Eye tracking doesn’t do a very good job of capturing what’s seen through peripheral vision, and that can have a dramatic impact with things like images and brand awareness. Again, these become important, based on the task assigned.
As with most things, people tend to take what they want from the research. Our latest study is well over 200 pages, including dozens of heat maps. But as with the first study, there will likely be one or two things that people tend to focus on. And so the work is perceived as being one dimensional. Believe me, we try to look at a lot of aspects when making our conclusions. We looked at hundreds of sessions during the analysis of the data and invested well over a thousand man hours.
Hi Gord,
Thank you. I appreciate the time that you took to respond to this post, and to present your thoughts on eye-tracking and its value in testing the interfaces of the search engines.
I also appreciate your sharing with us some background about your studies, and even some limitations of eyetracking. Part of my skepticism stems from not enough knowledge about your studies, and the limited information that I have about them.
I’m going to take advantage of responding to your post to point out another study from Laura Granka, which I thought was interesting, in her days before she joined Google. It is a powerpoint slideshow, but interesting:
Eye-Tracking Analysis of User Behavior in WWW-Search (ppt)
I was just wondering how different an eye tracking study would be between the SERPS and let’s say an e-commerce store. Check amazon for example, their buy now button is not located in the ‘golden triangle’..it’s on the right side of the page. According to a lot of eye tracking studies, nobody is watching there…
Are there any eye tracking studies on the www on e-commerce stores ?
Thxs !
Dave
Hi Everyone,
for those who are interested in low-cost and fast eyetracking I suggest having a look at computer vision based approach. It combines saliency maps http://www.scholarpedia.org/article/Saliency_map, object recognition http://en.wikipedia.org/wiki/Computer_vision and human visual attention mechanisms http://scholar.google.com/scholar?q=human+visual+attention+mechanisms&complete=1&hl=en&esrch=GoogleSuggestBeta&um=1&ie=UTF-8&oi=scholart.
See how real eyetracking compares to computer vision-based one:
http://www.youtube.com/watch?v=bZOVcmwHZZk
http://www.youtube.com/watch?v=1rAl9BiCtKQ
Cheers
Hi Tsishkou,
Thanks for the links – some very interesting stuff there. I’ve been seeing a number of patent filings from the search engines that discuss visual segmentation of webpages to understand which segments exist upon a page, so that parts of pages can be compared against others to determine if they relate to each other, if information extracted from different segments can be clusted with segments from other pages on specific topics or objects.
The computer vision based approaches of detection and segmentation may begin to play an interesting role in how search engines work in the future. Add Computer Vision based Eyetracking, and things could start becoming really interesting.
Hi Everyone,
there all kinds of interesting discussion about eye-tracking for website optimization on the net. To summarize:
(1) F-shape
(2)Automated Computer “vision” algorithms to simulate looking
(3)Saliency
(4)left-side page vs. right-side page
(5)reading vs. scanning
(6)how the task directs what people look at
(7)’confused’ Spaghetti vs ‘directed’ simple path looking
(8)now people are looking into the physiological mechanisms driving where people look
* What is MISSING is a proper behavioural perspective on this.
What we know as ‘popular’ eye-tracking has been largely developed by Computer Scientists, Engineers, and general hacks. Way, way back, the people who were using eye-tracking were people in the field of Optometric and neuroscience research. Research in these two fields help us understand the physiology behind eye-tracking. What is absolutely necessary is a perspective on all of this from Behavioural Neuropyschology. This field has existed for a long time and has the experience to inform us how to design proper eye-tracking studies and how to properly interpret eye-tracking data. I highly suggest a quick read of an article posted in the StepForth website: http://www.stepforth.com/blog/getting-best-eye-tracking-results-for-your-website.php
Thank you, Dr. Zeman,
I think just looking at heatmaps, without an informed analysis like you suggest may result in conclusions that might not be merited. Appreciate your input on the topic.
An earlier commenter mentioned low cost eye tracking. Our company is releasing EyeGuide, an eye tracking hardware/software system for under $1,500. Hopefully this will make eye tracking research more accessible to all researchers instead of a limited few.
Hi Greg,
Eye tracking should be more accessible to researchers, so it’s good to see someone going down the path that you are. Thanks.
What I’m wondering in all those eyetracking studies: how do the testing people react when they know somebody’s tracking them? Do they act different?
I think there’s always an certain error rate (as well in the computer predicted eyetracking, the human eyetracking or the in betweens)
Hi Pieter,
I think in any system where such testing and measuring goes on, the awareness of that testing is going to impact the results.
Check out the system I’ve described under the subheading “Webpage or eBook Placeholder based upon Gaze” in my post Google Picks Up Hardware and Media Patents from Outland Research. That technology watches where you are looking on a screen to save your place on the page, so that you could return to it easily. But it could just as easily be used to do more in terms of tracking where you are looking on a page. It seems pretty invasive for it to do more than what is described, though. But possible.