Many tasks are trivial for humans but continue to challenge even the most sophisticated computer programs. Traditional computational approaches to solving such problems focus on improving artificial intelligence algorithms. Here, we advocate a different approach: the constructive channeling of human brainpower through computer games. Toward this goal, we present general design principles for the development and evaluation of a class of games we call “games with a purpose,†or GWAPs, in which people, as a side effect of playing, perform tasks computers are unable to perform.
A paper from Yahoo researchers, Thumbs-Up: A Game for Playing to Rank Search Results, describes a game that they developed and tested internally at Yahoo to allow participants to compete against each other in ranking how relevant pages are for specific search queries.
To play the game, players start by logging in and are randomly matched with another player.
They are shown the same query term or phrase and images of two web pages that are supposed to be relevant for the query. To score points, the players have to agree on the same page as the most relevant query.
This is similar to the ESP game developed a few years back to tag images, which the Google Image Labeler game is based upon.
The paper quoted at the start of this post is cited in the Yahoo paper and is co-written by Luis von Ahn, who was behind the development of the ESP game. He describes many other games with a purpose that has been developed since.
Would you play “Thumbs-Up” if Yahoo released it to the public?
Interesting idea. I’ve always considered SEO as a game with the ultimate prize being the top ranking. Remember those old arcade games such as Pacman and Space Invaders? They never ended (well seemed to never end), instead of bragging to friends that you beat the game, you bragged that you had a better score. Perhaps games are the way to go.
yeah ,sure But we are already playing games of rankings and links here and I really love this game
I would definitely try this, but not sure how long it would keep my interest.
Hi Robert and Miami web design and People Finder,
I’ve been thinking about this a good deal since I came across the paper, and wrote this post. I’m not sure that I like the idea of incorporting game results into search engine rankings, especially when the impact of those games could affect people’s livelihoods. I’m not as concerned with the ranking of images in something like the ESP game, or Google’s Image Labeler program, because those can have much less impact.
The idea of “games with a purpose” is interesting, but I see some potential problems with the approach, such as people playing without aiming to really increase the relevancy of results. I’d like to see more research on the topic, but I’m not convinced that this is a good idea.
I find SEO interesting, and enjoyable, but I’m not sure that I could bring myself to view it as a game either. The potential impact on site owners, on consumers, and on the public good brings with it a certain amount of respect for the how search rankings might influence people’s lives. In an ideal world, seach engines help site owners and people interested in what those sites offer together and make a positive difference in their lives and livelihoods.
If the aim of the game is to select the same web page as the other player, then players will probably change their game plan as they learn about the other player to ensure matches and a better score.
That behaviour may not have been seen testing within Yahoo, however would like to see if the public are more interested in relevancy or a new high score!
—
Chris
Hi Chris,
I guess the ideal aim would be both relevancy and a high score at the same time. I don’t believe that you get to know who the other person is who is participating when you play, so some of that bias might be eliminated from this process. Also, there’s a limited number of selections that you might make per game, so the opportunity to learn more about the other person whom you’re working with is also limited.
Will I play? Sure! Sounds like fun.
I think search engines will ALWAYS require some sort of human verification, there is only so much that 1’s and 0’s will tell you. Quantitative analysis is great, but qualitative analysis using humans is something really lacking I find in the space, the only way to keep up with the huge amounts of data popping up in new web pages is viral human involvement.
Hi Webmaster,
Thanks. 🙂 Getting humans involved sounds like a great idea on the surface, at least as long as you can find a way to have them involved in a way that is engaging and will provide helpful and useful information. The “games with a purpose” that I’ve been seeing do seem to do that fairly well.
I’d definitely have a go! Although I am definitely with you when I say that incorporating this into the actual search results would be a backwards step. With all the Search engines trying to perfect their algorithm’s constantly I don’t think they’ll ever turn to human ‘error’ to rank results! 🙂
Hi Sussex Web Designer,
You raise a good point, But I don’t think that there’s any escaping the influence of humans in the ranking of pages.
For instance, algorithms follow assumptions created by humans on how pages should be ranked.
Many “machine learning” algorithms rely upon the use of sample sets that have been ranked by people as well.
And search engines do use human evaluators to check upon and tune the relevancy of their results. I saw an ad on Craig’s list earlier today for a “relevancy judge,” or judges, placed by a company that often works with Google to find evaulators. The pay for this part time position, located in the US, was $14/hour.
I would play the game because I find SEO fascinating and would want to see the latest ideas in the works. However, this type of SEO really isn’t much different than building authority via links. In fact, building authority via any type of “popularity measure” isn’t different than backlinks in theory, just different in forum.
I suspect the best SEO results in the long run will incorporate many variables of building trust in a website – perhaps “thumbs-up” style games, social media votes, bookmarking votes, Google’s +1, and of course backlinks.
Hi Peter,
These kinds of games are aimed at tasks that are often easier for people to perform than computers, and I can see a role for them as a way to get feedback that the search engines otherwise might not have the ability to receive. I’m not sure that the aim of the game is to be a “popularity” measure in quite the same way that a +1 vote is, but popularity may play a role in the results recieved.
I expect that judgments regarding “relevance” are often tempered by many of the same factors that may determine how popular a site might be, such as the layout and design of pages, whether the language used is suited for more of a mainstream audience than a technical one, and more. The “most” relevant site might be one which finds ways to appeal to a wider audience even if it might be less relevant for a particular query than a site created for a more specialized audience.