By Bill Slawski, on February 12th, 2010When a search engine indexes an image on the Web, it often has to rely upon the words that it finds associated with that picture. Those words could include the file name, alt text for the image, a caption, as well as other text on the same page.
Those words can be misleading, however, and search engines are trying other approaches to identifying the actual content contained within images. One of the approaches that Google has taken to index images is to have people play each other in a game to label those images. There’s a possibility that Google may add another image game, like the one seen in the screen shot below:
The new game from Google is described in a patent filing published this week, incorporating a way to identify objects within images. The patent application is:
Object Identification in Images
Invented by Yushi Jing, Michael Fink, Michele Covell, Shumeet Baluja
Assigned to Google
US Patent Application 20100034466
Published February 11, 2010
Filed: August 10, 2009
Google is doing more than using games to understand images, and has started using automated ways to compare images to each other to identify different features contained within those images. A couple of years ago, we saw a Google paper come out titled PageRank for Product Image Search, which described how Google might use that technology. You can see that image technology in action in Google Image Swirl, which came out in Google Labs in November of last year.
Interestingly, a couple of the writers listed on that paper are also listed as inventors of this patent application.
The patent tells us about the possibility of Google releasing an “image treasure hunt” game, where players look through images to identify a particular object within a particular image:
In one example, a particular image including a dog is selected as the target of the “image treasure hunt,” and users who are playing the game are told that the target is a dog. Users then proceed to search through images to find the particular image of the dog, which is the target of the “image treasure hunt.”
Each time a user identifies an image with the dog, the user indicates the location of the dog in the image to see whether that dog in the image is the target of the “image treasure hunt.” As users identify dogs in various images, the locations of dogs in various images is stored to enable retrieval later of each image based on a dog being included in the image. As such, the “image treasure hunt” helps catalog the types of objects included in images.
A person finding an object might be asked to click upon it, or trace its outline. It’s also possible that they would be asked to provide more details about the object. For instance, if the object searched for is a dog, a person finding it might indicate that it’s a particular breed of dog, or that it might be in a certain setting or performing some kind of activity, such as a German Shepherd playing Frisbee at the beach.
It’s possible that once an object has been identified in an image that future displays of that image might show a clickable region around the image that links to more information about the object within that image.
The patent also describes how this game might use incentives to encourage people to play, such as having contests that may or may not include prizes.
The images that might be used in this game could include photos, map images, and pictures from video. So it’s possible that Google could use this method of identifying objects within images to improve their image search, video search, and Google Maps.
Why would Google spend so much effort setting up a game to identify objects within images?
The inventor of the Google Image Labeler game, Luis von Ahn, has an interesting presentation that was given as a Google Tech Talk in 2006, titled Human Computation, where he describes the ability of humans to perform some tasks that are relatively easy for people and hard for computers.
Google Maps has a new feature this morning. If you go to Google Maps, and click on the green flask in the top right corner, you’ll see a new window open in the middle of your screen labeled Google Maps Labs. Amongst the features presently in the lab are:
- Drag ‘n’ Zoom
- Aerial Imagery
- Back to Beta
- Where in the World Game
- Rotatable Maps
- What’s Around Here?
- LatLng Tooltip
- LatLng Marker
- Smart Zoom
Kind of interesting that at least one of those is a game. Since the image treasure hunt could potentially be used with images associated with maps, it’s possible that we might see it appear within the Google Maps Labs applications. Since it could also be used to help improve image search and video search by having people identify objects in images and videos, it might spring up over at Google Labs.
Very interesting method of gathering information and when its rolled out on a large scale it will work with really good results, the danger is while its in its infancy.
I did something similar to this in an attempt to create an automated manner of solving captchas. I set up a now dead blog which asked people to enter a captcha when they posted, I positioned this below the post so it was not always seen or used and if someone didnt fill it out then it didnt matter and the post was still made, the point was not to remove spam.
The blog captured the input into the box and attempted to save that input against the specific captcha – this allowed me to see which image required exactly which input to proceed. When you get 1-5 results then you cant really trust them to be correct (captcha are sometimes hard to read) but when you had over 20 and they were 95% the same you knew it was okay.
This concept would apply to google building an image knowledge base although as they will have a lot more people looking they will have to deal with a lot more strange people and people with a sense of humour identifying the images for them…… There will also no doubt, if this became mainstream, be an “attack” in which people purposefully identified a ridiculous image as something it was not in a “george bush, miserable failure” type spam effort but these things are usually highly amusing to watch!
Hi Jimmy,
Interesting experiment that you performed. While researching this post, I read about the attempts of some people to “game” the Google Image Labeler program to get more points when labeling images. The point of those efforts wasn’t to make Google search results spammier, or to provide bad results to searchers, but rather to let people playing the game earn more points. I was surprised to read about it.
It appears that people at Google were paying attention to the way people were “cheating,” and responded with filters and other ways of keeping bad data from entering the system. If you look at the suggestions and advice that are provided at sites that discuss how to get more points out of Google labeler, they mostly now tend to stress things like providing more detailed labels, that if they are shared give more points to both participants.
There is the potential for abuse of systems like these games, and there’s a need to develop robust ways of identifying and filtering such abuse. But the payoff for that work may be very much worth the effort.
As Luis von Ahn noted in the video clip I linked to, in the previous year it was estimated that people spent 9 Billion man hours playing solitaire. The amount of time it took to build the Empire State Building was 9 million man hours, and the amount of time it took to build the Panama Canal was 20 Million man hours (less than a day of solitaire). People solving problems like what labels to give an image, or what objects are within images could accomplish something that computers just aren’t able to do yet. It’s worth the effort to learn how to solve potential attacks to systems like these.
I was just website reader a year ago, as many time I heard how Google will be a giant information repository that makes many of us worried. But after read this article made me appreciate them.
Luis von Ahn point of view approach how he use people daily habit to capture relevance data told me, told me that understanding of human behavior, culture, and sociology perspective of a society have great impact on they perceptive and success.
Interesting model. We have at least 3-5 % of search traffic coming from picture searches already. And It’ ’s picking up. Great blog
You really have interesting topics. Thanks for sharing this. This is really a great, clever and fun strategy that Google used. They sure have such creative minds running in the background.
I hope someone who really understands game theory can unpack this, but the Google Image Labeler game seems to me an interesting case of a non-zero-sum game. Something like an iterated prisoner’s dilemma where cooperation also benefits the police (Google). The only way to gain unfair advantage over the “police” in this scenario is through a pre-arranged contract (just write t**s). So for this project to be successful from Google’s point of view, they have to have calculated the extent to which successful cooperation depends on specificity in labeling, but a pre-arranged contract breaks that calculation.
I’m really not sure if that helps explain why this game is so interesting (and addictive), but I would appreciate it if anyone can correct or amend my application of game theory here. I just think that since the structure of this Image Labeler game is so similar to the scenarios used in explaining game theory, there must be a way to talk about it in those terms.
Bill, I discovered Google Labeler here at your SEO website. I contribute regularly by playing the game because I have over 2,000 images on my site and it’s only fair that I help them out. I also make money from their AdSense program. But there are a few problems with the game itself. The small photos that they show you to identify are of poor quality and sometimes you can’t make out what the picture is. This leads players to just label photos in broad terms like “blue” or “person”. Not really specific enough. Also, I wish they would add a selection button to rate the quality of the photo: very poor, poor, good, excellent. Yes, that’s always open to opinion, so there would be problems there. And who wants to work for free for them, unless it’s fun. Also, I’d like a label about the view. Close up, distant, etc. I took a survey that they had on the Labeler page and suggested that players be able to earn free AdWord ads for labeling pictures. I don’t know if this had anything to do with it, but suddenly Google ads for their online dictionary/translation feature started showing up on my site, which has AdSense ads on it. I never saw them advertise on my site before, so I was thrilled. I stopped playing and the ads disappeared. I started playing again and there they are. Has anyone had this happen? Maybe it’s because I filled out their survey. Is Google becoming like the Russian psychologist Pavlov? Guess that makes me the dog. Humm!
This this just like reCaptcha where it gives the reader a scanned part of a book and in the process helps in digitizing it(book). But this instead involves pictures. Quite smart for Google I must say. And knowing they have like millions of online users using their technology, they’re really putting that to good use. Now this is something new. I wonder what spammers and those with bad intentions will do to this now. O.o
Very interesting idea for image labeling. It takes me back to my college days when we use to pump quarters into the local bar’s MegaTouch machine for a few games of Picture Hunt. With a couple pitchers of beer and a roll of quarters, there was no end to finding the differences between each photo. Game playing is very much a “Tom Sawyer” way of getting others to paint your fence for you!
Bill, you always seem to dig a bit deeper into what Google’s doing. I’m very curious and excited to see what’s on the horizon for video.
Hi ignandy,
Luis von Ahn’s research on how people can solve some problems that computers can, and how they might do so within the context of a game is interesting. Will human behavior be enough, within a context like this, to solve those problems? I’m not sure yet, but it’s interesting to watch.
Hi Schulz,
Thanks. I do think people sometimes underestimate how much using images within the context of their pages can help them, not only from a communications stance, but also in helping pages rank for certain terms, and in bringing traffic to their pages.
It will be interesting to see if efforts like this one, which are aimed at helping to understand what images actually contain, will make a difference.
Hi Andrew,
Imagine my surprise, when halfway through writing this post, I came across the new Google Maps labs, with another game, and some interesting and creative new features. I’m wondering if this game will show up there sometime in the very near future.
Hi Internetsvijet,
I’d like to hear that perspective as well. The Image Labeler game is a combination of collaboration and competition that I’m not quite seeing in this treasure hunt game. I’d like to hear an analysis of the thoughts behind the treasure hunt.
Hi Christina,
I hadn’t heard any criticisms of the Google Image Labeler before – I wonder if Google has received much in the way of feedback like yours.
Also pretty interesting about the ads that you were seeing. I would guess coincidence, but it’s quite possible that I’m wrong.
Hi Michael,
There definitely is some similarity between Google’s image labeler game and recaptcha. Luis von Ahn, who came up with the original idea behind the game was also involved in Recaptcha, which Google ended up purchasing. The common idea behind both is that they present problems to humans that are too difficult for machines to solve. If you didn’t get a chance to watch the video presentation from Luis von Ahn that I linked to towards the end of my post, you might find it interesting. He does discuss what “spammers and those with bad intentions” have attempted to do with Recaptcha systems like the one he came up with.
Hi frodotodo,
Good games have people wanting to play them, I guess.
This patent filing does mention that they might take screen captures from video to include in the image treasure hunts. It’s possible that it might be another way for Google to get a better sense of what is contained in videos.
Actually, the idea is quite old: http://en.wikipedia.org/wiki/ESP_game
Hi Alyona,
The ESP game is old, but this post isn’t about the ESP game, or even Google’s Image Labeler game, which was licensed to Google at least 4 years ago. Back in 2006, I wrote about Microsoft’s patent filing on a game powered search engine back – laying Games and Answering Questions, which named Luis von Ahn as one of the inventors. But this post isn’t about that either.
This post is about a patent filing that was just published less than a week ago, which describes a different “game with a purpose,” as Luis von Ahn refers to them. I’d imagine that the idea probably predates Luis von Ahn, and the ESP game. The first captchas seem to date back to at least 1997, taking advantage of the ability of people to complete tasks that are too difficult for computers:
http://en.wikipedia.org/wiki/CAPTCHA
I love the idea. i love the idea more if i could figure out how to get 100s or 1000s of people to tag my product feeds with the right attributes. i’ve played that google naming contest game with the images a few times to see how it works and it would be awesome for meta data or description data to incorporate all of those different viewpoints. i’d pay for that service if i could and it was reasonable (of course).
[...] The Google Image Treasure Hunt – SEO by the Sea [...]
Hi Ctanner,
Google Webmaster tools used to allow you to decide if images from your site would be included in the Google Image Labeler game. Google announced that feature back in 2006. I don’t know why that choice isn’t available any more, but the image labeler game is still around.
Cool idea however there seems to be some flawed logic. For instance, if you see a picture of a group of men in suits and you have no other information about them, you would give them the label “Men in Suits”. If these men were infact FBI agents, the better description would be “FBI Agents”. The webmaster would know what label suited the picture best, the general public would not. I think google is outsmarting themselves on this.
Hi John Ray,
I’m not sure. I think there’s still some value in the description “men in suits.”
If a webmaster wanted to get across the point that a group of men were FBI agents, it wouldn’t be a bad idea for him or her to use a picture that shows those men in front of a building sign that says “FBI,” or use a picture of those men wearing FBI jackets. Making intelligent choices when choosing which pictures you use on your web site is something that can serve a webmaster well, and if search engines can get smarter about understanding the content of images, then the choice of what we show in pictures on our web sites needs to get smarter, too.
This will really help image searches I am sure. What is amazing to me that simply by relying on keywords around the images along with url text, they already have such an accurate image search! No doubt there stands to be some improvement, and leave it to Google to lead the way. Thanks for sharing this Bill!
Hi Andrew,
You’re welcome.
It is surprising that image search results based upon text associated with those images tend to show decent results. It is possible though that Google may be doing more than just using text associated with those images.
Hi,
I found your site while searching for some background information about roleplaying games and was a bit surprised at the detail and effort that big G will expend to profile gamers.
Thank you for some really informative articles, although they do leave me slightly disquieted.
Hi Dave,
I’m not really sure much profiling is actually going on in what is described in this patent filing. The idea instead is to create games that might help solve problems that computers might not be so good at solving on their own.
I recently read somewhere about what is called “Crowdsourcing” and that’s what Google Image thing appears to be… But I remember I read about this project in particular (http://en.wikipedia.org/wiki/ESP_Game) and I now see it was acquired by GG so my guess is that Treasure Hunt is actually an extension of it or something…
Hi RD,
The ideas behind games like the ESP game and the treasure search are examples of crowdsourcing in action. Google didn’t acquire the ESP game, but rather licensed some of the technology behind it for the Image Labeler game.
I think John brought up an interesting point about how subjective some of image naming can potentially become. I wonder if Google will develop a way to develop relationships between two different descriptions of the same image. Can men in suits be FBI Agents?
Hi Guillermo,
There’s always going to be some level or objectivity/subjectivity to any image. A webmaster can take steps to help a search engine by using a file name, alt text, a caption, and even other text on a page that might be associated with an image.
If that webmaster wants people to look at the image, and be able to understand what the image is about without looking at that text, it doesn’t hurt to choose the best picture possible, and limit some of that subjectivity. So, if I was writing a page about the FBI, and I wanted to choose a picture that made it clear that people in the picture were FBI agents, I would try really hard to make sure I got a shot of at least one of those people wearing one of those blue jackets that FBI agents sometimes wear that have “FBI” on the back in big yellow letters – and that the “FBI” was clearly visible to viewers of the page.
I was just looking for information on this and came across this page – interesting overview. Google has become ever more advanced in this area over the last few years. It’s interesting now that you can actually append the following to the end of an image search url:
&imgtype=face
and all that will be returned in your search are faces. You can perform similar queries with:
&imgtype=lineart and &imgtype=body etc.
While this is nothing new it is still an interesting field when looking at the broader picture of just how Google plans to categorise the world’s images.
OCR is going to play a big part, and I think “games” such as this, where Google relies upon human judgement to categorise information or images, will become increasingly significant.
Hi Chris,
I’ve been wondering why Google doesn’t document these specialized type image searches.
I do think Google will start doing more with OCR at some point in the future. It does seem to make sense for Google to try out games like this as well, since they try to solve problems that humans can solve much more easily than computers.
@Jimmy – nice trick, too bad your blog is dead, I woud like to see this work.
not
I think eventually we will come to a point where computers will be able to figure out what is in the image