When a search engine indexes an image on the Web, it often has to rely upon the words that it finds associated with that picture. Those words could include the file name, alt text for the image, a caption, as well as other text on the same page.
Those words can be misleading, however, and search engines are trying other approaches to identifying the actual content contained within images. One of the approaches that Google has taken to index images is to have people play each other in a game to label those images. There’s a possibility that Google may add another image game, like the one seen in the screen shot below:
The new game from Google is described in a patent filing published this week, incorporating a way to identify objects within images. The patent application is:
Object Identification in Images
Invented by Yushi Jing, Michael Fink, Michele Covell, Shumeet Baluja
Assigned to Google
US Patent Application 20100034466
Published February 11, 2010
Filed: August 10, 2009
Google is doing more than using games to understand images, and has started using automated ways to compare images to each other to identify different features contained within those images. A couple of years ago, we saw a Google paper come out titled PageRank for Product Image Search, which described how Google might use that technology. You can see that image technology in action in Google Image Swirl, which came out in Google Labs in November of last year.
Interestingly, a couple of the writers listed on that paper are also listed as inventors of this patent application.
The patent tells us about the possibility of Google releasing an “image treasure hunt” game, where players look through images to identify a particular object within a particular image:
In one example, a particular image including a dog is selected as the target of the “image treasure hunt,” and users who are playing the game are told that the target is a dog. Users then proceed to search through images to find the particular image of the dog, which is the target of the “image treasure hunt.”
Each time a user identifies an image with the dog, the user indicates the location of the dog in the image to see whether that dog in the image is the target of the “image treasure hunt.” As users identify dogs in various images, the locations of dogs in various images is stored to enable retrieval later of each image based on a dog being included in the image. As such, the “image treasure hunt” helps catalog the types of objects included in images.
A person finding an object might be asked to click upon it, or trace its outline. It’s also possible that they would be asked to provide more details about the object. For instance, if the object searched for is a dog, a person finding it might indicate that it’s a particular breed of dog, or that it might be in a certain setting or performing some kind of activity, such as a German Shepherd playing Frisbee at the beach.
It’s possible that once an object has been identified in an image that future displays of that image might show a clickable region around the image that links to more information about the object within that image.
The patent also describes how this game might use incentives to encourage people to play, such as having contests that may or may not include prizes.
The images that might be used in this game could include photos, map images, and pictures from video. So it’s possible that Google could use this method of identifying objects within images to improve their image search, video search, and Google Maps.
Why would Google spend so much effort setting up a game to identify objects within images?
The inventor of the Google Image Labeler game, Luis von Ahn, has an interesting presentation that was given as a Google Tech Talk in 2006, titled Human Computation, where he describes the ability of humans to perform some tasks that are relatively easy for people and hard for computers.
Google Maps has a new feature this morning. If you go to Google Maps, and click on the green flask in the top right corner, you’ll see a new window open in the middle of your screen labeled Google Maps Labs. Amongst the features presently in the lab are:
- Drag ‘n’ Zoom
- Aerial Imagery
- Back to Beta
- Where in the World Game
- Rotatable Maps
- What’s Around Here?
- LatLng Tooltip
- LatLng Marker
- Smart Zoom
Kind of interesting that at least one of those is a game. Since the image treasure hunt could potentially be used with images associated with maps, it’s possible that we might see it appear within the Google Maps Labs applications. Since it could also be used to help improve image search and video search by having people identify objects in images and videos, it might spring up over at Google Labs.
Very interesting method of gathering information and when its rolled out on a large scale it will work with really good results, the danger is while its in its infancy.
I did something similar to this in an attempt to create an automated manner of solving captchas. I set up a now dead blog which asked people to enter a captcha when they posted, I positioned this below the post so it was not always seen or used and if someone didnt fill it out then it didnt matter and the post was still made, the point was not to remove spam.
The blog captured the input into the box and attempted to save that input against the specific captcha – this allowed me to see which image required exactly which input to proceed. When you get 1-5 results then you cant really trust them to be correct (captcha are sometimes hard to read) but when you had over 20 and they were 95% the same you knew it was okay.
This concept would apply to google building an image knowledge base although as they will have a lot more people looking they will have to deal with a lot more strange people and people with a sense of humour identifying the images for them…… There will also no doubt, if this became mainstream, be an “attack” in which people purposefully identified a ridiculous image as something it was not in a “george bush, miserable failure” type spam effort but these things are usually highly amusing to watch!
Hi Jimmy,
Interesting experiment that you performed. While researching this post, I read about the attempts of some people to “game” the Google Image Labeler program to get more points when labeling images. The point of those efforts wasn’t to make Google search results spammier, or to provide bad results to searchers, but rather to let people playing the game earn more points. I was surprised to read about it.
It appears that people at Google were paying attention to the way people were “cheating,” and responded with filters and other ways of keeping bad data from entering the system. If you look at the suggestions and advice that are provided at sites that discuss how to get more points out of Google labeler, they mostly now tend to stress things like providing more detailed labels, that if they are shared give more points to both participants.
There is the potential for abuse of systems like these games, and there’s a need to develop robust ways of identifying and filtering such abuse. But the payoff for that work may be very much worth the effort.
As Luis von Ahn noted in the video clip I linked to, in the previous year it was estimated that people spent 9 Billion man hours playing solitaire. The amount of time it took to build the Empire State Building was 9 million man hours, and the amount of time it took to build the Panama Canal was 20 Million man hours (less than a day of solitaire). People solving problems like what labels to give an image, or what objects are within images could accomplish something that computers just aren’t able to do yet. It’s worth the effort to learn how to solve potential attacks to systems like these.
I was just website reader a year ago, as many time I heard how Google will be a giant information repository that makes many of us worried. But after read this article made me appreciate them.
Luis von Ahn point of view approach how he use people daily habit to capture relevance data told me, told me that understanding of human behavior, culture, and sociology perspective of a society have great impact on they perceptive and success.
Interesting model. We have at least 3-5 % of search traffic coming from picture searches already. And It’ ‘s picking up. Great blog
You really have interesting topics. Thanks for sharing this. This is really a great, clever and fun strategy that Google used. They sure have such creative minds running in the background.
I hope someone who really understands game theory can unpack this, but the Google Image Labeler game seems to me an interesting case of a non-zero-sum game. Something like an iterated prisoner’s dilemma where cooperation also benefits the police (Google). The only way to gain unfair advantage over the “police” in this scenario is through a pre-arranged contract (just write t**s). So for this project to be successful from Google’s point of view, they have to have calculated the extent to which successful cooperation depends on specificity in labeling, but a pre-arranged contract breaks that calculation.
I’m really not sure if that helps explain why this game is so interesting (and addictive), but I would appreciate it if anyone can correct or amend my application of game theory here. I just think that since the structure of this Image Labeler game is so similar to the scenarios used in explaining game theory, there must be a way to talk about it in those terms.
Bill, I discovered Google Labeler here at your SEO website. I contribute regularly by playing the game because I have over 2,000 images on my site and it’s only fair that I help them out. I also make money from their AdSense program. But there are a few problems with the game itself. The small photos that they show you to identify are of poor quality and sometimes you can’t make out what the picture is. This leads players to just label photos in broad terms like “blue” or “person”. Not really specific enough. Also, I wish they would add a selection button to rate the quality of the photo: very poor, poor, good, excellent. Yes, that’s always open to opinion, so there would be problems there. And who wants to work for free for them, unless it’s fun. Also, I’d like a label about the view. Close up, distant, etc. I took a survey that they had on the Labeler page and suggested that players be able to earn free AdWord ads for labeling pictures. I don’t know if this had anything to do with it, but suddenly Google ads for their online dictionary/translation feature started showing up on my site, which has AdSense ads on it. I never saw them advertise on my site before, so I was thrilled. I stopped playing and the ads disappeared. I started playing again and there they are. Has anyone had this happen? Maybe it’s because I filled out their survey. Is Google becoming like the Russian psychologist Pavlov? Guess that makes me the dog. Humm!
This this just like reCaptcha where it gives the reader a scanned part of a book and in the process helps in digitizing it(book). But this instead involves pictures. Quite smart for Google I must say. And knowing they have like millions of online users using their technology, they’re really putting that to good use. Now this is something new. I wonder what spammers and those with bad intentions will do to this now. O.o
Very interesting idea for image labeling. It takes me back to my college days when we use to pump quarters into the local bar’s MegaTouch machine for a few games of Picture Hunt. With a couple pitchers of beer and a roll of quarters, there was no end to finding the differences between each photo. Game playing is very much a “Tom Sawyer” way of getting others to paint your fence for you!
Bill, you always seem to dig a bit deeper into what Google’s doing. I’m very curious and excited to see what’s on the horizon for video.
Hi ignandy,
Luis von Ahn’s research on how people can solve some problems that computers can, and how they might do so within the context of a game is interesting. Will human behavior be enough, within a context like this, to solve those problems? I’m not sure yet, but it’s interesting to watch.
Hi Schulz,
Thanks. I do think people sometimes underestimate how much using images within the context of their pages can help them, not only from a communications stance, but also in helping pages rank for certain terms, and in bringing traffic to their pages.
It will be interesting to see if efforts like this one, which are aimed at helping to understand what images actually contain, will make a difference.
Hi Andrew,
Imagine my surprise, when halfway through writing this post, I came across the new Google Maps labs, with another game, and some interesting and creative new features. I’m wondering if this game will show up there sometime in the very near future.
Hi Internetsvijet,
I’d like to hear that perspective as well. The Image Labeler game is a combination of collaboration and competition that I’m not quite seeing in this treasure hunt game. I’d like to hear an analysis of the thoughts behind the treasure hunt.
Hi Christina,
I hadn’t heard any criticisms of the Google Image Labeler before – I wonder if Google has received much in the way of feedback like yours.
Also pretty interesting about the ads that you were seeing. I would guess coincidence, but it’s quite possible that I’m wrong.
Hi Michael,
There definitely is some similarity between Google’s image labeler game and recaptcha. Luis von Ahn, who came up with the original idea behind the game was also involved in Recaptcha, which Google ended up purchasing. The common idea behind both is that they present problems to humans that are too difficult for machines to solve. If you didn’t get a chance to watch the video presentation from Luis von Ahn that I linked to towards the end of my post, you might find it interesting. He does discuss what “spammers and those with bad intentions” have attempted to do with Recaptcha systems like the one he came up with.
Hi frodotodo,
Good games have people wanting to play them, I guess. 🙂
This patent filing does mention that they might take screen captures from video to include in the image treasure hunts. It’s possible that it might be another way for Google to get a better sense of what is contained in videos.
Actually, the idea is quite old: http://en.wikipedia.org/wiki/ESP_game
Hi Alyona,
The ESP game is old, but this post isn’t about the ESP game, or even Google’s Image Labeler game, which was licensed to Google at least 4 years ago. Back in 2006, I wrote about Microsoft’s patent filing on a game powered search engine back – Playing Games and Answering Questions, which named Luis von Ahn as one of the inventors. But this post isn’t about that either.
This post is about a patent filing that was just published less than a week ago, which describes a different “game with a purpose,” as Luis von Ahn refers to them. I’d imagine that the idea probably predates Luis von Ahn, and the ESP game. The first captchas seem to date back to at least 1997, taking advantage of the ability of people to complete tasks that are too difficult for computers:
http://en.wikipedia.org/wiki/CAPTCHA
I love the idea. i love the idea more if i could figure out how to get 100s or 1000s of people to tag my product feeds with the right attributes. i’ve played that google naming contest game with the images a few times to see how it works and it would be awesome for meta data or description data to incorporate all of those different viewpoints. i’d pay for that service if i could and it was reasonable (of course).
Hi Ctanner,
Google Webmaster tools used to allow you to decide if images from your site would be included in the Google Image Labeler game. Google announced that feature back in 2006. I don’t know why that choice isn’t available any more, but the image labeler game is still around.
Cool idea however there seems to be some flawed logic. For instance, if you see a picture of a group of men in suits and you have no other information about them, you would give them the label “Men in Suits”. If these men were infact FBI agents, the better description would be “FBI Agents”. The webmaster would know what label suited the picture best, the general public would not. I think google is outsmarting themselves on this.
Hi John Ray,
I’m not sure. I think there’s still some value in the description “men in suits.”
If a webmaster wanted to get across the point that a group of men were FBI agents, it wouldn’t be a bad idea for him or her to use a picture that shows those men in front of a building sign that says “FBI,” or use a picture of those men wearing FBI jackets. Making intelligent choices when choosing which pictures you use on your web site is something that can serve a webmaster well, and if search engines can get smarter about understanding the content of images, then the choice of what we show in pictures on our web sites needs to get smarter, too.
This will really help image searches I am sure. What is amazing to me that simply by relying on keywords around the images along with url text, they already have such an accurate image search! No doubt there stands to be some improvement, and leave it to Google to lead the way. Thanks for sharing this Bill!
Hi Andrew,
You’re welcome.
It is surprising that image search results based upon text associated with those images tend to show decent results. It is possible though that Google may be doing more than just using text associated with those images.
Hi,
I found your site while searching for some background information about roleplaying games and was a bit surprised at the detail and effort that big G will expend to profile gamers.
Thank you for some really informative articles, although they do leave me slightly disquieted.
Hi Dave,
I’m not really sure much profiling is actually going on in what is described in this patent filing. The idea instead is to create games that might help solve problems that computers might not be so good at solving on their own.
I recently read somewhere about what is called “Crowdsourcing” and that’s what Google Image thing appears to be… But I remember I read about this project in particular (http://en.wikipedia.org/wiki/ESP_Game) and I now see it was acquired by GG so my guess is that Treasure Hunt is actually an extension of it or something…
Hi RD,
The ideas behind games like the ESP game and the treasure search are examples of crowdsourcing in action. Google didn’t acquire the ESP game, but rather licensed some of the technology behind it for the Image Labeler game.
I think John brought up an interesting point about how subjective some of image naming can potentially become. I wonder if Google will develop a way to develop relationships between two different descriptions of the same image. Can men in suits be FBI Agents?
Hi Guillermo,
There’s always going to be some level or objectivity/subjectivity to any image. A webmaster can take steps to help a search engine by using a file name, alt text, a caption, and even other text on a page that might be associated with an image.
If that webmaster wants people to look at the image, and be able to understand what the image is about without looking at that text, it doesn’t hurt to choose the best picture possible, and limit some of that subjectivity. So, if I was writing a page about the FBI, and I wanted to choose a picture that made it clear that people in the picture were FBI agents, I would try really hard to make sure I got a shot of at least one of those people wearing one of those blue jackets that FBI agents sometimes wear that have “FBI” on the back in big yellow letters – and that the “FBI” was clearly visible to viewers of the page.
I was just looking for information on this and came across this page – interesting overview. Google has become ever more advanced in this area over the last few years. It’s interesting now that you can actually append the following to the end of an image search url:
&imgtype=face
and all that will be returned in your search are faces. You can perform similar queries with:
&imgtype=lineart and &imgtype=body etc.
While this is nothing new it is still an interesting field when looking at the broader picture of just how Google plans to categorise the world’s images.
OCR is going to play a big part, and I think “games” such as this, where Google relies upon human judgement to categorise information or images, will become increasingly significant.
Hi Chris,
I’ve been wondering why Google doesn’t document these specialized type image searches.
I do think Google will start doing more with OCR at some point in the future. It does seem to make sense for Google to try out games like this as well, since they try to solve problems that humans can solve much more easily than computers.
@Jimmy – nice trick, too bad your blog is dead, I woud like to see this work.
I think eventually we will come to a point where computers will be able to figure out what is in the image 🙂 not
Great topic and some smart words from the peeps above. im not sure in what is being said about images at all i think we should always do our best to insure spiders, users can all understand what the image is saying.
Hi Philip,
Thanks. The challenge is that the search engines have some limitations in interpreting what an image is actually about.
They can look at the file name, any alt text used with an image, a caption for the picture if there is one, and maybe some of the other words on the page or in links to the image or the page that the image appears upon. You’re right that we should do as good a job as we can in making sure that those do a good job of telling us what an image is about, but that’s still pretty limiting. What if someone does everything right in relation to the words associated with the picture, and then uploads the wrong image? None of those words will be accurate at all.
The Image treasure hunt that this post is about, and Google’s image labeler game take advantage of a different approach, where people play a game against each other which incidentially also lets a search engine know more about what an image is about. The idea behind games like this is that people are better at some tasks than computers, and people are willing to spend a lot of time playing games like the one described in this post. Combine that ability of people to perform some tasks very well with their willingness to play, and they can do something that computers have a hard time with.
Very interesting and informative post! Google is at it again and this time they are getting us to do their work for them in the format of a game. Genius! Hopefully this helps to make Google images easier to find in the future.
That’s a great method of commitment forming across target audiences. I’m actually familiar with similar methods from a marketing perspective, but Google have surprised me with this. The collaborative potential in this endeavor is incredible, and consider the traffic generation options – if Google decide that today is the Chevrolet picture treasure hunt, a traffic tsunami will drop on all pages containing images optimized for “Chevrolet”.
That’s just one lame example, but consider the consequences…
Hi Mike,
Thanks. I guess the Google Image Labeler game has gotten a good response from the public.
This patent describes playing this game in Google’s Street Views, but I’m wondering if a game like it in YouTube wouldn’t go over even better.
Hi Dan,
Interesting thoughts. The patent really doesn’t delve much into potential advertising approaches behind the game – it seems more intended to help Google identify what appears in pictures and videos so that they could be easier to find. But, you may be right about the potential for marketing.
So I guess we agree on that. I can only imagine Google’s marketing director sitting in his office, feet up, reviewing the monthly revenue report and asking his redheaded, long legged secretary to pass it down that the most expensive PPC keywords are to be targeted in the next week’s picture hunt 🙂
I get your drift Bill. Sorry for the occasional manifestation of my lousy (an understatement) sense of humor 🙂
Hi Dan,
Not sure that Google would aim at targeting this kind of information based solely upon how it would help advertising. If Google could find a way to automate this process so that they could get as many people as possible involved in participating, that would more likely be the approach they would take with something like this – the larger the amount of data they have, the more useful the service would actually become.
Hi Dan,
No problem. Just as Google’s approach to indexing is somewhat of a black box, the business practices within the Googleplex can be as well. 🙂
I find it pretty hard to identify object sometimes. Hope they decide to use a version we actually can understand. I just hate to try over and over again, up to 10 times or more.
Hi Markus,
Objects or Captchas?
Hi Bill,
Sometime back I heard that some major search engines are looking at the option to use Facial recognition system type software for Image Search to provide better results. Don’t you think that this could perhaps be better way than the Image Hunting Game to provide accurate image results?
Hi Max,
Facial recognition identification might work well with faces, but the kinds of images included in this treasure hunt game include many other types of objects other than just faces. While the search engines are also working upon object recognition, the purpose of games like this is to have people power some of the discovery of different types of objects, so that object recognition software can have a lot of data to use to do its job better.
Interesting. I wonder if Google is using the image label as a way to categorize the shapes that it finds in the image. It would be interesting to see how data like that would correlate if a few random examples were pulled and subject to scrutiny. I am betting that there would be a pretty relevant correlation…maybe better than 25% accurate when human-polled…???
Mark
Hi Mark,
I suspect that the image labeler information is being used as part of the image similarity search that Google is now using for image search – the “sort by subject” search that you can now do with images. See:
http://googleblog.blogspot.com/2011/05/sort-by-subject-in-google-images.html
This looks like an interesting approach for better image indexation. The idea is great, I just wonder how many people will be willing to play “label my images”?
Hi Robert,
Another image based game from Google was just retired last year (Google Image Labeler), but had run since 2006. I suspect that if it wasn’t helpful to Google, they would have retired it much sooner.
It does seem like people are willing to spend times on games like this, and if they can help the search engine by doing the things that people are better at than computers, I think they are worth trying.