How Google Might Offer Face Search by Using Pictures from Social Networks

Sharing is caring!

If Google decided to include a facial recognition search as part of the Visual Search described in a Google patent application a couple of weeks ago, a couple of questions need to be addressed by the search engine.

A twitter followers page for George Washington. Happy President's Day.

One is, where would they get the pictures to power that facial recognition software (hint in the image above)? The other is, how would they best avoid privacy concerns?

A patent filing from last week provides some possible answers.

Chances are that many of tomorrow’s searches will be spoken, or started with the click of a camera. Someone uses their phone to ask for information from a Google or Bing, or snaps a picture and uploads it to the search engine, hoping that it can identify the objects in the image.

Google has a somewhat primitive visual search engine in Google Goggles, which will search for information about some types of images, such as logos, artwork, types of wine, landmarks, books, and text, from pictures sent to the search engine. But it has limitations. The patent filing I wrote about a couple of weeks ago described how Google might identify different objects in the same picture, and conduct multiple searches based upon those images.

For instance, if you took a picture of a sports drink box, it might search for information about the logo and the trademark on the box. It might identify the product displayed there, and even perform a search on the face of a celebrity endorser.

A view of the different types of results that might be returned in response to the use of the image as a query, including products, logos, facial recognition, product and advertising matches, and web results.

A Google whitepaper from 2008, Large Scale Learning and Recognition of Faces in Web Videos describes one way Google might identify the faces of celebrities.

For a facial recognition search, Google could use images collected of celebrities in news reports, web sites, and information reposititories such as the IMDB and wikipedia. Google might also be able to identify people from images in public domain galleries.

For a facial recognition search to work well, Google would need access to both a large number of images, and information about those images such as the names of the people in the pictures. One of the largest sources of labeled images on the Web exist in social networks such as Facebook and Twitter, and in social image galleries such as Flickr and Picasa web albums, in applications such as GMail and Google Buzz, in blogging communities such as livejournal, and in publically accessible URLs on the Web.

Google might limit the access of people searching to images from people they are connected to upon those social networks, and possibly to people connected to friends. The patent filing notes that it may provide a way for people to opt in to be included in this type of image search as well.

The images that are returned on a face search could be ranked and ordered a number of ways. One would be based upon a visual similarity algorithm. Another would be based upon ranking information related to social interconnectivity.

A third ranking signal that might be used could be based upon location. A location information module included in the facial recognition search might find current location information for a searcher and a person searched for by looking at location information from a few different sources. One might be GPS receivers in mobile devices. Another might be the IP addresses of desktop computers used by those people. A third might be a published location by that person (for example, a tweet from someone noting that “I am currently at a conference in Boston.”

The patent application is:

Facial Recognition with Social Network Aiding
Invented by David Petroum, Andrew Rabinovich, and Hartwig Adam
US Patent Application 20110038512
Published February 17, 2011
Filed: August 5, 2010


A facial recognition search system identifies one or more likely names (or other personal identifiers) corresponding to the facial image(s) in a query as follows. After receiving the visual query with one or more facial images, the system identifies images that potentially match the respective facial image in accordance with visual similarity criteria.

Then one or more persons associated with the potential images are identified. For each identified person, person-specific data comprising metrics of social connectivity to the requester are retrieved from a plurality of applications such as communications applications, social networking applications, calendar applications, and collaborative applications.

An ordered list of persons is then generated by ranking the identified persons in accordance with at least metrics of visual similarity between the respective facial image and the potential image matches and with the social connection metrics. Finally, at least one person identifier from the list is sent to the requester.

When this system looks at faces, it would use a visual feature extracting program to collect information about:

  • The indoor habitat where the image was taken
  • The outdoor habitat where the image was taken
  • Gender
  • Race
  • Facial characteristics
  • Facial hair characteristics
  • Head hair characteristics
  • Headware information
  • Eye color
  • When and where the picture was taken, and
  • Who else might be in the picture

Person specific data might be taken from public information databases, and which contain connection metrics of social connectivity between the person associated with a potential image match and the requester. These applications can include:

  • Social network databases
  • Social microblog databases
  • Blog databases
  • Email databases
  • IM databases
  • Calendar databases
  • Contact lists, and/or
  • Public URLs


Google’s visual search may someday offer search results that include facial recognition results, but may both rank and limit those results based upon the availability of images found for celebrities, images in the public domain, and images found in social networking applications. Access to those many of those images may be limited based upon connections within those social networks, much in the way that Google limits social search results to people whom you are connected to on those networks.

The patent filing lists a number of related patents that weren’t included in the earlier patent filing on visual search that I wrote about earlier, including a couple that potentially have implications for local search as well. Those two are “Actionable Search Results for Street View Visual Queries” and “Hybrid Use Location Sensor Data and Visual Query to Return Local Listing for Visual Query.”

I’ll be keeping an eye open for those, to write about them when they are published.

Sharing is caring!

48 thoughts on “How Google Might Offer Face Search by Using Pictures from Social Networks”

  1. That would be one of the creepier things I have seen. I know that they are looking to do this with pictures of landscapes and landmarks, but I wouldn’t feel good about people taking my picture and finding information about me.

    I understand that privacy is becoming as scarce as the passenger pigeon, but I hope that things like this don’t come to pass.

    I have already been on enough dates with women who have googled my name to creep me out. Plus, it takes away from dinner conversaion! :)

  2. Hey Bill,

    The facial recognition search seems like a great concept. But like you said, how would they avoid privacy concerns? I look forward to reading more on the related patents filing on visual search. Thanks for sharing.

  3. Very interesting, it’s amazing how much object and facial recognition is coming along these days!

    I suppose the use of facial recognition search from social media etc would only be possible from publicly visible profilespages anyway so I don’t think privacy would be much of an issue here – I don’t think the general public would see it the same way though!

  4. This has the potential to be really great. Social media might be a resource to gather people’s pictures but as for landmarks, etc.. I’m sure the collection of pictures can give the engines a significant amount of information to make accurate determinations.

  5. I agree about the security and privacy measures but we have a certain responsibility to maintain our own privacy etc. It will be interesting for paintings, buildings and things but I wonder how accurate it will be.

  6. I agree with most of what has been said. It would be a neat feature for things like paintings and landmarks but they would definitely have to incorporate a lot of security and privacy measures.

  7. Implications for privacy? Anti-stalking laws etc? Seems to make the Patriot Act available at a PC near you if you ask me.

  8. I think that we should be responsible for all the pictures that we are posting on the internet. If you have posted pictures on social network sites then it only means that it’s ok for you for other people to see it. On the other hand, people will take advantage of this innovation and can be a way to invade someone’s privacy.

  9. Another privacy concern is that using Google Goggles + Face Search, you could simply take a picture of someone to determine their name, as well as other personal information about them. Image what a world that would be!

  10. Definitely interested in how this matures. I am not entirely convinced of the privacy argument being made by opponents because I think if an image is visible on the web it’s fair game.

  11. Yeah maybe it is a little bit scary, but think about all of the good that could come of this. You see someone special across the cafe, you have a moment when you walk by, but your buddies are ready to leave. You snap a picture of their face, use Google’s app, search for them on facebook, send a message and have lasting love.

  12. Give how much innovation has been driven by porn, makes we wonder if facial recognition is the only physical characteristic that will be usable in searches within the next decade. Putting aside my juvenile attempt at humor, facial recognition searching could be a great tool for residential burglars if they could use their camera to confirm that an individual/family was at a location away from home.

  13. Hi Mike,

    Would it make a difference if the only results someone might see for a facial search are from celebrities whose images are plastered all over the Web, or from public domain images, or from people who are contacts of theirs on social networks that they might be a member of?

  14. Hi John,

    It was interesting that the patent mentioned the possibility that people could potentiall opt-in to a system like this, so that their images could be shown to searches. Often Google (and other search engines) will require you to purposefull opt-out. For example, don’t want a web page indexed – you need to put a statement in your robots.txt file. Don’t want a search engine to show cached copies of your pages, you need to use a noarchive meta tag.

    I think the part of the patent that describes how they would limit results to celebrities, public domain images, and people searchers are connected to in social networks is an attempt to address at least some privacy concerns.

  15. Hi Elliot

    Google acquired a number of patents from Neven Vision a few years ago that included facial recognition and object recognition, and I’ve been wondering if they would do much with those. It looks like they’ve been working on it.

    I’ve been wondering how the public would respond to something like this as well.

  16. Hi WDS,

    I think image search has the potential to really be awesome.

    Imagine taking a picture of a bug, or a plant, or a painting, and being able to do a search on it, and get useful information back. Nice for farmers, for artists, and potentially everyone else as well.

  17. Hi Robyn,

    The facial recognition aspect of this does make you think that Google has to be careful about what information that they share.

    I mentioned a few unpublished patent filings in my conclusion as well, and a couple of those include location information. Imagine having a picture, not knowing where it was taken, and being able to find out based upon the buildings and background images in the photo. There may be some concern about privacy there as well.

  18. Hi Carly,

    I’m guess that the reason why we’re not seeing all the different aspects of this kind of image search is that accuracy is still something being worked upon. I’m guessing that if and when Google launches these different types of search, that it will be a work in progress for a long while.

  19. Hi Matthew,

    At this point, the more troubling aspects of Google’s visual search still haven’t been released. Google Goggle’s allows you to do things like search for artwork or barcodes or product names. The facial recognition part, and the local information part still aren’t here yet. I think those are the parts that might have the biggest privacy concerns. We’re likely going to have to wait to see what Google does to address those.

  20. Hi Ricky,

    But what about when someone else takes a picture of you (with or without your knowledge), and they post those on the Web without telling you?

  21. Hi Haris,

    I think it could cause problems. I think that’s why your results might be limited by your social network connections for many of the pictures Google might return, much like the social search results that Google shows you when you’re signed in are limited to people you’re connected to.

  22. Hi Franko,

    As I responded to another comment, sometimes other people publish pictures of you on the Web without your knowledge. What do you do then?

  23. Hi Vince,

    I agree – it’s going to be interesting to see how visual search matures. I do expect that we will see some privacy lawsuits as well.

  24. Hi Randy,

    Google’s safe search setting seems to be able to keep a lot of images out that people might be concerned about.

    I’m not so sure that facial search is as potentially risky as location based services like foursquare. As you’re becoming the mayor of a convenience store, someone else might be making themselves at home in your house. :(

  25. Interesting post. I’ve been wondering when a tool like this might become available. Could be very handy. As for the privacy end of it, perhaps one more reason to keep those party hardy pics off social networking sites.

  26. Wow Bill, this is right out of James Bond movie! Will be interested to see the privacy aspects of this. People think Facebook is challenging the laws now, this will be very interesting.

  27. Pretty interesting, I’m wondering why some have concerns with privacy. How is it different from people searching your name in Google Search or Images. If it’s available on the web it will only be another way to find about it. Don’t let the pictures get in there in the first place if you don’t want to be found…
    Can’t wait to see how Google will bring all these details together and do their launch!

  28. Hi jcee,

    The facial recognition part of Google’s visual search isn’t available yet, but if we see it sometime in the future, chances are that we will see it on the Google Goggle’s page.

    We know that in Google’s Picasa, you can have the software index pictures by person, with the software recognizing people’s faces in those photos. That capability has been around for a while. Bringing that technology to a much larger body of images is a bigger challenge, but likely within Google’s ability.

  29. Hi Jamie,

    There is a connotation that comes with the term “facial recognition” that draws up thrillers and mysteries and legal dramas, and even movies like 1984.

    I think there are going to be people who will be very opposed to enabling technology like this, and that might be one of the things that is holding Google back from adding this kind of functionality to their visual search.

    Interesting that you mention Facebook. Google would love to have more access to Facebook’s data, and Facebook is keeping them from it. That might be another thing holding Google back from providing a face search at this time.

  30. Hi Charles,

    I guess that it’s one thing to be able to see images of people when those are mostly unorganized on the Web and require that people do some work to find them, and another thing when someone makes it much easier, and provides a way to take an image and do a search on that alone. It’s something that we sometimes see in crime dramas on television, and that we may not expect to be able to do in our own homes.

  31. This has FAR reaching implications beyond simply social network search. I wonder now how this will affect the so called “people search” engines out there as well like Intelius, etc..

  32. Imagine if something like this was applied to apprehending criminals. If you saw someone suspicious, you could snap their photo and transmit the location and image to a police department which could filter the image, identify the person and if they are indeed wanted for a crime, send units to apprehend them. I know this sounds a lot like Minority Report or science fiction but this could actually happen especially if the investigation was done passively through law enforcement monitoring mobile uploads to social networks.

  33. Pingback: » Pandia Search Engine News Wrap-up February 27
  34. Ewwwwww – that creeps me out. I’m not usually concerned about privacy. I figure what you say online is public, deal with it. But if someone else can take my photo and Google can identify it as me and publicize it – that’s stalkerish.

    Making up my own words today.

  35. Hi Craig,

    While this could possibly use social signals, it isn’t intended as part of social network searching – it’s part of a much different visual search. I don’t know how it would impact people search engines, since those don’t seem to offer a service like this, where you can search on an image. It’s possible that it could increase their usage, though.

  36. Hi Will,

    Many law enforcement agencies do use facial recognition software, though I don’t think they look to images found on the Web as part of the databases used to conduct those searches. I could imagine that police and other investigators might be interested in using a system like this.

  37. Hi Juliemarg,

    I think there’s an element to offering facial recognition to the public on the Web that’s going to bother a lot of people. Will Google make this available, or might it bother too many people for them to even consider it. I’m not sure.

  38. I find this extremely interesting! But I agree with most with the comments concerning security and privacy. Although this is a cool concept and may not be a big deal when recognizing places/landmarks, historical figures, or celebrities I am not sure how I feel about it recognizing everyone using the internet. Granted it was mentioned that you would most likely need to be part of a social network for your image to appear, but the concept still creates lingering concerns.

    Interesting post and I look forward to hear more details regarding this issue!


  39. Hi Dustin,

    I suspect that we will see Google implementing face search at some point, but there does seem to be a great amount of potential for concern. I’ll be keeping an eye out.

  40. It is amazing how technology is evolving. However, it does open up a lot of debate on privacy issues. I heard on the radio today that Chicago just passed a new law allowing people to take pictures and videos and text them to 911 to catch criminals…

  41. Hi Genna,

    Interesting law. I was reading the newly revamped streetviews pages from Google, and one of the things that they stressed in their privacy section was that “Street View contains imagery that is no different from what you might see driving or walking down the street.” In other words, the streetview images were taken in public areas, with public views. They also stress there that license plates and the identities of people are obscured, and “You can request further blurring of images.”

    Technology is creating new challenges to privacy, and our notions of privacy are likely changing in some ways in response to that technology too.

  42. It’s apps like this from google which scares me lol. This could be sold as a great concept but could be used for all the wrong things by the wrong people.

  43. Hi Craig,

    There have been a couple of news articles in the past few days about Google and facial recognition software where Google has stated that it scares them somewhat too. :)

Comments are closed.