New Visual Search Photo Features from Google

Sharing is caring!

These details come from an anonymous source who gave us more details on the project. The report states a new feature will be integrated, allowing users to outline specific image areas to target their searches.

Added 1/13/2020 – Google Goggles no longer exists. Still, many of the features that it had, and might have had added to it, such as the ability to focus upon objects within a photo, are features that are included in Google Lens. I am updating this post to point to that app instead of Google Goggles.

In Google Goggles, one can only search the whole image, which has proven to bring plenty of discrepancies. In addition, images often display plenty of distractions, background items, and other objects that may throw off a search result. According to the sketch provided, the system will also recommend retailers for purchasing products and other details.

Furthermore, it is said this technology has also been tested in “wearable computing devices.” This could suggest this technology may come to products like Google Glass and possibly even VR (or AR) headsets.

Google Camera app rumored to get Google Goggles functionality

Today, we’re to understand – from an anonymous source, as it were – that Google may soon be releasing a new camera feature they’ve had in testing for some time. This feature would allow the user to use their standard Google Camera app to search for information based on what they’re able to see – but more than that.

Google is building a better Android camera that can search what it sees

Details given to Android Authority through an anonymous source state that the integration would include a new feature “allowing users to outline specific areas of the image” to target their searches directly. This would improve the current iteration of Google Goggles, which only allows you to search whole images.

Google may reportedly add augmented reality support to its camera app

I’ve noticed some stories about new visual search features available through Google leaked out to the media from an anonymous source. And I also noticed a patent application filed by Google, which was just published. It contains at least one image that is the same as one supplied to Android Authority, which makes me wonder if the anonymous source contacted them was from Google.

The patent filing does describe how a person might take an image from a photograph and outline part of that image with a stylus or a finger, and have search features available to them like those available from within Google Goggles.

Back in 2011, I wrote the post The Future of Google’s Visual Phone Search?, which supposedly would offer the ability to provide mixtures of searches simultaneously on an image, of the types offered through the Google Goggles App. That type of rich visual search never materialized from Google. The types of search under that richer visual search patent filing from 2011, included:

  • A facial recognition search
  • An OCR search for text in the image
  • An image-to-terms search system, which may use object recognition
  • A product recognition search, which could recognize two dimensional images such as book covers and CDs,and three dimensional images such as furniture
  • A bar code recognition search
  • A named entity recognition search, which could provide information about specific people, places, and things
  • A landmark recognition search, recognizing actual landmarks and possibly images advertised on billboards
  • A place recognition search that might be aided by geo-location information provided by something like a GPS receiver
  • A color recognition search, and
  • A similar image search, which looks for images similar to the one that you’ve used as a query

A wearable device that could be Google Glass might use a camera to watch an object from a photo be outlined, and that outlining could trigger a visual search. Based on the image outlined, the search could be one of these specialized searches that we see from Google Goggles. So you would ideally take a photograph and then outline an object within the image, and that outlining would be captured by your wearable device.

The patent filing that provides more details on this rumored visual search is:

Object Outlining to Initiate a Visual Search
Filed: 04.12.2015
Published: 24.03.2016
Inventors: Thad Eugene Starner, Irfan Essa

Methods and devices for initiating a search of an object are disclosed. In one embodiment, a method is disclosed that includes receiving sensor data from a sensor on a wearable computing device and, based on the sensor data, detecting a movement that defines an outline of an area in the sensor data. The method further includes identifying an object located in the area and initiating a search on the object. In another embodiment, a server is disclosed that includes an interface configured to receive sensor data from a sensor on a wearable computing device, at least one processor, and data storage comprising instructions executable by at least one processor to detect, based on the sensor data, a movement that defines an outline of an area in the sensor data, identify an object that is located in the area, and initiate a search on the object.

Thad Starner is, among other things, a technical lead manager for Google Glass and the director for the Contextual Computing Group at Georgia Tech Ifran Essa teaches at the Georgia Tech Institute for Robotics and Intelligent Machines and does Research on Video Analysis and Enhancement for Google.

Take Aways

The patent tells us that the advantage of the approach in this patent is that outlining the object can speed up the search, and “computing power may not be expended to perform searches in which the user is not interested.”

That makes it sound like it is a reaction to the multiple searches for the patent I wrote about in 2011.

I’ve used Google Goggles before, and it can be a little tricky to take a picture of one object for the app to function upon. And you may want to search for something in a photo that you’ve taken that you wanted to capture as part of a photograph, rather than using Google Goggles. And you may want to save that photo as a photo, which I don’t think you can do with Google Goggles.

This appears to require that someone use a wearable such as Glass (It hints that other wearables could be used) along with a camera from a smartphone to perform visual searches. It’s the first patent that I’ve seen from Google that requires a combined use such as that – it’s possible that there have been others, but I didn’t see them. Given how popular phones with cameras are these days, I don’t think it’s a problem that combines multiple devices like this. It seems to make Google Glass have more uses. For example, I suspect that you can outlines objects on images that you didn’t photograph as well.

Sharing is caring!

20 thoughts on “New Visual Search Photo Features from Google”

  1. Hello! I could have sworn I’ve visited this web site
    before but after browsing through some of the articles I realized it’s new to me. Anyhow, I’m definitely pleased I found it and I’ll be bookmarking it and checking back often!

  2. If Google Goggles works by breaking a photograph into several parts and then compare them with its vast database and returns relevant search results for image content, then it is possible that the new function is to decompose the image and isolating it from its surroundings. The problem, and not small, is like getting an interface that can interact with the device. The possibilities are wonderful …

  3. Hi Huberto,

    The patent I wrote about in 2011 tried to break a photo down into different parts – that process was never released to the public, and it may have been too hard to do. The process described in this patent simplifies things greatly, by limiting what images are being processed. I’m hoping we see it released to the public. I agree, the possibilities are wonderful.

  4. Thanks for sharing. I hope it will be released. Don’t care when, but I want to try it so badly

  5. Hi Bill? Thanks for sharing this post. By the way I enjoyed your interview on RainmakerFM I came across on Foundation Digital about RankBrain. Anyway just to add to this topic, I read on NDTV that Google camera App is to get Google goggles features. The news has also come up on Android authority and eWeek. I think that would be very interesting and would make the App more useful.

  6. Im sure fashion websites have been breaking down images for years, so when you you hover over part of the image it takes you to a section of the site where you can purchase that item of clothing. Granted this isnt searching a database its just hyperlinking but cool nonetheless

    great article by the way.

  7. Hi Ricky,

    What you are writing about sounds like something different – a way of using different areas of an image to link with – that has been part of HTML for a while. This new feature hinted at by a patent involves an actual search on part of the image. It would be pretty interesting.

  8. wow!!! that is a good efforts by google!Thanks for providing such specified & detailed information.
    Looking further for more knowledgable articles like this.

  9. I can see numerous possibilities in searching portions of photos in my work (architect design and construction) and I am sure there are countless other industries, areas of study or just person use in which Google’s new search will be useful. The bugs to work through must be daunting though. Great post in an overall great block. Thanks for the time spent!

  10. Hi Len,

    Thanks for explaining why you would find value in this approach to photos! I was a little surprised to have come across this patent, after having read about the rumor of this process. It seems like Google is being more careful with information being released about new features available through Google Glass with this newer version of it having come out. It will be interesting seeing it released.

  11. Great ideas for photo features from Google! Thanks for providing such kind of information.

  12. Thank you for sharing the information. I hope it will be released. It does sound pretty exciting.

  13. Hi Greg,

    The patent doesn’t go into a lot of detail on how they would implement a number of aspects of the process described in this patent. It would be exciting to see it put into place, though.

  14. Thanks for sharing The informational post with us.

    I think this is very good efforts by google.I do believe it will be released very soon.Thanks for providing such kind of information with us.

  15. Hey Bill,
    A great post! Thanks for sharing. Its really a cool and amazing feature. Waiting for the release.

Comments are closed.