A few months ago, I stopped at a store to search for a new phone. The place I stopped at offered a couple of dozen different models of phones with different features, and I narrowed my search down to three or four that listed features on their boxes that sounded interesting. Usually, I’ll look online before buying something like this, but I needed a new phone, and wanted to get one quickly.
I pulled out my mobile phone in the middle of the shopping aisle and started to search for reviews of the choices of phones in front of me. It would have been great if I could just take a picture of them and had more information about them come up automatically, including reviews and alternative prices elsewhere, to compare costs as well.
A patent application from Google describes a way of using a mobile phone to take pictures of items and sending them to the search engine to have it search through Google Product Search as well as other web sites, to find prices and reviews and other information about those items, and make a purchase online if you would like to. The images could be of a shirt that a friend is wearing, or a bicycle that you see parked on the street, or a package on a store shelf.
The search could be based upon the actual image itself and words that might appear on a box for the image or other information. The information that you receive could include such things as technical specifications, nutritional value for food items, country of origin, prices from various vendors, and more. If the phone was GPS enabled, Google might see that I was in the middle of a specific store and look up the store’s online catalog to show me that item and other items offered by the store.
In addition to mentioning Google Product Search, the patent filing describes how Google Checkout could be used to allow people to buy products they see as part of this process as well. However, it tells us that neither Product Search nor Checkout is essential. People could be brought to the websites of vendors who might offer the products in question, and could use the ordering processes on those sites as well.
The process would start out by you taking a picture with your camera phone, or that you’ve uploaded to your computer or that someone sent to you electronically, or that you’ve found on a web site.
The image is sent to the search engine, along with text on a package, if available, where matches could be searched for at Google product search or elsewhere on the Web.
You may be shown the same item, or items that contain the same features or which are related, as well as related items such as accessories.
Information about those items are sent back to your phone, or your desktop computer, where you can decide whether you want to purchase the items.
It’s possible that you could use Google Checkout to make a purchase, though the patent filing mentions that purchases could also possibly be made upon vendors’ sites as well.
The patent filing is:
Image Capture for Purchases
(Note: the USPTO link appears to be broken, but the patent filing is still available through WIPO – Image Capture for Purchases)
Invented by Jian Yuan and Yushi Jing
US Patent Application 20090319388
Published December 24, 2009
Filed: June 20, 2008
Abstract
A computer-implemented item identification method that includes identifying an item in an image received from a remote electronic device; transmitting search results containing information about the item for one or more vendors of the item; and, transmitting to the remote device code for executing an order for the item from the one or more vendors of the item.
Actual Image Search
The image that you take with your camera and send to Google would be compared to other images, and “feature points,” or areas in a picture where there are sudden changes in pixel colors or in the brightness of the image, might be analyzed to come up with what might be considered a “sort of digital line drawing of the object in the image.” Your image might be compared to other images collected by the search engine to find a match.
The search engine might have collected other information about products, such as manufacturer names and model names, like the kind of information found at Google Product Search.
That kind of information may have been gathered by the search engine in a number of ways, such as:
- By the search engine directly from vendors (e.g., by vendors submitting data in a pre-approved format),
- Manually (e.g., by agents copying information from vendor pages),
- Semi-manually (e.g., by agents marking up portions of pages and automatic systems extracting product data from similarly-formatted pages), or;
- Automatically (e.g., by a search engine crawler program trained to recognize product and pricing data).
The search engine might attempt to show close matches, which might include the same model number and brand as the items shown in images, or which have the same features but aren’t the same brand. It might show similar matches, which could be the same brand but a different model, or related items – such as a headphone case when the item shown is a set of headphones.
The patent filing describes some of the different technologies that might be used to capture images and information about items that a person sends a picture of, including image technologies that recognize different objects, and Optical Character Recognition, for products that have written upon them, or that are within boxes that mention the name and possibly the product numbers for those items. Barcodes, Universal Product Codes, or ISBNs from books could also help identify products.
The images that can be used in this system might be from something that you see and take a picture of in person, or through the media such as on a television show or in a magazine, or from a picture message that someone sends to you, or from an image that you see on the Web, and right-click upon to “learn more about an item in the image.”
An interesting example from the patent filing – take a picture of a “chocolate cupcake with chocolate ganache and buttercream frosting,” and the search engine will attempt to find a match, and tell you where you might be able to buy that cupcake:
At an initial step, the process receives an image. For example, an image received through a picture message can be received. The image can be a specialty item that the user would like to buy. In some examples, the image can show a chocolate cupcake with chocolate ganache and buttercream frosting.
The process then identifies feature points in the image. For example, the feature points on the cupcake wrapper can be determined from its accordion shape using the shadows and lights in the image. The color palette of the image can also be used to determine potential matching flavors in the cupcake (e.g., chocolate versus lemon). A preliminary check can also be made at this point to determine if a particular item can be found in the image–for example, if the image is very out of focus, a contiguous group of points may not be found, and the user can be told to submit a better image.
If you offer products on the web through an eCommerce site, this cupcake example may offer some helpful suggestions to make it more likely that a product from your site may show up in a search like this. Ideally, it seems that you would want your products to show up in Google’s product search as a first step. But, if you offer products that may look different at different angles (say shoes instead of art prints), you may want to include more than one image at different angles. You may also want to choose good names for your images as well:
The process then compares the image to an image library. For example, the cupcake image can match an East End Chocolate Stout cupcake from Dozen Cupcakes in Pittsburgh. The image can also match a cookies-and-creme cupcake from Coco’s Cupcakes in Pittsburgh. In other implementations, comparisons can be determined through a naming filter. For example, if the image file has a name, such as “cupcake,” the image library can be filtered to images having cupcakes in them. More likely, because various cupcakes are not very distinct from each other, the image could simply match an image associated with the tag “chocolate cupcake” or “lemon cupcake” or the like, and not a particular brand of the cupcake.
Google Goggles and Visual Search
Google just recently released into Google Labs a kind of visual search they’ve named Google Googles, which lets you use a picture to perform a search. The search engine describes how this visual search can be used with landmarks, books, contact information (such as business cards), artworks, places, wine, and logos. The Google Goggles Help pages also mention barcodes and products as things you can take pictures of and search for:
Remember that Google Goggles works best on books & DVDs, landmarks, logos, contact info, artwork, businesses, products, barcodes, or text. Right now, it’s not so good at pictures of animals, plants, cars, furniture, or apparel.
In the “book” example, Google provides links to let you “compare prices at Google Product Search,” and “Preview this Book at Google Book Search.” That seems like a good starting point from Google towards letting you search for, and buy other items online as well.
Conclusion
With Google Goggles, this kind of product search seems to have a good start. Google Goggles has some limitations, such as requiring a device running Android 1.6 or above, with a QVGA, and an autofocus camera, but it’s possible that Google might make a visual item search available on other devices as well, including searches through a desktop computer for pictures you upload or find online or that are sent to you.
It’s possible that this kind of visual search could become very popular. If you run an eCommerce store, you may want to take steps to make it more likely that your images can show up in matches for this kind of visual search.
Thats crazy! Just shows how much technology has come on these days! It could make life easier when shops are really busy but it could just make us a lot lazier than we already are…
Bill, as a developer of websites and e-commerce stores, I think the ability of Google to match an image with the images stored in their database is potentially huge. It can mean more sales to some clients. But the bugs might not make the process work perfectly.
I’m glad to see Google “pushing the envelope”.
Cheers!
I use Picasa (Google’s photo management software) which has a new feature that was released in the last update that scans all of your images for faces and groups them together so you can label them all at once. Essentially, this allows you to easily organize all of you photos without having to go through and label each one separately.
They would probably use a similar technology for the service you described above. Pretty cool.
Hi Gregg,
I was pretty excited when I came across the patent. I think it’s potentially pretty huge as well. If ecommerce site owners haven’t pursued making sure that they are listed in Google Product Search, and haven’t worked on making sure their pages are as search engine friendly as possible, they may be missing an opportunity here.
As the quote from the Google Goggles page I included in the post tells us, they still have some bugs to overcome – but it looks like people with phones using Android version 1.6 can use a search like this for books, DVDs, and other products.
Hi G Money,
I do think they are using a similar technology. The facial recognition technology you describe in Picasa is based upon research from Neven Vision, which Google acquired a few years ago. In addition to publishing patents about facial recognition, Neven Vision also have done a considerable amount of research on object recognition.
I didn’t go into too much depth on the actual image search technologies mentioned in the patent filing, but it does cite some sources of technology that Google might use. But yes, some of their expertise in the area does come from the same source as the feature found in Picasa.
Hi James,
It might make us lazier, but then again, it might make us all better at comparison shopping, and more involved in researching the purchases we make. 🙂
That would be great! I think I saw this kind of app in an iPhone. But instead it gives you information about a particular place or location that you point in your video. Isn’t that somewhat the same?
Indeed this is interesting feature, I am just thinking what the regular retails shops are going to do about this?
Will they let everybody run around taking photos inside their shops?
Google are moving on to more and more areas, and as I see it they are starting competing with there customers, will they be allowed to do that?
Morten
Hi Henry,
The iPhone does offer a number of applications that can give you information about images that you upload, but that’s a little different than this. This is more tied to a visual search that can help you find products that you might want to buy, either in Google Product Search, or elsewhere on the Web. The patent does mention that they might try to identify your location through something like GPS, so that if for instance, you are in the middle of a Target store, they might show you the online catalog for Target, and the product that you’ve taken a picture of there.
I did write about Google’s local search barcode project recently, in a post titled Google Barcodes and Place Rank Transforming Local Search. In that post, I’ve written about a patent filing from Google that would let you take a picture of a barcode associated with a business, or a parking lot, a landmark, and other places where people might place barcodes, and find out more information about that place.
Google Goggles looks like it might allow you to take pictures of landmarks and places, and learn more about them without the need for a barcode, which is even closer to the iPhone apps that you’re writing about.
Hi Morten,
I’ve been wondering if we might see stores chasing away people who pull out cameras to take pictures of items on shelves. I know that some places like jewelry stores tend to be a little paranoid about cameras, thinking that there might be a security risk associated with pictures taken of their property. I was yelled at a couple of years ago by a barista at a Starbucks for trying to take a picture inside their coffee house. I’m not sure how most stores will react.
It’s possible to frame something like this as Google competing with their customers, but Google isn’t selling any of these products directly. The patent filing does describe the use of Google Product Search and Google Checkout potentially being part of the process involved, though it also mentions that Google could deliver shoppers directly to vendor’s sites online and that those shoppers could use the checkout process at those vendors’ sites. It does have the potential to lead consumers to order online instead of buying from the store they are in, especially if they come across better prices, and similar products that might have better features and/or reviews.
So yes, this could be seen as creating competition, but it could also be seen as creating some opportunities for people who might be willing to take advantage of it.
That’s huge! I have heard of being able to take a pic of UPC code or similar mark and being able to pull up all the pertinent info (specs, reviews, compare prices, etc), but being able to do it for any item with a picture opens a lot of doors. I really like the idea you’ve mentioned of having GPS enabled phone and having local search and comparison with vendors within your area. So many opportunities! Thanks for a great post, and Happy Holidays! – Troy
This would not surprise me at all. This would really change the way people optimize images in search. Probably get some much more relevant images showing up in search results.
Hi Bill, thanks for the article. I guess I don’t see the “BIG DEAL” here…at least based on the fact that consumers are already turning to the ‘Net for their shopping needs. If you’re already doing most of your product research and shopping on-line, submitting an image is no faster than submitting a text string.
Here’s where I see object recognition coming in handy: A) Bricks and mortar impulse items. B) Finding alternate locations for small-ticket items you’re attempting to buy locally but are out-of-stock at the current store you’re in. C) Running into new and interesting items in the non-virtual world (which doesn’t necessarily imply purchase intent).
In any case, I wouldn’t characterize any of these actions as cornerstones of consumer shopping/purchasing behavior but who knows; those that like to shop for shopping’s sake may just think this is the greatest thing since the “Priceless” ad campaign. 8^P
Hi Troy,
I’ve heard of a number of experiments with camera phones in places like Japan, with barcodes and even with images alone, to provide information to searchers. I’m not sure how similar they might be to what Google is proposing with this patent filing. But, merging local search and product search with a visual search like this is pretty exciting, and I think it does have the potential to transform the way we shop, and the way we find information.
Hope your holidays were good ones!
Hi Nick,
I think we will see a better visual search from Google based upon many similar ideas. Don’t know if you caught this white paper from Google, but it’s worth a look:
PageRank for Product Image Search.
Google still has some work to do in this area, but the release of Google Goggles seems to indicate they they’ve been making some strides. I think it will make a difference in how images end up ranking in search results, and how images can affect the organic rankings of web pages.
Hi Dave,
I think the thing that attracts me to what Google is presenting in this patent is the chance to shop offline, and make meaningful and informed decisions through visual searches on your phone while you are shopping. Most stores only carry so many choices of products, and a search on your phone may reinforce your decision to buy what’s in front of you, or to choose an alternative that you might find when faced with the decision to purchase.
If the ability to do that search comes down to pushing a button to take a picture, it’s more likely that people will. It’s pretty clunky to try to do a text search on a small phone, and navigate through a number of different web sites looking for reviews. Making that process considerably easier could potentially have a big impact – I suspect that we will have a chance to find out sometime in the near future.
@Bill
i agree, its for sure going to be interesting to see how different parties will react, when the service is roled out, i am sure you will have reaction in both directions.
brgds, Morten
Hi Morten,
Thanks. Google Goggles seems to be a good start – I’m wondering if the process described in this patent filingwill evolve out of what Google has done with that so far.
This would be really good for comparing prices on things. This would be great for people who sell on ebay. They would be able to see what an item is currently going for online and determine if it would be worth reselling.
That would be great! This would really change the way people optimize images in search.
ehm… the link to the patent returns `patent not found’.
search in google patents is also `not found’.
I wonder why it was removed….
Thanks to google I could find it here: http://www.faqs.org/patents/app/20090319388
This is kind of a big deal. There was a few iphone apps floating around, but i had heard a few of them were corrupting them, how true that is i do not know.
Hi People Finder,
I’m a big believer in comparison shopping, and if Google does move forward with this process, I would be more than happy to use it that way.
As for people using this to compare prices, such as sellers on eBay, I’m all for this kind of business intelligence as well. You go to a garage sale, and see something that you think looks interesting – why not take a picture and see if Google can tell you something about it?
Hi NiceBlogger,
I agree with you completely – I think as Google gets a better grasp on indentifying images based upon what they contain rather than upon the alt text and captions and other content on a page, it will change around the way that people optimize images in search. I also think it would be beneficial for many ecommerce site owners to include more than one image for many products, taken at different angles.
Hi Oren,
Thanks for the heads up, and the new link.
I’m wondering why it was removed, too. I did find a link to it at WIPO as well, and included that in my post.
Hi Dan,
It does seem like some competition between Google and Apple is starting to appear. We do have a good idea of why Google’s CEO, Eric Schmidt, resigned from the Apple Board of Directors now.
Interesting concept bill, thanks for sharing!
You’re welcome, Keith.
I actually saw something about this on tv not too long ago(i think it was cnbc). They were showing how you could take a picture of something like the golden gate bridge, and if everything goes according to plan, it will pull up what it was you took a picture of(the GGB), and all kinds of information about it.
Hi Mike,
Wish I had seen that.
I think there are a few places that offer something like this. Google does at Google Goggles The offline world and the online world are merging closer and closer together.
Just came across this post. It’s been a while and Google still hasn’t released this technology. Any word on when its coming out?
Hi Oliver,
Google Goggles seems like the place where we might see this implemented, but I can’t give you any specific dates.
Thanks for the response Bill! I have really seen this technology take off yet. I think it’s going to be awesome when it does! I quickly see google trying to get into commerce (ie. the attempt to buy Groupon), upgrades in their “products” search.. Got to love innovation! thanks for the reply.
Hi Oliver,
You’re welcome. I’ve been finding myself looking up more and more products while looking at them in the stores I’m shopping at, to see if I can find reviews. It would be a lot easier if I could just take a picture of a barcode. I think we’ll see more people doing that in the future.