Voice Queries and Visual Queries and Automated Assistants
I’m on the second day of a trip to New York City, giving presentations at SMX East on both the potential impact of mobile devices to the future of search, and on how reputation and authority signals might impact the rankings and visibility of authors and publishers and commentators on the Web.
My first presentation was in the “local and mobile” mobile track of the conference as part of a session titled “Meet Siri: Apple’s Google Killer?” where I joined Bryson Meunier, Will Scott, Andrew Shotland, and moderator Greg Sterling in discussing the potential impact of Apple’s Siri and voice search on SEO and search.
When I read the title for this proposed session a couple of months back, I couldn’t help but start to draft a pitch to join in on the conversation. I’ve been carefully watching patents and papers from Google and Apple and others about inventions and interfaces that might transform the way we search in the future to one focusing more upon voice queries and visual queries and also transforming the way that people might share information and market businesses online.
Is Siri Smarter Than We Perceive it to Be?
The Siri we see now on the iPhone 5 and the iPhone 4S only hints at some of the potential capabilities of voice queries via this intelligent automated assistant. Apple’s name is on 2 patents at this point that describe what other features we might see in the future. Both patents are immense, at around 150 printed pages or so each.
I wrote about Apple’s Siri patent this January. A continuation patent sharing the same name and substantially the same description was published by the US Patent and Trademark Office (USPTO) last week – which sent many tech blogs into a frenzy of speculation about what it might mean. None of them actually looked at the claims section of the newer version, which was where the two differed, with the more recent providing more detail on how Siri might better take advantage of the context of searches.
At the heart of Siri is an Intelligent Ontology that attempts to understand the intent behind searches by understanding the domain or or topic of a search, tasks that might be associated with that topic, phrases and terms that might help Siri acts upon those tasks, information about entities that may also be related, vocabulary that can expand upon the topic further, and information and service providers who may be of assistance.
For example, someone may ask Siri about places to get Italian food nearby for dinner. Siri can search and provide some suggestions and may be able to directly book reservations if so instructed. Siri can find the names of possible restaurants, knows the difference between Italian food and Korean BBQ, and could work with services like OpenTable to make reservations. It takes a wide range of vocabulary, task management, and ability to interact to make those types of decisions.
My presentation wasn’t intended to say bad things about Siri, but rather to look at the evolution of new approaches from Google that seem to be a response. I’m excited to see how Siri continues to develop in the future, and I’m hoping that its growth inspires new innovation from places like Google.
Google Glass, Google Now, and Google Field Trip
Google’s Project Glass hasn’t brought us products that have been released to the public yet, but it is something that is potentially very transformative in that it can receive not only supports voice queries but also supports visual queries of the type seen at Google Goggles.
Google is also showing us aspects of their own automated intelligent Assistant with Google Now, and Google Field Trip.
I pulled an example from Google’s patent Providing digital content based on expected user behavior to describe on the difference that I see between Google Now and Siri, based on a reading of the patents from both.
Google Now and Siri may both recognize that you like to go to the local baseball stadium every so often. Siri may determine that you like baseball, and may start showing you the scores for the local team. Google Now may also determine that you like baseball, but notice that you only go to games against a certain competitor of the local team, and start showing you scores for that competitor.
Google Now and Google Field Trip both show how advertisements and offers might make their way on the mobile screen. Google now by doing things like learning your commute to work, so that it can warn you about traffic congestion while you still have a chance to plot an alternative route, and learning when you detour to a local coffee house on certain days, so that it might show you coupons for that coffee house, or others near it. Google Field Trip allows you to decide about what kind of content you might see, including historical marker data, locations where movies were filmed, and even advertisements for local businesses as you get near to them.
Google Acquisitions and Alternative Interfaces
I take a detour in the patent to show off some of the patent acquisitions that Google has in the past couple of years that would support the use of Google’s Project Glass. One set includes Terahop patents that would make it easier for Google to know the location of people indoors, including large indoor spaces like shopping malls and airports.
Google also recently acquired a number of patents from deCarta, which deCarta (then using the name Tecontar) acquired in the early 2000s when they acquired a company named Gravitate.
The company appears to have been ahead of its time, focusing upon the use of social, local, and mobile technologies to connect people near each other, connect people who might have common interests near one another, to set off alerts based upon a location for advertisements and for location-based events. These are all capabilities that Would appear to be ideal for Android and Google Glasses.
A number of patents from Google directly about the kind of heads up displays at the heart of Project Glass describe interfaces such as a simple touch screen on the side of the glasses, a user interface that changes based upon your activities. It would be simpler and sparser while you were running or jogging or driving, and more detailed while you were walking, and even more so while you are standing still or seated.
Another shows markers that you can wear on your fingers, such as rings, or gloves that have reflective tape on them, that you could use instead of a mouse or keyboard to enter information into a computer system or onto the Web. This goes beyond voice queries, to ones supported by gestures.
Another patent from Google describes a predictive application selector that might track your iris movements and remember your usage of applications in the past to determine which ones to launch from a heads up display.
Google has also acquired some patents that might not be used with Google Glass, but point out the company’s ability to take risks and explore other options. For example, Google acquired a number of patents from Louis Rosenberg that include a Harry-Potter styled wand game controller that works with voice commands that the pointing of a wand, or media devices that you can turn on and off with just a glance at them.
Google may also potentially view inventor Ralph Osterhout as a possible competitor. I’ve noticed a large number of patent applications being published at the USPTO recently with his name on them that involve augmented reality glasses that can run applications like Gmail, Google Earth, see virtual signs and advertisements on the sides of buildings, see-through fog and smoke, and much more.
Conclusion
Siri isn’t a Google Killer, as the title of the SMX session suggests, but it’s definitely something that is Voice Queries and Visual Queries The Future of Search? with Google in interesting ways.
If you spend some time working upon how voice queries and visual queries might impact search and SEO, and how the different parts of an active ontology from Apple or predictive models from Google might transform marketing on the Web, you might be a step or two ahead of others.
I present this afternoon on how signals of authority might be used by search engines to rank content in search results. It’s another technology from the search engines that may be future-focused but has the potential to have a substantial impact on what we see from the search engines.
Bill, great to meet you yesterday as I’ve enjoyed your writing for a while. I agree that spoken and visual queries are where search is headed, and for many of us who use Google Goggles, Field Trip and Google Now regularly it’s where search is today.
As I wrote on my blog earlier, the whole discussion reminds me of the quotes from Sergey Brin and Eric Schmidt about wanting Google to be the “third half of your brain”. You mentioned a work of fiction that explores these changes, but there’s a similar work by theoretical physicist Michio Kaku called Physics of the Future in which he describes technology in the next 100 years based on prototypes that exist today. One of those technologies allows us to control connected objects using only our minds. If you’re interested in this kind of stuff it’s definitely worth a read.
Again, great meeting you. Keep in touch.
Best,
Bryson
When spoken search queries become more popular, I’m going to primarily focus on ranking for guttural sounds, conversions won’t be there, but I’ll build up brand recognition for throat lozenge companies that way aiming towards harsh speakers.
@Brent that’s a good approach to the long tail.
Spoken queries are a natural next step, but mostly only mobile. I can’t imagine an office full of people vocalising their searches (which is possible now), especially for certain types of query. However, if it becomes the norm for mobile searches to be spoken then a trend may develop for certain types of short query to be vocal. Perhaps short conversion-likely searches – “buy coffee beans”, or local business searches – as opposed to informational searches (e.g. “google chrome show in folder not working”).
I can’t wait to turn things on and off by looking at them, however to be bombarded with manipulative marketing tactics constantly does not sound to healthy. Sounds like they are planning to fill my day and brain every second they can, if allowed.
Bill, great post as always. I was especially intrigued with the mention you mad of Terahop patents:
“One set includes Terahop patents that would make it easier for Google to know the location of people indoors, including large indoor spaces like shopping malls and airports.”
It grabbed me a bit as I follow Nokia closely since they are such a large patent holder, and I believe they have some of the most advanced patents related to identifying the location of individuals indoors. Curious if you know of Microsoft acquiring any similar patents or not? It makes me wonder if Microsoft may move to acquire these types of patents at some point in the future if they don’t already have them.
I enjoy your take on thinking of search more from a technological perspective rather than a marketing focused perspective. Both are crucial I believe (and sadly many don’t think this way) to turly understanding how to leverage search for marketing (if that’s your goal).
Thanks again for the work and look forward to checking out your presentation on authority signals
Wow, I can’t wait to do SEO for pixel searches (anxiety slowly creeping in).
I came to this realization a few months ago, that if Siri worked, it would be a legitimate threat to search engines, allowing people to jump past questions and go straight to answers.
We rely on search engines now because they’re not very good, and need to present options. Sort of like a human operator in a driverless car.
But eventually, the voice queries will get so good you won’t need search results. You’ll search for a hotel in Vancouver, and your phone will negotiate ferociously and lock down the absolute best deal according to your personal tastes.
This is a topic dear to my heart since I believe voice (assuming we can filter out noise) is the ideal input for a mobile device given the size of human fingers. I was always intrigued that high-level Google reaction to SIRI when it came out was against the artificial intelligence aspect. Their line was that your mobile device should be just that, a communication device, and not a virtual assistant. I’ve not seen any change in that Google opinion, but I think they’re dead wrong. 🙂
Bill this is the seconde article I read and really like what you’re doing.
Didn’t even thought of comparring Apple’s Siri with Googles products (Glass, Now or Field trip). However in all this, I consider a really great thing that there is thow competitor striving to achive the best automatic search there. Google hasn’t let us down, well if we exclude the seo ranking which is a little messed up on some parts), I’m also looking forward to how Siri will evolve and how Google will respond!
Bogdan
I am speechless, really, after reading about all of these new innovations (especially the idea of a search engine that REALLY seems to pay attention to your behavior and gets to know you). I’m really glad I stumbled upon your blog.
I tend to agree with you, however, since I don’t always know what I am looking for online, so how would anyone else?
The world of SEO seems to get more complicated by the second!
Hi Barry,
Good to see you. Google does seem to have changed their opinion regarding intelligent asssistants, or their patents tell a story that the upper level management at Google hasn’t been. I know that Google’s Peter Norvig has stated a number of times that Google hasn’t reached the level of artificial intelligence, but I suspect his notion of what that entails likely goes beyond what a Siri or Google Now might provide.
If you look at the patents Google has produced involving things like Google Now, it’s clear that they’ve joined the camp of providing information on an as needed basis in the way that a Jeeves or a Siri might if either was an intelligent assistant. I also think that the point behind Google Glasses is to provide information in a way that augments a person’s life in an intelligent manner by providing them with useful tools at times when they might be needed.
I hold out little hope of Apple doing a better job on understanding my broad Scots accent and venacular than Google currently does… ken whit a mean like?
@Brent – Given my experiences with Siri you may want to go with expletives, those are typically the secondary search after Siri mucks up your initial one with its terrible voice recognition.
I’ve been using my voice search on my Droid X and comparing it to searches performed for the same phrases on Siri and am consistently happier with the results I get. More often than not, Siri just runs a search anyhow, but fails to recognize the phrase as well as my Droid. Regardless, this is something I haven’t thought much about previously.
Current Siri output frequently does bring about a round of expletives or laughs from the user. There is so much to be done in this field and it is exciting to see Apple and Google trying to deliver the best option, competition at its best and surely benefits for all users down the line.
Loving the competition. I hope it keeps pushing both sides towards building interesting things.
Hi Bill, I think you are spot on in regards to visual search and spoken queries eventually overtaking a considerable part of, it not all of search.
Search is becoming increasingly mobile and hands free. I also tend to think that public and private apps are starting to put a serious dent in traditional online search, and will continue to erode that arena.
Was just at the Microsoft Windows 8 Roadshow and Sony went as far as introducing their own gesture based technology for operating one of their All in one pc.
Definitely a relevant post, voice and gestures are here to stay.
Hi Bill
Very important and Great post. I like this post. If you spend some time working upon how spoken and visual queries might impact search and SEO, and how the different parts of an active ontology from Apple or predictive models from Google might transform marketing on the Web, you might be a step or two ahead of others. Thanks for share it.
I gave up on Siri at its initial launch in the UK. The product just wasn’t ready for the market at the time, especially for its poor localised listing. It may have gotten better but I’m happy with a good old Google text search!
Spoken queries are a natural next step, but mostly only mobile. I can’t imagine an office full of people vocalising their searches (which is possible now), especially for certain types of query. However, if it becomes the norm for mobile searches to be spoken then a trend may develop for certain types of short query to be vocal. Perhaps short conversion-likely searches – “buy coffee beansâ€, or local business searches – as opposed to informational searches (e.g. “google chrome show in folder not workingâ€).