A patent application was published today which describes the kind of intelligent automated assistant that we see in use on Apple’s iPhone 4S, known as Siri. But the Siri patent isn’t necessarily limited to the iPhone application itself, and the describes how such a system could be used in a number of ways, including with mobile phones, PDAs, tablets, game consoles, embedded computer systems in cars, and possibly others. This assistant might provide information and services on a single client device or multiple devices, and possibly in combination with applications and information on servers as well.
It could also act as an active participant in messaging platforms such as email, instant messaging, discussion forums, group chat sessions, and customer support sessions.
The intelligent assistant in the Siri Patent could go on the internet to discover, find, choose among, purchase, reserve, or order products and services, using multiple sources of data if needed to check reviews, find prices from different places and their locations and operating hours, locate information about events, performances and attractions, places to go, places to stay, places to eat and drink, and times to meet others.
It could also combine natural language dialog with a graphical user interface that can:
- Search, including location-based search
- Navigate, providing maps and directions
- Lookup information on databases (such as finding businesses or people by name or other properties)
- Provide weather conditions and forecasts
- Check prices of market items or status of financial transactions
- Monitor traffic or the status of flights
- Access and update calendars and schedules
- Manage reminders, alerts, tasks and projects
- Communicate over email or other messaging platforms
- Operate devices locally or remotely (e.g., dialing telephones, controlling light and temperature and home security devices, playing music or video)
- Initiate, operate, and control many functions and apps available on the device
- Offer personal recommendations for activities, products, services, source of entertainment, time management, etc.
What does this intelligent assistant mean to search? The Siri patent filing tells us:
Unlike search engines which only return links and content, some embodiments of automated assistants described herein may automate research and problem-solving activities.
We’re also told that the personal information that the assistant learns about while interacting with its user enables it to provide better-personalized search results than we might get from search engines, and also improve how someone interacts with the Web by automating things such as filling out forms or signing up for services.
In addition to speech recognition and input, the intelligent assistant in the Siri Patent also will offer graphical options to do things like calling a business, saving information about the business to remember for later, share the location with someone else by email, show the location on a map, save personal notes about the business.
The Siri patent also describes how “Active Ontologies” serve as the underlying infrastructure behind how the intelligent assistant builds models about data and knowledge. For example, a “dining out domain model” might be linked to a “restaurant concept” as well as a “meal event concept.”
The Siri patent application is very long and provides a wide range of examples and information about how the assistant works, interacts with others, understands paraphrases, interprets language and vocabulary, and recognizes language patterns, and other features as well.
Intelligent Automated Assistant
Invented by Thomas Robert Gruber, Adam John Cheyer, Dag Kittlaus, Didier Rene Guzzoni, Christopher Dean Brigham, Richard Donald Giuli, Marcello Bastea-Forte, Harry Joseph Saddler
Assigned to Apple Inc.
US Patent Application 20120016678
Published January 19, 2012
Filed: January 10, 2011
Abstract
An intelligent automated assistant system engages with the user in an integrated, conversational manner using natural language dialog, and invokes external services when appropriate to obtain information or perform various actions. The system can be implemented using any of a number of different platforms, such as the web, email, smartphone, and the like, or any combination thereof. In one embodiment, the system is based on sets of interrelated domains and tasks and employs additional functionally powered by external services with which the system can interact.
Siri Patent Conclusion
Will Apple’s intelligent assistant Siri improve search by automating search activities and problem-solving as the Siri patent notes?
Will Siri learn from both interactions with its user and information that it saves in both its short term and long term memories (yes, according to the Siri patent filing it has both) to offer an improved personalized search?
Those are interesting questions, and Apple doesn’t seem to have the search infrastructure in place to improve upon search services on its own. But that doesn’t mean that they aren’t working on their own versions of things like how to rank local search results.
Bill, I am sorry for this crazy comment, but here goes:
Again, I see us moving one step closer to AI. Sure Apple may not have all of the pieces of the puzzle, but…
…in the classic novel “Twenty Thousand Leagues Under The Sea”, Captain Nemo didn’t have all of the scientists working with a cumulative knowledge of the project, but, rather, bits and pieces of the whole technology.
Dividing it up between different business entities is a great way to conceal this technology.
I notice in many of your posts about patent applications, there are references made to decision making abilities being instilled into algorithms on various fronts.
How do we know these will not end up being an array of modules?
Ultimately, this will all build into something that is more than the sum of its parts…its inevitable.
Mark
BTW. I don’t even have an iPhone…I use a BlackJack…how old school is that? LOL!
Mark, I bet you would freak out if you saw the patent for the technology to track and store your information and habits and develop an individual profile based on the decisions you make through technology like Siri, Facebook, Twitter, Google+ and more to eventually predict your own decisions.
The apple is taking care of everything, most industries are investing in applications for this platform. Damn I like Android, it’s a very cool system, but being attracted to the Apple platforms.
I will test this app,
Hugs!
Too bad another company run by some sugar water salesman beat these folks to the punch and conceptualized and promoted this concept back in the late 80’s. Compare what’s in the video with the claims. Prior art anyone?
http://www.youtube.com/watch?v=3WdS4TscWH8
I still bleed 6 colors over here, but give me a break!
– Ted
Can’t help but think of “AI” and “Surrogates” with this.
I have used the iphone 4s for just over 3 months now and it fits in very well within my mobile car mechanic business. I was sceptical about Siri as i have used voice activated software before without very good results. I was pleasantly surprised that it needed hardly any time to work out my accent. I think the potential is huge and am glad that IBM have filled a patent as this is bound to speed up the development of this type of software.
I’m not impressed with the current search version in the iPhomne 4S. So I’m looking forward to this improvement. Although I’m sure this information will be sold to the highest, actually ANY, bidder. But that’s the age in which we live.
We are definitely going towards a more performant AI, that can build upon it’s interaction experience with it’s “buddy” (the user). I wonder if this will not affect privacy issues as well as wondering how well (how soon)this can work out in other languages. We will need more input but SEO will definitely have to adapt quickly. Is it the end of targeting frequent spelling mistakes on keywords?
I would really like to improve how Siri is activated (and receives user input). Rather than needing to explicitly start Siri, it would be nice if the phone would actively listen for a user command. So that any sentence prefaced by the phone owner saying “Siri” would be parsed and treated like a command to process.
I’m sure it’s coming at some point. Obviously it would put a drain on battery life to ask the phone to constantly be listening for user input. But, it’s exciting how the rate of technology acceleration is increasing!
I think that Apple’s Siri deserves a patent. It is a shame that patents are so often abused these days…they no longer serve the purpose they were intended to serve
Patenting this App is advantageous on their part as the industry is highly competitive. You will never know competitors are having their own versions already weeks after your product launching.
Even in absence of STEVE JOBS, Apple is doing really great. I’ll definitely test this. Hope to see you some more apps in recent future. Thanks Bill for sharing this with us.
Great post Bill. I think Siri is one of the finest apps that is available for iPhone. One thing I realized after I became a bestselling author and long before Inc Magazine voted my company as one of the fastest growing companies is the I couldn’t have managed it without the Siri. when I hear the industry experts say ‘Apple will be collecting your voice imprint and storing that data on their servers’ I feel bad, But still it is one of the most important apps that I use
Hi Mark,
It’s possible that we are moving in that direction.
Why is Google’s head of research (Peter Norvig), one of the world’s foremost experts on AI? And the same guy who co-authored one of the most significant text books on the subject (http://aima.cs.berkeley.edu/)?
I don’t think that many of the people who are working on different parts and pieces of technology like Siri or Google have artificial intelligence as their end goal, but I do think they are moving us down that path.
Hi Neo,
The apple patent I linked to in the post (one of the longest patents I’ve seen in quite some while), does introduce a lot of those topics.
Hi Katipsoi,
I haven’t had a chance to test it either, but I’d definitely like to.
Hi Ted,
Thanks very much for sharing the link to that video. The Knowledge Navigator is definitely the work of some people dreaming big. Maybe we’ll get there someday soon. 🙂
Didn’t someone try to use a device portrayed in a movie as prior art in a patent infringement defense case sometime in the past year?
Hi Allan,
I was thinking Terminator myself. 🙁
Hi Richard,
Interesting to hear that you’re finding Siri so useful to running your business.
The patent is actually Apple’s rather than IBMs.
Hi Felipe,
Since I don’t own a iPhone 4s, it was a little hard for me to tell what was already part of Siri as it is now, and what aspects it might have developed are on the patent only, but it sounds like the patent adds things that aren’t quite built into Siri now.
Apple definitely doesn’t have the search capabilities that Google does now, but I could see them possibly moving into that area sometime in the future. At this point, it seems like they are edging their way forward in the local search arena.
Hi The Frugal Dad,
Not sure that you’ve told me enough for me to understand why patents don’t serve the purpose they were intended to.
My take on the purpose of patents is that they are supposed to spur innovation by having inventors bring their innovations into the public view in exchange for a right to exclude others from copying those innovations for a limited amount of time.
There are patents that are granted that maybe shouldn’t have been, and there are people who are using patents in ways that seem to impede innovation, but I still think there are patents that fit that original intent.
Hi Maria,
The mobile phone industry is exploding with new technology, and new methods to use phones. When we get to something like this intelligent automated assistant though, I think the market is a lot smaller. I’m not sure how many companies out there are capable of putting something like this together.
Hi Eliseo,
Reading through the patent, I was impressed with how much this personal assistant might learn about someone who uses it, how much their “personalized” search results might depend upon how well it “remembers” that person’s habits and interests, and wondering how much of that might be shared with others. Privacy was one of my concerns, too.
Will Siri make typing mistakes during searches? Maybe not.
Hi Danny,
There was a fair amount included in the patent about activation of Siri, and the patent may include some improvements in that area over what Siri presently does. I don’t know enough about how Siri works (no iPhone 4s here) to be able to tell.
Hi Mark,
You’re welcome. I’ve been wondering how much impact Steve Jobs might have had in the creation of Siri, and how much different it might have been if he had been healthier.
Hi Daniel,
It sounds like the Siri has had a pretty big impact on how you run your business. Interesting.
This app is terrible. I’m sorry. Maybe its my accent, being from south Georgia and spending the rest of my life in Nashville, Tennessee – but Siri can’t understand over half of what I say!!! It’s great in concept but its got a long, long way to go. My opinion I guess…
Hi Thomas,
I didn’t write this post to try to promote Siri – there are plenty of people at Apple doing that, so there’s no need to apologize. 🙂
Appreciate your sharing your experience with it though. I think it takes a lot of work, and a very large amount of data to get voice recognition software to work correctly, and take into account different accents, and for Siri to be really successful, they probably need to work upon that aspect of how Siri works.
HI Bill,
Siri is a great app. Apple is way ahead of their competitors.
Credits for Steve Jobs. He was a very successful CEO.
Bill, I admire how you reply to every comment, most Internet marketers don’t bother to reply and only think about the money. Not you.
Thank you
Rick
Hi Rick,
Thanks. The Siri patent is definitely one of the longest that I’ve run across, and seems to include a number of things that don’t appear to be presently in the version of Siri that is available now. Not sure if that is because of business reasons or technical limitations or to some degree both. I suspect that it’s a product that will be given lots of attention, and time to evolve.