Imagine recording your life, so that you can search through it, and play it back later. Things that you record through audio and video might be sent to your search database where pictures you take might be processed. Images of faces may go through facial recognition software. Landmarks and objects might also be recognized as well. You might be able to write or speak queries like the following:
- What was the playlist of songs at the party last night?
- What were the paintings I saw when I was on vacation in Paris
- Who were the people at the business lunch this afternoon?
- How many books did I read in May?
It’s possible that you might be able to collect information like this, and have it associated with both your user ID and a digital signature to keep it from others, unless you decided to join with a group such as a family, or firefighters, or co-workers, to create a shared database for one or more events.
Lifestreaming is More Than Picture Taking
Last Summer, Mashable told us that Google Glass Will Have Automatic Picture-Taking Mode, where they mentioned an email from Google Co-Founder Sergey Brin describing how he had set Google Glass to take a picture every 10 seconds, and upload those to a server, while he was on a road trip. It appears that this picture taking was only part of the story.
Google was granted a patent last week that shows how you can collect information on devices with cameras and microphones, including wearable eyeglasses like Google Glass. Once collected and processed, you could search via text, spoken command, audio recording, and video recording to find information from your lifestream (a term not used in the patent).
The patent was co-invented by Hartmut Neven, who is the Technical Lead Manager, Image Recognition at Google, according to his linked in profile. He came to Google via a 2006 acquisition, where Neven Vision brought both facial and object recognition technology to Google.
Inventor David Petrou is a Senior Staff Software Engineer and has been with Google since 2005. He lists Google Glass as the project he is presently working on at both LinkedIn and Google Plus. Jacob Smullyan joined Google in January of 2011, according to the resume on his website, and he tells us there that he’s been working on the Google Goggles project. Inventor Hartwig Adam also appears to have come to Google from Neven Vision as well and lists Google Goggles as the project he is working upon in his Google Plus profile.
Method and Apparatus for Enabling a Searchable History of Real-World User Experiences
Invented by Hartmut Neven, David Petrou, Jacob Smullyan, and Hartwig Adam
US Patent Application 20130036134
Published February 7, 2013
Filed: June 11, 2012
US Patent Application 20130036134
Published February 7, 2013
Abstract
A method and apparatus for enabling a searchable history of real-world user experiences are described. The method may include capturing media data by a mobile computing device. The method may also include transmitting the captured media data to a server computer system, the server computer system to perform one or more recognition processes on the captured media data and add the captured media data to a history of real-world experiences of a user of the mobile computing device when the one or more recognition processes find a match.
This lifestreaming method may also include transmitting a query of the user to the server computer system to initiate a search of the history or real-world experiences, and receiving results relevant to the query that include data indicative of the media data in the history of real-world experiences.
This invention does seem to move Google into a realm of science fiction, where people can collect information and search for in like looking in an old photo album. Imagine a family reunion where attendees contribute their audio and video to a film shared with others about the event? Or a security team that can playback an incident they’ve recorded after doing facial recognition on everyone they saw between one hour and another? Will our definition of privacy need to be redefined? If Google doesn’t get there first with this type of lifestreaming recording, will others bring us to that point anyway?
When lifestreaming might be used
The patent tells us that there might be many ways to trigger recordings through a device like this:
A purposeful recording, where its use is triggered on and off by a user.
A location-based recording, where it is set to turn on at specific locations, such as upon entering Google’s headquarters.
A popular location recording, where the audio and video turn on at locations where lots of other people have recorded in the past.
An always on system, where a wearer doesn’t have to initiate the capture of media.
A person using this lifestreaming system could toggle back and forth between different modes of capturing information as well.
The patent also describes how digital signatures attached to recordings would be created, how information captured would be sent to a personal information database and analyzed, and how someone could then go on to search that lifestreaming information.
Google Glass has the potential to do a lot more than just snap a photo every 10 seconds. It can be the vehicle for creating a search engine of your personal experiences, and one that could be used for business purposes as well. This patent describes a way of using Google Glass as part of a lifestreaming search. Google Glass has evolved into more of an enterprise work device than a consumer device, which makes it questionable as to whether the kind of lifestreaming envisioned in this patent will ever come about.
What might Google do with Location history? I wrote about some other patents that use location history. These are about patents from Google that use location history:
- Google’s Mobile Location History
- Google Tracking How Busy Places are by Looking at Location History
- Google Lifestreaming?
- Google Patents Identifying User Location Spam
- Google Patent Granted on Mobile Location Detection
- Location Extensions Augmented Advertisements
Last Updated June 25, 2019.
Interesting find, thanks – especially with this article on ‘Worldstreams’ from Wired this month as background reading:
http://www.wired.com/opinion/2013/02/the-end-of-the-web-computers-and-search-as-we-know-it/
This is soooo cooool and totally sci-fi, I’m pretty pumped for this to be widespread. However, privacy will become a HUGE issue. There needs to be preemptive changes to privacy laws and establishment of easy to explain/follow rules.
Another thing to consider is the bandwidth usage. The current (US) network infrastructure may not be able to handle live recording/uploading of the world if this becomes mainstream. I imagine we would see a huge expansion of Google Fiber that is somehow coupled with Google Glass.
-Oleg
It’s definitely a great idea… I always wanted to archive my life online. And it appears like this Google thing has something to do with that.
This is really amazing stuff, it really makes some of those 90’s sci-fi movie technology look possible.
Having a search engine for you own experiences could span far beyond the own joy or reliving your memories and maybe even reach into the medical field. Maybe even to help reconstruct memories from brain injuries.
Google is certainly the only innovator by far in the online search with their inevitable attitude towards the growth of search in unimaginable manner, now that they are even concentrating on actual life search with its Google glass hardware is what makes them stand apart. The more data that an user is willing to give to Google, the more personalized and relevant will be the result the user is going to get without their privacy getting affected.
This seems redundant. You watch a movie of your life when you go to heaven!
But seriously, these servers have to be super secure. Even then, you still could have sketchy Google employees abusing their power. Kind’ve a scary thought.
wow, I agree with Oleg. This opens a brand new can of worms with regard to privacy and how it can be pre-emptively managed, as well as the data implications. If something like that was mainstream our current data usage will look like childs play.
Wow, this certainly adds a whole new dimension to Google. Not sure if I would like my whole life to be archived online personally, but it’s all out there now anyway. I also agree that privacy will become an even bigger issue than it already is – unfortunately not everyone that uses the Internet uses it to do good.
As cool as this is, its starting to get unnerving just how much information is potentially available to Big Brother about my private life. I hate being THAT guy, but the reality is information can always be bought for the right price. Knowing my life is an open book to a corporation makes me uneasy…
Unfortunately the only solution is not using the internet. But I’m not about to give up my daily dose of cute kittens just yet 😉
I still can’t believe this could happen- I mean, I feel weird because I didn’t get a cellphone until I was 15. A digital scrapbook of your entire lifetime??! I don’t know if I’d like for it to be searchable by anyone else but me, obviously, privacy will be the number one concern, but having a tool like that for private use would be incredible.
I don’t think even Facebook’s privacy managers would attempt to create a device that would essentially be storing your vision in the cloud – the consequences of which would most lawyers quake, I have a feeling. Some states don’t even allow you to record a conversation unless you have all the participant’s permission. Constantly recording videos, or even just images? It would run into easily as many issues.
Not that I don’t appreciate the Google Glasses project for what it is – a truly revolutionary idea. But with revolution will come a backlash, and I hope that those in charge of the project are ready for it when it arrives.
This is a cool but very powerful ability/capability. The power that comes with this type of technology can easily be abused. It’s interesting to note that most of us don’t want the government to be recording or monitoring us, but it’s okay if Google does….Right?
I agree with Dorothy. This stuff worries me. It’s a trolls and stalkers playground if they ever hack into someone’s account.
That being said, it’s an aweseome technological advancement.
The idea of it is cool, but not sure if it’s something I would personally want in my life. The worry of someone being able to access this data freaks me out, as who knows what sort of data it could provide that may benefit another person?
I can’t imagine how the technology becomes a bridge to our past lives. That’s pretty cool in some case, only if we know that our information aren’t used against us or without our consent. Because as far as I know there are humors that I’ve heard that Google is collecting our information and selling it. I hope it isn’t true. 🙂
Incredible. The medical-records aspect of this is intriguing, but personal privacy seems to be a common fear of this new technology.
While the idea is promising, there is some stuff I would rather forget than be documented. I also get the idea that our brain power would become lazy with this. Why try and remember something when you can google it and find it in a database somewhere.
Kind of like driving, I got a car so I walk less.
Hi James,
I don’t think that journals or diaries or photo journals made us more stupid, even though we could keep information about past events in those, and improve our memories by using them. We develop processes and templates for work for the same reason – so we don’t have to reinvent something every time we see it again. Being able to search our past can help to augment our memories, and improve our lives. The events we see today are going to have differences from the things we’ve experienced in the past, but if we have a way to see how we may have addressed something in the past, it’s not something that’s making us lazy, but rather something that allows us to view our past and learn from it.
Having a car doesn’t mean that you have to walk less, but it does mean that you can go a lot faster in a lot less time. 🙂
Sounds incredible. Although there are a few things I’d rather forget!
It starts being confusing… Who is Google and who is Facebook…? Google is trying to become more and more social by sharing anything (relevant or not…) and Facebook is adding Search to its social network.
It seems that Social (and semantic) Search is the new quest!
I suppose this is a way for Google to look friendly and helpful at the fact that is storing ALL your data about your purchases, movements, interests, and your LIFE. They pretend to help you trace your history in a fun way, when as always they will be using that information for their own commercial interests. Wake up people. Did you know that google has been copying millions of copyrighted books to create some sort of universal free library? Again, this is for their own monopolistic interest, as nice and interesting the idea may seem. Be careful, Google is a corporation, not Father Christmas.
Hi Eric,
Facebook should have had a working site search a long time ago. Google’s social network is as much about providing a better search – capable of both sharing social annotations and personalizing search better, as it is creating a social network. I can’t say that either is losing its identity – both seem to be in line with where they’ve been going for years.
Had to sit down when reading this—-seems overwhelming to know that a person’s life (mine) can be played back right before my eyes via the web. Ditto the sentiments that there are just something I’d rather forget. I vote that we simply come up with a new definition of “privacy.” The meaning is changing, anyway, right before our eyes.
WE are gradually giving up more of our privacy in order to live more of our life online. Will this be the final frontier, where we accept that the concept of privacy is old fashioned and not required?
Cheers
John
Call me old school, but I have no interest in a Google doing this to my life. If I want to chronicle my life, I’ll do it on my own, in my own way. Technology is great for so many things. However, I think people forget technology is just a fancy word for tools and become infatuated with integrating it into every aspect of their lives. That’s fine, but it comes with a cost. And just don’t try to force other people to have their life be an open book if they don’t want it that way.