I’ve written about some of the patents involved in Google’s Project Glass in the past, and very recently about the Google Ventures’ funded Magic Leap. Project Glass still exists, but it appears to now have new leadership and a new direction.
And then seemingly out of nowhere Microsoft announces a pair of goggles that they’ve been developing secretly, named the Hololens. And they’ve been feeding news sources some interesting information about them, like the article at Wired titled, “Project HoloLens: Our Exclusive Hands-On With Microsoft’s Holographic Goggles“.
Augmented Reality Exercise Glasses?
The Wired magazine article tells us that Alex Kipman, one of the inventors of the Kinect, is one of the chief inventors of the Hololens. So I started searching at the USPTO web site for patents co-invented by him which mentioned things like a “mixed reality display device.” The patent that image above is from is innocently named “Exercising applications for personal audio/visual system“, and tells us in its abstract that:
The technology described herein includes a see-through, near-eye, mixed reality display device for providing customized experiences for a user. The personal A/V apparatus serves as an exercise program that is always with the user, motivates the user, visually tells the user how to exercise, and lets the user exercise with other people who are not present
So the Hololens is an exercise program that you can easily carry around with you and lets you exercise with people who aren’t there.
Mixing Reality with Virtual Images
The patent Mixed reality display accommodation hints at much more than an exercise program, and defines its invention in much more detail:
A mixed reality accommodation system and related methods are provided. In one example, a head-mounted display device includes a plurality of sensors and a display system for presenting holographic objects. A mixed-reality safety program is configured to receive a holographic object and associated content provider ID from a source.
The program assigns a trust level to the object based on the content provider ID. If the trust level is less than a threshold, the object is displayed according to the first set of safety rules that provide a protective level of display restrictions.
If the trust level is greater than or equal to the threshold, the object is displayed according to the second set of safety rules that provide a permissive level of display restrictions that are less than the protective level of display restrictions.
Some of that trust level language reminds me of what we would hear coming from our TVs when bad things would happen on the holodeck in Star Trek.
I enjoyed some of the odd patent images from these patents too, which I found to be much better than simple flow charts.
Transforming Reality into Virtual Objects
From the patent Direct interaction system mixed reality environments, we learn of another potential capability of the goggles:
The head-mounted display device can create a three-dimensional map of the surroundings within which virtual and real objects may be seen. Users can interact with virtual objects by selecting them, for example by looking at a virtual object.
Once selected, a user may thereafter manipulate or move the virtual object, for example by grabbing and moving it or performing some other predefined gesture concerning the object.
Geolocation and Augmented Reality
Another patent talks about Geo-location, and setting locations up for different moods under an augmented reality approach. So we could possibly see scenes such as:
That patent is on Mixed reality filtering. This patent could easily be re-written into a Star Trek episode as well, as in when the Holodeck might be overwhelming:
Large amounts of virtual reality information may be available for presentation to a user. Some of this information may be associated with a particular location that may be within view of the user.
With so much virtual reality information available, managing the presentation of this information to a user, and the user’s interaction with such information can prove challenging.
Several other Microsoft patents looked clearly like they were related as well, including some that seemed very useful, like one on a Wearable food nutrition feedback system. Rather than give those away here, I’m interested in seeing how Microsoft might introduce those features to potential buyers of the glasses.
Augmented Reality and Advertisements
A new patent that was granted today, that doesn’t have chief inventor Alex Kipman’s name on it. It sounds very much like it should go with these patents, but it introduces a different element that the others are missing. The patent is:
Augmenting a field of view (8,943,420)
The abstract tells us:
The claimed subject matter relates to an architecture that can enhance an experience associated with indicia related to a local environment. In particular, the architecture can receive an image that depicts a view of the local environment including a set of entities represented in the image.
One or more of the entities can be matched or correlated to modeled entities included in a geospatial model of the environment, potentially based upon location and direction, to scope or frame the view depicted in the image to a modeled view. Also, the architecture can select additional content that can be presented.
The additional content typically relates to services or data associated with modeled entities included in the geospatial model or associated with modeled entities included in an image-based data store.
Take Aways
This patent refers to the possibility that some of the augmented reality presented to a viewer using the goggles might be based upon an advertising model that advertisers would be able to bid upon.
I can’t remember any ads on the Star Trek Holodeck, or even hinted at in Project Glass, or what we’ve seen so far of Magic Leap.
And we might not through the Hololens Glasses either. But the possibility of being presented with ads does exist.
So Microsoft has also joined the fray! It would be interesting to work out with virtual exercise partners. Complete application and usefulness of this device would be interesting.
Hi Bill Slawski
You have written a fantastic hands on review on Microsoft holodeck look hololens patents. I must say Microsoft is trying to grab the Hardware business as like Google and trying to make us amazed. Let’s see if this become success or fail like Google Glass.
Hi Robert.
The patent is assigned to Microsoft, even though Alex Kipman is not listed on it as an inventor. In many cases, inventors will assign the patents they are granted to the company that they are working for at the time those are worked upon.
I guess we won’t know that Microsoft is selling advertising in Hololens augmented reality until we see that advertising offered.
I’m beyond excited for augmented reality advertising. The possibilities for a local smb are off the charts, especially for retail. Looking forward to watching this unfold. Bill, I didn’t click through the patent, but do you know who holds that if Alex Kipman doesn’t?
Thanks,
Robert
The possibilities for a local smb are off the charts, especially for retail.I must say Microsoft is trying to grab the Hardware business as like Google and trying to make us amazed.
You are right Bill, infact i also agree with Munna, Microsoft always kept itself for a break through, as we all know that Windows do have it’s popularity but Google has taken over with just a simple search engine lolz… I believe… for a big break… Microsoft should work on Bing sure by advertising….. you know a beautiful ad with good lyrics will definitely work (Y)
Hi Robert.
You are right. I am agree with you.Let’s see if this become success or fail like Google Glass.
Although Microsoft hasn’t been out of the game, it would be great to see them get back up a little higher. Maybe even actually be competition for Google. I may even enjoy working out now, without getting bored after two minutes. The idea of working out with virtual exercise partners almost seems a bit absurd at first, but on more thought it seems like it could be pretty cool. Great read Bill. I hope this becomes a success and doesn’t fall before it has a chance to fly. It could be quite interesting!
Thanks, Matt. I’ve been reading some great review of the hololens, so Microsoft might have themselves a hit on their hands.
hi,i’m so happy to reading this post.
I hope this becomes a success and doesn’t fall before it has a chance to fly.
It could be quite interesting!
thank you.
I’m so excited for VR and AR to be a reality. I still remember my first pair of Jurassic park VR goggles. It was basically just a mono 512×512 screen shown through a mirror mounted on your head and controlled with a conventional controller, but at the time, I thought it was the shiz.
I think the development of augmented reality and virtual reality technology is creating and integrating a new layer of data that is seamlessly applied to our existing world. A lot of people talk about the “internet of things” where all of our devices, appliances, and real-world tangibles have digital counterparts that are accessible through AR and VR devices.
This phenomenon of augmented and virtual reality is reminiscent of an anime I watched called “Denno Coil,” that takes place in a future where AR goggles are ubiquitous and have had a massive impact on the way people live their lives.
In this world, everyone wears AR goggles and every part of our tactile reality has a digital counterpart that is laid over the top of it.
In the anime, hacking the digital representations of physical objects became a legitimate issue, with AR hackers being able to manipulate these digital versions of real world items, it created mass confusion among the populace who depended on data from their AR goggles.
Eventually, the lines between digital AR and tangible reality become blurred because of the immersive experience of AR, leading into the main conflict of the series. I am intrigued to see how humans adapt to AR and VR technology and whether or not it will parallel this scenario.
Hi Zenman,
An interesting Scenario you set out. I guess we have to pay attention carefully and as smartly as we can