Apple Augmented Reality Improved with Acquisition of Project Tango Collaborator Flyby Media

Sharing is caring!

What does this acquisition of Flyby Media mean for Apple’s Efforts in developing Augmented Reality Devices and Media?

I noticed this headline in the news this morning: Apple patent details visual-based AR navigation, confirms Flyby Media acquisition.

The article tells us that augmented reality can be accomplished using a phone, such as image data from a camera and sensor data from devices such as a gyroscope and an accelerometer. Furthermore, that information can be augmented by marrying images of reality with computer-generated imagery.

I checked out the patent referred to in the article, and it had me wondering what role Flyby Media played in Google’s Project Tango, which we are told that they used in the Apple Insider article.

The patent linked to in that news story is in Apple’s name, but the inventors are originally from Flyby Media. The patent is:

Visual-based inertial navigation
Inventors: Alex Flint, Oleg Naroditsky, Christopher P. Broaddus, Andriy Grygorenko, Stergios Roumeliotis, and Oriel Bergig
Assigned to: Apple Inc.
Regents of the University of Minnesota
US Patent 9,424,647
Granted August 23, 2016
Filed: August 12, 2014

Abstract

A method includes: receiving sensor measurements from a pre-processing module, in which the sensor measurements include image data and inertial data for a device; transferring, using a processor, information derived from the sensor measurements, from the first set of variables associated with the first window of time to the second set of variables associated with the second window of time, in which the first and second windows consecutively overlap in time; and outputting, to a post-processing module, a state of the device based on the transferred information.

An image from the patent shows how indoor mapping might be improved by the collection of information using imagery from Flyby Media:

The Apple Insider article points to this earlier article that describes other acquisitions of companies working in Augmented Reality: Rumor: Apple has ‘hundreds of staff’ working on virtual & augmented reality projects.

Two of the inventors listed on this patent supposedly now work at Apple, Naroditsky and Grygorenko, and one, Roumeliotis, is a professor at the University of Minnesota.

I was curious about what other patents Flyby Media might have, and the last sentence in the abstract from the first one, about augmented reality games, was interesting.

Systems and methods for efficient 3D tracking of weakly textured planar surfaces for augmented reality applications
Inventors: Christopher Broaddus, Andriy Grygorenko, Netanel Hagbi, and Oriel Y. Bergig;
Assignee: Flyby Media, Inc.
US Patent 9,147,251
Granted: September 29, 2015
Filed: August 3, 2012

Abstract

The present system provides an on the fly simple to complex 6DOF registration approach using the direct method. On the fly means it does not require training time. A user points a phone/camera to a planar surface and can start tracking it instantly. Simple to complex means the system performs registration in multiple levels of complexity from 2DOF to 6DOF. By increasing the complexity model, the system enables more surfaces to be tracked. The system can avoid local minima solutions for tracked surfaces, providing a more robust and accurate 6DOF tracking. Even surfaces that are very weak in features can be tracked in 6DOF, and virtual content can be registered. The system enables playing Augmented Reality games on low-end devices such as mobile phones on almost any surface in the real world.

Systems and methods for tracking natural planar shapes for augmented reality applications
Inventors: Netanel Hagbi, Oriel Y. Bergig, and Jihad A. Elsana
Assigned to: Flyby Media, Inc.
US Patent 8,644,551
Granted: February 4, 2014
Filed: October 15, 2010

Abstract

The present disclosure relates to systems and methods for tracking planar shapes for augmented reality (AR) applications. Systems for real-time recognition and camera six degrees of freedom pose-estimation from planar shapes are disclosed. Recognizable shapes can be augmented with 3D content. Recognizable shapes can be in the form of a predefined library being updated online using a network. Shapes can be added to the library when the user points to a shape and asks the system to recognize it. The systems perform shape recognition by analyzing contour structures and generating projective invariant signatures. Image features are further extracted for pose estimation and tracking. Sample points are matched by evolving an active contour in real-time.

It’s interesting seeing how Apple is advancing in Augmented Reality Technology by acquisitions of technology and hiring employees from companies such as Flyby Media. Will Apple be focusing upon games such as Pokemon Go, or do they have other ideas in mind as I alluded to in this post on a new Apple patent on Augmented Reality Displays from them: Might We Soon See Apple Augmented Reality?. We see Apple is working upon building devices that can display augmented reality. In addition, they are acquiring expertise and technology that assist in using augmented reality in applications such as indoor mapping.

Sharing is caring!