Taking the Magic Leap into the Future - Part 1

​Magic Leap has been in the press for a number of years now. They got investors to part with a lot of $$$ and pour it into a startup for a completely new kind of technology. There had to be something really compelling for this to happen. Even as the Magic Leap One has just been released into the wild, the company has made clear that this is just an interim product intended for developers, early adopters, and enthusiasts.

So why not wait for the product to sport improved features like wider field of view, offer a wealth of Apps, and of course, become more affordable? I will answer it in two short words: Spatial Computing.

What Magic Leap has done is to rethink computing so that the immersive environment is really as much an operating system as it is a platform for running apps. They took to heart the notion of seamlessly melding the virtual created content with the real world so that the two are not perceived separately. In that sense, the term MR for Mixed Reality (or as I sometimes think, "Melded" Reality) is more appropriate for the Magic Leap than AR or XR.

Instead of extolling the virtues or ranting about the product or technology, I'd like to delve into some of the ideas behind Magic Leap, why you should take interest in it, and where it may go. 

---

This is not your father's VR gear

At first glance you may think Magic Leap is out to make a fashion statement. Perhaps it is, but the statement has to do with functional design. To begin with, you'll notice the visor is positioned above and not over the ears. There are several reasons for this. Magic Leap incorporates builtin headphones that can take advantage of Resonance Audio, Should you prefer to use your own headphone or earbuds, you can easily do so, as the Light Pack allows you to connect a standard 3.5mm jack. There is another not so obvious benefit to having the visor extending above the ears. It permits you to comfortably blend the ambient sounds in your environment with the audio output from the Magic Leap device. In VR you want to shut out the world, but in AR having the ambient sounds as they naturally occur is a plus.

---

Input, Sensors and Navigation

.It is fascinating how we so often derive from nature to inspire technology. Perhaps Magic Leap wasn't influenced by jumping spiders, but you have to admit that there's an uncanny similarity in the way biological and electronic sensors are assembled.

Mapping your physical space
First came the Tango tablet, then the Hololens and now Magic Leap. What characterizes these devices is that they know where they are in physical space. What's more, they can handle occlusion mapping dynamically. The video below shows the Magic Leap creating building a geometric mesh as you walk around. The color bands depicts the current distance from your view, and therefore the color of the mesh pattern can change as you move closer or farther away from the generated mesh.

Mapping Hand Gestures
One of the features of the Magic Leap One is the ability to track hand gestures, which includes a single finger, making a fist, pinching thumb and forefinger, a thumb's up, using the forefinger and thumb to form an "L", the back of an open hand, and "OK" gesture, and forming a "C" shape. The video below shows this in action. 

Notice that the Magic Leap easily distinguishes between right and left hand, and can simultaneously track both hands. In the video you will see various hovering colorized balls. These indicate the relative positional mapping of the hand fingers and joints. As the captured video is not 3D (default video capture is from the left eye), the placement of the colorized markers don't quite match what you would see while wearing the Magic Leap.

All these features are easily programmable, as the Magic Leap SDK provides a fairly rich and easily accessible API (Application Programming Interface). This API works with Unity, Unreal, and Helio- a Magic Leap web browser. I will outline this in a future blog post. For the moment, I want to stay with the theme of Magic Leap sensory input features.

Another important aspect with Magic Leap is the ability to capture and perform data analytics. The Magic Leap API exposes internal sensor data sufficiently so I can write a program to produce a log file like the kind shown below which is from one of the sessions involving hand gesture recognition. 

Eye of the Gazer
One of the modalities of user input is the ability to actively perform eye tracking. I am not talking about head pose, but rather what you are actually looking at as your eyes roam your field of view. The video below shows this in action.

In this rudimentary example I thought it would be useful to basic eye gaze activity in action. Unconscious eye gaze can influence behavior in the rendered scene. As the user glances at the cube on the right it turns red and rotates and while the user gazes at the sphere it turns green. These are simple static objects. We could also affect the animated behavior of avatars. We could influence the plot line in an interactive story. ​We could create new kinds of user interfaces. We could create training software for special needs such as for recovering stroke patients. We could combine eye gaze with voice input to do something like "Make that one red" where 'that one' is the object we are looking at.

More to follow
There's lots more I have to say about Magic Leap and will be doled out in future installments. In the meantime, we have an active meetup group (​https://www.meetup.com/NY-Magic-Leap-Meetup) that you are welcomed to join.