More recently there are devices that capture stereoscopic photos and videos for iPhone, Androids, and GoPros. They have nice viewers to display images in 3D.
But what if you want to go further and maybe turn them into a 3D gallery, or perhaps add some interactivity? It would behoove you to place and render these assets in a VR setting with programmatic interactivity, such as Google Cardboard, Gear VR, or Oculus.
It turns out there are some challenges if you want to place stereoscopic images within a traditional vr scene. This is because you have to render the photographic images differently for the left and right eye. This article gives you the basics on how to accomplish this in Unity for both Google Cardboard and Gear VR/Oculus.
Before launching into the nitty gritty I want to say a few things. The techniques described are almost identical for Google Cardboard and Gear VR/Oculus. They are different enough that they'll be discussed separately. Placing 3D video in a VR scene works, but it is vastly more complex, as it is highly dependent upon deployment platform, file formats, and the like. So 3D video is not going to be discussed in this article (perhaps, it'll appear in a future article).
The basic assumption here is that you already have familiarity developing VR Apps with Unity using their built-in VR support for Gear VR/Oculus or an external package for Google Cardboard. An additional assumption is that you are on Unity 5.2 or later. The steps provided here apply equally well whether you are developing on a Mac or PC or are using the Pro or Personal Edition of Unity.
Stereoscopic images have a left and right part, corresponding to what the left and right eyes each sees. In the Unity natively supported VR there is only a need for one Camera. By enabling VR support in the Player settings, Unity automatically generates two sets of Game Views slightly offset from one another based on the position and orientation of the main camera. This works fine for a generic game view, but it immediately throws a monkey wrench into the gears when you want to place in your scene two sets of images, one for each eye.
So basically, you have to tell Unity to construct the view for each eye a bit differently. Fortunately, for images, there's really no scripting involved.
Putting Stereoscopic images in Google Cardboard
Start out with a stereoscopic image like the following:
You will need split the image into two parts; one that will be visible only to the left eye, and the other only visible to the right eye. This is easily done with a photo-editing program like PhotoShop. Just cleave the image into two separate halves. You can save it as a PNG or JPG file. For purposes of this article, we'll be using the files called Left_Eye_Image.png and Right_Eye_Image.png.
You will be importing these into your VR scene in Unity. If you don't already have an existing Unity Cardboard project, you can start from scratch. Just be sure to do the necessary generic steps for creating Unity Cardboard Apps. For example if you are developing for Android you'd setup the Android SDK, within Unity switch build platform to Android, import the Unity for Cardboard Integration package, adjust your Player/Build settings, and do the necessary setup of Developer Options on your Android phone. Chances are if you successfully built any Google Cardboard App with Unity, you've done all these steps. The only thing you would need to do, is make sure you have the right build settings for this Unity Project.
So start out with any Hello World Unity project that's Cardboard enabled. If you have a generic main camera, disable or delete it, as Cardboard will be doing the camera work.
From the Unity Standard Assets, import into your project a FPSController asset. This will include a FPSController Prefab. Drag the prefab to your Hierarchy. You can position the FPSController anywhere you want. In the example used for this article the x-y-z position is (9,25, 2).
You will notice that the FPSController has a child called FirstPersonCharacter. If there is a checkmark for Camera. Uncheck it to disable this camera.
In your Cardboard folder for your Project there's a Prefabs folder that contains a CardboardMain prefab. Drag this prefab directly onto the FPSController in your Hierarchy so that both FirstPersonCharacter and CardboardMain are direct children of the FPSController.
So the overall Hierarchy organization should look something like the following:
There are items shown in this hierarchy which we'll be introducing shortly (such as the Quads). For the moment do not concern yourself with MyMesh component. This was something that's created for other assets in the scene as well as some scripts to manage interaction.
In the Assets folder of your project create a Resources folder. Import the two files- Left_Eye_Image.png and Right_Eye_Image.png and place them inside the Resources folder. At this point make sure you have saved the current scene. It can be any name of your choosing. When it comes time to build your App be sure to set your build settings to this scene.
In your project Hierarchy add a flat surface having a mesh renderer and a shader into your scene. You might use a plane or a quad. In our example, we'll use Quads. Give this quad the name Quad_Left. Drag the Left_Eye_Image asset from your Resources folder on top of the Quad_Left item in your Hierarchy. Now your Quad_Left item will use the Left_Eye_Image.png file as its shader.
Size and position your Quad_Left so that it's readily visible in your Game View and has the proportions you desire. In this example the PNG file is 854x1486 pixels. The quad we are using has the scale of 54.24 for x, 94.38 for y, and 1 for z; so it keeps the same aspect ratio as the PNG file. In the example shown, the x-y-z position of the Quad_Left is (8, 8, 60).
Everything you did for the Quad_Left needs to be done equivalently for Quad_Right. Basically, it's easy to copy and paste the Quad_Left and rename the pasted item to Quad_Right Of course, be sure to replace the Left_Eye_Image.png with Right_Eye_Image.png.
Both the Left_Quad and Right_Quad are directly on top of one another. Our next task is to tell the left and right cameras inside CardboardMain to exclude the quad that doesn't correspond to the matching eye.
Here are the basic steps:
Edit your Tags and Layers.
Add two new layers, one of which will only be visible to the left camera and one that will only be visible to the right camera. So create the layers here. The filtering gets done downstream.
Now we're going to tag the Quad_Left and Quad_Right objects accordingly. Select your Quad_Left in the Hierarchy. In the Inspector panel change the Layer tag from Default to Left_Eye_Layer..
Change the layer from Default to Left_Eye_Layer.
The resulting setting for Quad_Left should look like the following:
For the Quad_Right it's essentially the same except that its set to Right_Eye_Layer.
You might think that at this point we're done; but not quite yet. You have to tell the cameras for the left and right eyes to exclude the quad tagged to the other eye. I guess you could call this "Turning a blind eye"?
In your Hierarchy select the MainCameraLeft. In the Inspector panel adjust the MainCameraLeft properties.
Note that Camera (under Transform) should be unchecked and that the CardboardEye script is checked. Make sure that Eye is set to Left.
Now make sure the Toggle Culling Mask (for your MainCameraLeft) is set to Right_Eye_Layer. This basically tells the left eye camera to ignore the quad that gets rendered for the right eye.
Now of course, do the equivalent for the MainCameraRight.
At this point you should be able to test this out in the Unity Editor (press Cmd-P for Mac or Ctrl-P for Windows).
For your build settings be sure to switch to your deployment platform (Android or IOS) and make the appropriate settings, including remembering to specify the project scene. Since this is a Cardboard App (as of Unity 5.2,/5.3) you would not enable Virtual Reality Supported. This is because Cardboard does all the heavy lifting.
When you get to Gear VR or Oculus it's a different story. Unity natively supports Gear and Oculus, so the Virtual Reality Supported setting would checked to enable VR.
Putting Stereoscopic images in Gear VR and Oculus
The setup for Gear VR or Oculus is very similar to what we did with Cardboard. So this will be an abbreviated description.
I want to outline a few distinctions between the Gear VR/Oculus setup vs. Cardboard.
Your Hierarchy would look something like the following:
You don't need to concern yourself with the item called mesh. It has some stuff that I threw in for other assets and interactions.
As you created a MainCamera-Left, do the equivalent for MainCamera-Right. A few points are worth mentioning:
So there you have it. Now you can mix stereoscopic images in your VR scenes!
At some point I will update this article; I suspect that the Google Cardboard one day will be natively supported inside Unity. Also, I would like to cover rendering 3D Video inside VR. This topic is too messy and complex to give you something you can easily use.