This codelab is deprecated. To learn how to use Tango to control the camera using the latest versions of the SDK, see our Unity Motion Tracking guide.

This codelab will teach you how to extend an existing Unity game to use Project Tango's motion tracking functionality to control the camera.

What you'll learn

What you'll need

Game Summary

For this codelab, we'll be using a simple and entertaining game, called "Lollygagger". The main game play is to travel through a virtual playfield, shooting Lollipop "cannonballs" at the Androids that are lollygagging around. Key features of note are:

  1. Start Unity. If you have not developed an Android application with Unity before, make sure you configure the Android SDK within Unity. In Unity, click Preferences > External Tools > Android SDK Location, then select the folder where you downloaded and unzipped the Android SDK.
  1. Click File > New Project and create a new project named ‘Lollygagger_tango'.

Let's build and run the non-Tango version of the game. From there, we'll add Project Tango support.

Import the game assets

Import the sample game scenes and assets into your project:

Configure the Android player

  1. Configure Unity to build an Android application:
  1. Add the game scenes to the build:

  1. Configure the Unity player settings:

  1. Save your project! Better safe than sorry :)
  2. Build and run the application on the device:

Congratulations! If you can play the starter project game on your Android device, you should be ready to go!

Troubleshooting

Even the most simple things can have gotchas! Here are a couple and what to do about it. If you're still having problems ask for help!

Issue

Solution

Issue

Solution

Now that we're able to run the game on the Project Tango device in 2D mode, let's add the Project Tango motion tracking functionality so you can use the device to move around in the game world.

Import and Add the Project Tango SDK

If you haven't already, download the Project Tango Unity SDK from the Project Tango developer site.

  1. Click on Assets > Import Package > Custom Package.
  2. Select the Project Tango SDK unity package TangoSDK_Nash.unitypackage that you downloaded.
  3. This brings up the Importing package window in Unity. Make sure all files are selected and click the Import button.
  4. Verify that you see an Assets/TangoSDK folder in the Project tab.

Configure the Tango prefab Gameobject

  1. Open the GameScene scene by double clicking Assets/GameScene.
  2. Drag and drop the Tango Manager prefab (Assets/TangoSDK/Core/Prefabs/Tango Manager) into the root of the Hierarchy tab.
  3. Select the Tango Manager in the hierarchy, and in the Inspector tab check Enable UX Library. This enables the UX Library Framework, which shows some helpful screens to users while using the application.

Disable touch based movement

Movement in the virtual world will be controlled by movement in the real world via Project Tango. As a result, we can remove the code that manages the movement via touch.

  1. In the Hierarchy panel expand the Player object to show the Main Camera object. Select Main Camera, then in the Inspector panel find the Mouse Looking script and disable it by clearing the checkbox.

Remove the motion controls associated with the project:

  1. Select Player in the Hierarchy panel.
  2. In the Inspector view, find the Movement script. Click on the gear icon and select Edit Script. This opens up the Movement.cs file in MonoDevelop.
  3. In MonoDevelop, remove the method FixedUpdate(). Save the file in MonoDevelop and come back to the Unity Editor.

You've just imported the Project Tango SDK, enabled the Tango Manager prefab which controls most of the Project Tango functionality for you, and removed the motion controls included with the project. We're going to use Project Tango to control the player's motion.

Next, we'll create a class to handle poses coming from Project Tango.

Create PoseController script

  1. Select the Player object in the Hierarchy panel.
  2. Click the Add Component at the very bottom of the Inspector panel.
  3. Select New Script, enter PoseController as the name, make sure that the Language is set to CSharp, then click Create and Add.

Add Skeleton code

Open the PoseController script by double clicking on it in the Assets folder. Then replace the contents with the following code:

using UnityEngine;
using System.Collections;
using Tango;
using System;

public class PoseController : MonoBehaviour , ITangoPose {
    private TangoApplication m_tangoApplication; // Instance for Tango Client
    private Vector3 m_tangoPosition; // Position from Pose Callback
    private Quaternion m_tangoRotation; // Rotation from Pose Callback
    private Vector3 m_startPosition; // Start Position of the camera
    private Vector3 m_lastPosition; // last position returned in Unity coordinates.
    
    // Controls movement scale, use 1.0f to be metric accurate
    // For the codelab, we adjust the scale so movement results in larger movements in the
    // virtual world.
    private float m_movementScale = 10.0f;

    // Use this for initialization
    void Start ()
    {
        // Initialize some variables
        m_tangoRotation = Quaternion.identity;
        m_tangoPosition = Vector3.zero;
        m_lastPosition = Vector3.zero;
        m_startPosition = transform.position;
        m_tangoApplication = FindObjectOfType<TangoApplication>();
        if(m_tangoApplication != null)
        {
            RequestPermissions();
        }
        else
        {
            Debug.Log("No Tango Manager found in scene.");
        }
    }


    // Permissions callback
    private void PermissionsCallback(bool success)
    {
        // TODO: Implement permissions callback
    }

    private void RequestPermissions()
    {
        //TODO: Register and request permissions callback
    }



    // Pose callbacks from Project Tango
    public void OnTangoPoseAvailable(Tango.TangoPoseData pose)
    {
        
        // TODO: Implement pose callback
    }

/// <summary>
    /// Transforms the Tango pose which is in Start of Service to Device frame to Unity coordinate system.
    /// </summary>
    /// <returns>The Tango Pose in unity coordinate system.</returns>
    /// <param name="translation">Translation.</param>
    /// <param name="rotation">Rotation.</param>
    /// <param name="scale">Scale.</param>
    Matrix4x4 TransformTangoPoseToUnityCoordinateSystem(Vector3 translation,
                 Quaternion rotation, Vector3 scale)
    {
        //TODO: Implement Transformation Helper method
        return Matrix4x4.identity;
    }


    // FixedUpdate is called at a fixed rate
    void FixedUpdate()
    {
           //TODO: Convert pose to Unity
      
      //TODO: Move the player
    }
}

You probably noticed that there are a lot of //TODO: comments in the starter code. Don't worry! In the next few sections we will be filling those in and explaining things on the way.

Save all the changes in MonoDevelop and switch to the Unity Editor. If there are compilation errors or other issues, they will appear in the Unity console log.

Goal

Project Tango applications using motion tracking need to ask users for permission to use the motion tracking camera and handle their response. In this section, we'll define and add the permission callbacks for our application.

Define the permissions callback

First, we will define how we handle the Permissions. Find the comment // TODO: Implement permissions callback and replace it with the following function:

if(success)
    {
        m_tangoApplication.InitApplication(); // Initialize Tango Client
        m_tangoApplication.InitProviders(string.Empty); // Initialize listeners
        m_tangoApplication.ConnectToService(); // Connect to Tango Service
    }
    else
    {
        AndroidHelper.ShowAndroidToastMessage("Motion Tracking Permissions Needed", true);
    }

If the permissions request was successful, we have some boilerplate to start the Project Tango framework. If the permissions request was unsuccessful (e.g. the user rejected a permission we require), we need to handle it. To keep things simple, we show a toast message in this example.

Request permissions

Now that we have a Permissions callback, we need to register it. Find the comment //TODO: Register and request permissions callback in the RequestPermissions() function and replace it with the following code:

// Request Tango permissions
m_tangoApplication.RegisterPermissionsCallback(PermissionsCallback);
m_tangoApplication.RequestNecessaryPermissionsAndConnect();
m_tangoApplication.Register(this);

This registers a callback and triggers the Permissions request dialog.

Run the app

Make sure all the source code is saved in MonoDevelop by clicking File > Save All. (If it is disabled, that means everything is ready!).

Switch back to the Unity editor and click File > Build & Run (or Cmd + B). If everything is ready, it will build the app and start it on your device.

Click Start in the game, and it should initialize the Tango environment by prompting for permission then initializing the motion tracking system.

Moving around still does not move the player in the game, so let's do that next!

Goal

After all that setup, we're still not doing anything with motion tracking. Let's change that! In this section, we will define our callback to handle poses from Project Tango and translate the data from Project Tango to Unity coordinate systems.

Define the pose callback

We'll be getting pose data through a callback, which we need to define. Find the comment // TODO: Implement pose callback in the OnTangoPoseAvailable function and replace it with the following code:

// Do nothing if we don't get a pose
if (pose == null) {
 Debug.Log("TangoPoseData is null.");
 return;
}
// The callback pose is for device with respect to start of service pose.
if (pose.framePair.baseFrame == TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_START_OF_SERVICE &&
    pose.framePair.targetFrame == TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_DEVICE)
{
    if (pose.status_code == TangoEnums.TangoPoseStatusType.TANGO_POSE_VALID)
    {
        // Cache the position and rotation to be set in the update function.
        m_tangoPosition = new Vector3((float)pose.translation [0],
                                      (float)pose.translation [1],
                                      (float)pose.translation [2]);

        m_tangoRotation = new Quaternion((float)pose.orientation [0],
                                         (float)pose.orientation [1],
                                         (float)pose.orientation [2],
                                         (float)pose.orientation [3]);
    }
    else // if the current pose is not valid we set the pose to identity
    {
        m_tangoPosition = Vector3.zero;
        m_tangoRotation = Quaternion.identity;
    }
}

There's a bit more going on here, so lets step through things.

First, if there is no pose data, let's just return from the method since there is no data process.

Next, we use the first if statement to define the base and target frames of reference that we're interested in. For motion tracking alone, you can consider this boilerplate since these are the only values we're interested in. If we were using area learning or depth perception, there would be other frames of reference we'd need to use. You can find more information on frames of reference in Project Tango at https://developers.google.com/project-tango/overview/frames-of-reference.

Finally, we check if the pose we got is valid. We can expect to get other pose statuses like initializing or, if we're unlucky, invalid, and don't want to do anything with those. If things look good, we grab the translation and orientation and save them in tangoPosition and tangoRotation, respectively. You can learn more about poses at https://developers.google.com/project-tango/overview/poses.

Add a Transformation Helper method

Since Tango's coordinate system is different than Unity's, we need to transform the tango pose position and rotation into Unity's coordinate system. Since we do it a lot, we can make a helper function. Find //TODO: Implement Transformation Helper method and remove the return statement. Then paste in this method:

        // Matrix for Tango coordinate frame to Unity coordinate frame conversion.
        // Start of service frame with respect to Unity world frame.
        Matrix4x4 m_uwTss;
        // Unity camera frame with respect to device frame.
        Matrix4x4 m_dTuc;

        m_uwTss = new Matrix4x4();
        m_uwTss.SetColumn (0, new Vector4 (1.0f, 0.0f, 0.0f, 0.0f));
        m_uwTss.SetColumn (1, new Vector4 (0.0f, 0.0f, 1.0f, 0.0f));
        m_uwTss.SetColumn (2, new Vector4 (0.0f, 1.0f, 0.0f, 0.0f));
        m_uwTss.SetColumn (3, new Vector4 (0.0f, 0.0f, 0.0f, 1.0f));

        m_dTuc = new Matrix4x4();
        m_dTuc.SetColumn (0, new Vector4 (1.0f, 0.0f, 0.0f, 0.0f));
        m_dTuc.SetColumn (1, new Vector4 (0.0f, 1.0f, 0.0f, 0.0f));
        m_dTuc.SetColumn (2, new Vector4 (0.0f, 0.0f, -1.0f, 0.0f));
        m_dTuc.SetColumn (3, new Vector4 (0.0f, 0.0f, 0.0f, 1.0f));

        Matrix4x4 ssTd = Matrix4x4.TRS(translation, rotation, scale);
        return m_uwTss * ssTd * m_dTuc;

Transform the pose to the Unity coordinate system

Find the comment //TODO: Convert pose to Unity in the function FixedUpdate() and replace it with the following code:

// Convert position and rotation from Tango's coordinate system to Unity's.
Matrix4x4 uwTuc = TransformTangoPoseToUnityCoordinateSystem(m_tangoPosition,
     m_tangoRotation, Vector3.one);
Vector3 newPosition = (uwTuc.GetColumn(3))* m_movementScale;
Quaternion newRotation = Quaternion.LookRotation(uwTuc.GetColumn(2),
     uwTuc.GetColumn(1));

Project Tango poses use a different coordinate system convention than Unity. In the starter code, we included the function TransformTangoPoseToUnityCoordinateSystem to handle the conversion for you.

Move the player

Finally, we set the new position and rotation to the converted values. At the bottom of FixedUpdate(), replace the comment: //TODO: Move the player with the following code:

// Calculate the difference in the poses received.  This allows us
// to recover when we hit something in the virtual world.
Vector3 delta = newPosition - m_lastPosition;
m_lastPosition = newPosition;
        Vector3 destination = rigidbody.position + delta;
        Vector3 vectorToTargetPosition = destination - transform.position;
        // If there is motion, move the player around the scene.
        if(vectorToTargetPosition.magnitude > 0.1f)
        {
            vectorToTargetPosition.Normalize();
            // Set the movement vector based on the axis input.
            Vector3 movement = vectorToTargetPosition;
            // Normalise the movement vector and make it proportional to the speed per second.
            movement = movement.normalized * 5f * Time.deltaTime;
    
            // Move the player to it's current position plus the movement.
            rigidbody.MovePosition (transform.position + movement);
 }
 else {
    rigidbody.velocity = Vector3.zero;
}
// always rotate, even if we don't move.
rigidbody.MoveRotation (newRotation);
// finally, let the game manager know the position of the player.
//GameManager.Instance.PlayerPosition = transform.position;

We're now done with PoseController, great job! Now save your file, go back to Unity, and select File > Build & Run to see your changes in action! Depending on your development environment, you might see a warning about line endings. You can safely convert or ignore.

After the app starts, you will be prompted to enable motion tracking. Make sure you pick Turn On Motion Tracking.

Troubleshooting

If your device gets stuck in initialization or something doesn't seem to be working right, try restarting your device and doing Build & Run again.

Congratulations! You've just converted an existing game to use Project Tango motion tracking. Where do you go from here?

Final scripts

Here is what your final scripts should look like. If you had any issues following the codelab, compare your code against these models, or copy/paste the final versions into your project.

Final PoseController.cs

using UnityEngine;
using System.Collections;
using Tango;
using System;

public class PoseController : MonoBehaviour , ITangoPose {
    private TangoApplication m_tangoApplication; // Instance for Tango Client
    private Vector3 m_tangoPosition; // Position from Pose Callback
    private Quaternion m_tangoRotation; // Rotation from Pose Callback
    private Vector3 m_startPosition; // Start Position of the camera
    private Vector3 m_lastPosition; // last position returned in Unity coordinates.
    
    // Controls movement scale, use 1.0f to be metric accurate
    // For the codelab, we adjust the scale so movement results in larger movements in the
    // virtual world.
    private float m_movementScale = 10.0f;
    
    // Use this for initialization
    void Start ()
    {
        // Initialize some variables
        m_tangoRotation = Quaternion.identity;
        m_tangoPosition = Vector3.zero;
        m_lastPosition = Vector3.zero;
        m_startPosition = transform.position;
        m_tangoApplication = FindObjectOfType<TangoApplication>();
        if(m_tangoApplication != null)
        {
            RequestPermissions();
        }
        else
        {
            Debug.Log("No Tango Manager found in scene.");
        }
    }
    
    // Permissions callback
    private void PermissionsCallback(bool success)
    {
        if(success)
        {
            m_tangoApplication.InitApplication(); // Initialize Tango Client
            m_tangoApplication.InitProviders(string.Empty); // Initialize listeners
            m_tangoApplication.ConnectToService(); // Connect to Tango Service
        }
        else
        {
            AndroidHelper.ShowAndroidToastMessage("Motion Tracking Permissions Needed", true);
        }

    }
    
    private void RequestPermissions()
    {
        // Request Tango permissions
        m_tangoApplication.RegisterPermissionsCallback(PermissionsCallback);
        m_tangoApplication.RequestNecessaryPermissionsAndConnect();
        m_tangoApplication.Register(this);
    }
    
    // Pose callbacks from Project Tango
    public void OnTangoPoseAvailable(Tango.TangoPoseData pose)
    {
        
        
        // Do nothing if we don't get a pose
        if (pose == null) {
            Debug.Log("TangoPoseData is null.");
            return;
        }
        // The callback pose is for device with respect to start of service pose.
        if (pose.framePair.baseFrame == TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_START_OF_SERVICE &&
            pose.framePair.targetFrame == TangoEnums.TangoCoordinateFrameType.TANGO_COORDINATE_FRAME_DEVICE)
        {
            if (pose.status_code == TangoEnums.TangoPoseStatusType.TANGO_POSE_VALID)
            {
                // Cache the position and rotation to be set in the update function.
                m_tangoPosition = new Vector3((float)pose.translation [0],
                                              (float)pose.translation [1],
                                              (float)pose.translation [2]);
                
                m_tangoRotation = new Quaternion((float)pose.orientation [0],
                                                 (float)pose.orientation [1],
                                                 (float)pose.orientation [2],
                                                 (float)pose.orientation [3]);
            }
            else // if the current pose is not valid we set the pose to identity
            {
                m_tangoPosition = Vector3.zero;
                m_tangoRotation = Quaternion.identity;
            }
        }
    }
    
    /// <summary>
    /// Transforms the Tango pose which is in Start of Service to Device frame to Unity coordinate system.
    /// </summary>
    /// <returns>The Tango Pose in unity coordinate system.</returns>
    /// <param name="translation">Translation.</param>
    /// <param name="rotation">Rotation.</param>
    /// <param name="scale">Scale.</param>
    Matrix4x4 TransformTangoPoseToUnityCoordinateSystem(Vector3 translation,
                                                        Quaternion rotation, Vector3 scale)
    {
        
        // Matrix for Tango coordinate frame to Unity coordinate frame conversion.
        // Start of service frame with respect to Unity world frame.
        Matrix4x4 m_uwTss;
        // Unity camera frame with respect to device frame.
        Matrix4x4 m_dTuc;
        
        m_uwTss = new Matrix4x4();
        m_uwTss.SetColumn (0, new Vector4 (1.0f, 0.0f, 0.0f, 0.0f));
        m_uwTss.SetColumn (1, new Vector4 (0.0f, 0.0f, 1.0f, 0.0f));
        m_uwTss.SetColumn (2, new Vector4 (0.0f, 1.0f, 0.0f, 0.0f));
        m_uwTss.SetColumn (3, new Vector4 (0.0f, 0.0f, 0.0f, 1.0f));
        
        m_dTuc = new Matrix4x4();
        m_dTuc.SetColumn (0, new Vector4 (1.0f, 0.0f, 0.0f, 0.0f));
        m_dTuc.SetColumn (1, new Vector4 (0.0f, 1.0f, 0.0f, 0.0f));
        m_dTuc.SetColumn (2, new Vector4 (0.0f, 0.0f, -1.0f, 0.0f));
        m_dTuc.SetColumn (3, new Vector4 (0.0f, 0.0f, 0.0f, 1.0f));
        
        Matrix4x4 ssTd = Matrix4x4.TRS(translation, rotation, scale);
        return m_uwTss * ssTd * m_dTuc;
    }
    
    
    // FixedUpdate is called at a fixed rate
    void FixedUpdate()
    {
        // Convert position and rotation from Tango's coordinate system to Unity's.
        Matrix4x4 uwTuc = TransformTangoPoseToUnityCoordinateSystem(m_tangoPosition,
                                                                    m_tangoRotation, Vector3.one);
        Vector3 newPosition = (uwTuc.GetColumn(3))* m_movementScale;
        Quaternion newRotation = Quaternion.LookRotation(uwTuc.GetColumn(2),
                                                         uwTuc.GetColumn(1));

        // Calculate the difference in the poses received.  This allows us
        // to recover when we hit something in the virtual world.
        Vector3 delta = newPosition - m_lastPosition;
        m_lastPosition = newPosition;
        Vector3 destination = rigidbody.position + delta;
        Vector3 vectorToTargetPosition = destination - transform.position;
        // If there is motion, move the player around the scene.
        if(vectorToTargetPosition.magnitude > 0.1f)
        {
            vectorToTargetPosition.Normalize();
            // Set the movement vector based on the axis input.
            Vector3 movement = vectorToTargetPosition;
            // Normalise the movement vector and make it proportional to the speed per second.
            movement = movement.normalized * 5f * Time.deltaTime;
            
            // Move the player to it's current position plus the movement.
            rigidbody.MovePosition (transform.position + movement);
        }
        else {
            rigidbody.velocity = Vector3.zero;
        }
        // always rotate, even if we don't move.
        rigidbody.MoveRotation (newRotation);
        // finally, let the game manager know the position of the player.
       // GameManager.Instance.PlayerPosition = transform.position;
    }
}