Overview

This is an mbed project for connecting using an IMU and a pushbutton to implement a hand controller for PC virtual reality gaming using an Oculus Rift. The same principles can be applied for other PC-based 3D gaming.

Group

Jordan Hartney (Section B), Emily Shibut (Section B)

Organization

The basic organization of the project is presented here:

An LMS9DS0 IMU is used to provide accelerometer, gyroscope, and compass readings. The IMU generates data, communicating with the mbed via I2C. The mbed has a serial connection to the PC, over which it sends all of the IMU data, as well as an alert if the trigger pushbutton has been pressed.

The C# script behind the corresponding game object opens up the serial connection, and launches a thread to read from it continuously. Filtering and synthesis of IMU data is performed by this thread as well.

Controller

To mount the controller on a glove without impeding hand movement, a very small breadboard had to be used. The limited size of the breadboard actually required some false connections between the microcontroller and the IMU breakout board to be made on unused pins so that both could fit (alternatively, some unused pins could be folded down). The trigger button (a pushbutton) was attached to the glove at the knuckle of the index finger for easy firing, with wires back to the breadboard soldered on.

Wiring Tables

The IMU uses an I2C bus and takes 3.3V power.

The pushbutton is connected to GND and P15.

Code

This code is run continuously. A small wait time is necessary for good graphics in the game. The extra DigitalIns are declared to prevent anything weird from happening with the extraneous IMU connections.

#include "mbed.h"
#include "LSM9DS0.h"

#define LSM9DS0_XM_ADDR  0x1D // Would be 0x1E if SDO_XM is LOW
#define LSM9DS0_G_ADDR   0x6B // Would be 0x6A if SDO_G is LOW

LSM9DS0 imu(p9, p10, LSM9DS0_G_ADDR, LSM9DS0_XM_ADDR);
Serial pc(USBTX, USBRX);

//Extra inputs for 'fake' connections, can ignore
DigitalIn in0(p21);
DigitalIn in1(p22);
DigitalIn in2(p23);
DigitalIn in3(p17);

//Trigger pushbutton
DigitalIn trigger(p15);

int main() {
    //Initialize IMU
    imu.begin();
    //Set pushbutton input mode
    trigger.mode(PullUp);
    while(1) {
        imu.readAccel();
        imu.readGyro();
        imu.readMag();
        //IMPORTANT - Unity only recognizes Unix-style line endings in the context of the C# SerialPort.ReadLine() function
        pc.printf("%f %f %f$%f %f %f$%f %f %f\r\n", imu.ax, imu.ay, imu.az, imu.gx, imu.gy, imu.gz, imu.mx, imu.my, imu.mz);
        wait(.005);
        if (!trigger)   //Print constant fire string
            pc.printf("PEW\r\n");
    }
}

Game

The game was developed using Unity and some of its tutorial assets.

Code

In a C# class (below) behind the object being controlled via the mbed, a thread is created to continuously read in from the serial port. Once valid data values have been received and parsed, computation of how to rotate the object is performed in UpdateForceVars(). Locks are used on both the boolean indicating a shot is pending and all the rotation variables, since they are updated by both the serial loop (producer) and the frame updates (consumer). The complementary filter is used for rotation calculations, with a yaw calculation using the compass added. Note the flat fix on yaw drift, a phenomena briefly described here.

using UnityEngine;
using UnityEngine.UI;
using System.Collections;
using System.IO.Ports;
using System;
using System.Threading;

public class mBedController : MonoBehaviour
{
    #region Static Vars and Constants
    /// <summary>
    /// Instance of this class, used for checking initialization state
    /// </summary>
    public static mBedController Instance;

    /// <summary>
    /// Accelerometer setting for calculations
    /// </summary>
	public const float ACCEL_SET = 8192.0f;

    /// <summary>
    /// Gyroscope setting for calculations
    /// </summary>
	public const float GYRO_SET = 450f;

    /// <summary>
    /// Constant string sent from mBed over serial if the trigger has been pulled
    /// </summary>
    public const string SHOOT = "PEW";

    #endregion

    #region Properties
    /// <summary>
    /// Serial port for communication with the mBed
    /// </summary>
	private SerialPort mbed = new SerialPort ();

    /// <summary>
    /// Lock for the angles, new angles, and dt variables (all updated together by both threads)
    /// </summary>
	private System.Object flock = new System.Object ();

    /// <summary>
    /// Lock for the shooting variable
    /// </summary>
    private System.Object slock = new System.Object();

    /// <summary>
    /// The angles of the transform rotation in the last frame
    /// </summary>
	private Vector3 angles = Vector3.zero;

    /// <summary>
    /// The angles of the transform rotation for the next fram
    /// </summary>
    private Vector3 newAngles = Vector3.zero;
    
    /// <summary>
    /// String -> double parse output variable
    /// </summary>
	private double parsed = 0.0f;

    /// <summary>
    /// String accelerometer input from IMU
    /// </summary>
	private string[] accelInput = new string[3];

    /// <summary>
    /// Parsed numeric accelerometer input
    /// </summary>
	private double[] accelValue = new double[3];

    /// <summary>
    /// String gyroscope input from IMU
    /// </summary>
	private string[] gyroInput = new string[3];

    /// <summary>
    /// Parsed numeric gyroscope input
    /// </summary>
	private double[] gyroValue = new double[3];

    /// <summary>
    /// String compass input from IMU
    /// </summary>
    private string[] compInput = new string [3];

    /// <summary>
    /// Parsed numeric compass input
    /// </summary>
    private double[] compValue = new double[3];

    /// <summary>
    /// Raw string input containing accelerometer, gyroscope, and compass data
    /// </summary>
	private string[] dataInput = new string[3];

    /// <summary>
    /// The time between frames (update only with FLOCK)
    /// </summary>
    private float dt = .01f;

    /// <summary>
    /// Thread for serial communication with mBed
    /// </summary>
	private Thread SerialThread;

    /// <summary>
    /// Whether the trigger has been pulled (and shot not fired), update only with SLOCK
    /// </summary>
    private bool shoot = false;
	
	/// <summary>
    /// COM port number, must be initialized prior to game starting
    /// </summary>
	public string ComPort = "COM2";
	
    /// <summary>
    /// Baud rate, initialize prior to game starting
    /// </summary>
	public int BaudRate = 9600;

    /// <summary>
    /// Read timeout for mBed, initialize prior to game starting
    /// </summary>
	public int ReadTimeout = 10;

    /// <summary>
    /// Write timeout for mBed, initialize prior to game starting
    /// </summary>
	public int WriteTimeout = 10;
	
	/// <summary>
    /// Whether to keep the serial thread loop running
    /// </summary>
	private bool isRunning = false;

    /// <summary>
    /// Public accessor for serial isRunning
    /// </summary>
	public bool IsRunning {
		get { return isRunning; }
		set { isRunning = value; }
	}

    /// <summary>
    /// Storage for character movement (after collisions, etc) - gun follows the character
    /// </summary>
    public Vector3 movement;

    /// <summary>
    /// Gun barrel from which shooting graphics are generated
    /// </summary>
    public GameObject barrel;
	#endregion
	
	#region Serial Thread

    /// <summary>
    /// Attempts to open the serial port and launch the serial thread
    /// </summary>
	public void OpenSerialPort ()
	{
		try {
			mbed = new SerialPort (ComPort, BaudRate);

			mbed.ReadTimeout = ReadTimeout;
			mbed.WriteTimeout = WriteTimeout;
			
			mbed.Open ();
			
			if (SerialThread == null) {
				StartSerialThread ();
			}
		} catch (Exception ex) {
			// Failed to open com port or start serial thread
			Debug.Log ("Error 1: " + ex.Message.ToString ());
		}
	}
	
    /// <summary>
    /// Attempts to close the serial port
    /// </summary>
	public void CloseSerialPort ()
	{
		try {
			// Close the serial port
			mbed.Close ();
			
		} catch (Exception ex) {
			if (mbed == null || mbed.IsOpen == false) {
				// Failed to close the serial port. Uncomment if
				// you wish but this is triggered as the port is
				// already closed and or null.
				
				// Debug.Log("Error 2A: " + "Port already closed!");
			} else {
				// Failed to close the serial port
				Debug.Log ("Error 2B: " + ex.Message.ToString ());
			}
		}
		
		print ("Serial port closed!");
	}
	
    /// <summary>
    /// Launches the serial thread loop
    /// </summary>
	public void StartSerialThread ()
	{
		try {
			// define the thread and assign function for thread loop
			SerialThread = new Thread (new ThreadStart (SerialThreadLoop));
			// Boolean used to determine the thread is running
			isRunning = true;
			// Start the thread
			SerialThread.Start ();
			
			print ("Serial thread started!");
		} catch (Exception ex) {
			// Failed to start thread
			Debug.Log ("Error 3: " + ex.Message.ToString ());
		}
	}
	
    /// <summary>
    /// Runs serial thread as long as isRunning is set to TRUE
    /// </summary>
	private void SerialThreadLoop ()
	{
		while (isRunning) {
			GenericSerialLoop ();
		}
		print ("Ending Thread");
	}
	
    /// <summary>
    /// Stops and kills the serial thread
    /// </summary>
	public void StopSerialThread ()
	{
		// Set isRunning to false to let the while loop
		// complete and drop out on next pass
		isRunning = false;
		
		// Pause a little to let this happen
		Thread.Sleep (100);
		
		// If the thread still exists kill it
		// A bit of a hack using Abort :p
		if (SerialThread != null) {
			SerialThread.Abort ();
			// serialThread.Join();
			Thread.Sleep (100);
			SerialThread = null;
		}
		
		// Reset the serial port to null
		if (mbed != null) {
			mbed = null;
		}
		
		print ("Ended Serial Loop Thread!");
	}
	
    /// <summary>
    /// Serial loop
    /// </summary>
	private void GenericSerialLoop ()
	{
		try {
			// Check that the port is open. If not skip and do nothing
			if (mbed.IsOpen) {
				// Read serial data until an '\r\n' is recieved (or timeout)
				string rData = mbed.ReadLine ();

                // Check if data has been recieved
                if (!string.IsNullOrEmpty(rData))
                {
                    //Check if a shot was fired
                    if (rData.StartsWith(SHOOT))
                    {
                        lock (slock) { shoot = true; }
                    }
                    else
                    {
                        dataInput = rData.Split('$');
                        //Check for valid IMU data
                        if (dataInput.Length == 3)
                            UpdateForceVars();
                    }
                }
			}
		} catch (TimeoutException timeout) {
            Debug.Log("Timeout");
		} catch (Exception ex) {
			// This could be thrown if we close the port whilst the thread 
			// is reading data. So check if this is the case!
			if (mbed.IsOpen) {
				// Something has gone wrong!
				Debug.Log ("Error 4: " + ex.Message.ToString ());
                CloseSerialPort();
			}
		}
	}
	
    /// <summary>
    /// Perform calculations on IMU data and decide transform rotation for the next frame
    /// </summary>
	private void UpdateForceVars ()
	{
		accelInput = dataInput [0].Split (' ');
		gyroInput = dataInput [1].Split (' ');
        compInput = dataInput[2].Split(' ');
        //Split up and parse data, check for valid format
        if (accelInput.Length == 3 && gyroInput.Length == 3 && compInput.Length == 3 && ConvertFloatArrays())
        {
            //Lock the rotation variables for update
            lock (flock)
            {
                newAngles.z = angles.z + (float)(gyroValue[0] / GYRO_SET) / dt;
                newAngles.x = angles.x - (float)(gyroValue[1] / GYRO_SET) / dt;
                newAngles.y = angles.y - (float)(gyroValue[2] / GYRO_SET) / dt;
                //Correction factor for yaw drift (and slightly uneven breakout board!)
                newAngles.y = newAngles.y - .45f;
                int fx = Math.Abs((int)accelValue[0]) + Math.Abs((int)accelValue[1]) + Math.Abs((int)accelValue[2]);

                //Filter out spikes and changes below the sensitivity settings
                if (fx > ACCEL_SET && fx < 52768)
                {
                    //MATH - filtering using complementary filter
                    float accZ = (float)(Math.Atan2(accelValue[1], accelValue[2]) * 180 / Math.PI);
                    newAngles.z = (.98f * angles.z) + accZ * .02f;
                    float accX = (float)(Math.Atan2(accelValue[0], accelValue[2]) * 180 / Math.PI);
                    newAngles.x = (.98f * angles.x) + accX * .02f;
                    float accY = -(float)(Math.Atan2((compValue[2] * Math.Sin(newAngles.x)) - (compValue[1] * Math.Cos(newAngles.x)),
                        (compValue[0] * Math.Cos(newAngles.z)) +
                        (compValue[1] * Math.Sin(newAngles.z) * Math.Sin(newAngles.x)) +
                        (compValue[2] * Math.Sin(newAngles.z) * Math.Cos(newAngles.x))));   
                    newAngles.y = (.98f * angles.y) + (accY * .02f);
                }
            }
        }
	}

    /// <summary>
    /// Checks for valid IMU formatting and converts to doubles
    /// </summary>
    /// <returns>Whether all nine values are valid</returns>
    private bool ConvertFloatArrays()
    {
        for (int i = 0; i<accelInput.Length;i++)
        {
            if (double.TryParse(accelInput[i], out parsed))
                accelValue[i] = parsed;
            else
                return false;
        }
        for (int i=0; i<gyroInput.Length; i++)
        {
            if (double.TryParse(gyroInput[i], out parsed))
                gyroValue[i] = parsed;
            else
                return false;
        }
        for (int i=0; i<compInput.Length; i++)
        {
            if (double.TryParse(compInput[i], out parsed))
                compValue[i] = parsed;
            else
                return false;
        }
        return true;
    }
	#endregion
	
	#region Unity Frame Events
    /// <summary>
    /// Essentially an initialization routine
    /// </summary>
	void Awake ()
	{
		Instance = this;			
	}
	
    /// <summary>
    /// Called at the beginning of the scene
    /// </summary>
	void Start ()
	{
        OpenSerialPort ();
	}

    /// <summary>
    /// Called with each iteration of physics calculations, regradless of frames/framerate
    /// </summary>
    void FixedUpdate()
    {
        //Lock the roation and dt variables
        lock (flock)
        {
            Vector3 rotation = newAngles - angles;
            //Translations for current setup (modify for different starting orientation, etc)
            rotation.x = -1 * rotation.x;
            rotation.z = -1 * rotation.z;
            transform.Rotate(rotation);
            angles = newAngles;
            //Store the dt for the next angle calculation
            dt = Time.deltaTime;
        }
        //Move the gun to the current position of the character
        GetComponent<Rigidbody>().MovePosition(movement);
    }

    /// <summary>
    /// Called each frame (independent of physics iterations/ FixedUpdate calls
    /// </summary>
    void Update()
    {
        Vector3 shootOrigin = transform.position;
        //Offsets of the shot origin from the center of the gun transform
        shootOrigin.y += .2f;
        shootOrigin.x += .2f;
        shootOrigin.z += .5f;
       //Store shot information - position and direction
        Ray ray = new Ray(barrel.transform.position, transform.forward);
        //lock the shoot variable
        lock (slock)
        {
            //Check the shoot variable
            if (shoot && barrel != null)
            {
                //Essentially a special Unity delegate operation, actual shooting performed by barrel end
                barrel.SendMessage("Shoot", ray, SendMessageOptions.RequireReceiver);
            }
            //'Consume' the shot
            shoot = false;
        }
    }

    /// <summary>
    /// (Slightly hacky) callback for character movement
    /// </summary>
    /// <param name="move"></param>
    public void UpdateMovement(Vector3 move)
    {
        movement = move;
    }
	
    /// <summary>
    /// Essentially a deconstructor
    /// </summary>
	void OnApplicationQuit ()
	{
		CloseSerialPort ();
		Thread.Sleep (500);
		StopSerialThread ();
		Thread.Sleep (500);
	}

    #endregion
}

The following snippets present the code necessary for correct execution of the mbedController above. All of the code behind game objects is not included, but this shows how the different scripts interact.

PlayerMovement is the class used to move the game character, which has to tell the mbed controller class to move as well. However, since collisions are calculated on the character after the Move() function, the current position is sent, not the user input. The character itself is moved with the arrow keys.

public class PlayerMovement : MonoBehaviour
{
    //The mbedController behind the gun
    public GameObject gun;

    //The offset of the gun from the character's transform
    Vector3 gunOffset = new Vector3(.5f, .5f, .5f);

    void FixedUpdate()
    {
        float h = Input.GetAxisRaw ("Horizontal");
        float v = Input.GetAxisRaw("Vertical");

        Move (h, v);
        Turning ();
        Animating (h, v);
        gun.SendMessage("UpdateMovement", transform.position + gunOffset, SendMessageOptions.RequireReceiver);
    }

The gun barrel itself uses a Raycast to simulate a shot and detect collisions. This is called by the mbedController.cs.

public class PlayerShooting : MonoBehaviour
{
    public int damagePerShot = 20;
    public float range = 100f;


    float timer;
    Ray shootRay;
    RaycastHit shootHit;
    int shootableMask;
    ParticleSystem gunParticles;
    LineRenderer gunLine;
    AudioSource gunAudio;
    Light gunLight;
    
    void Shoot(Ray ray)
    {
        timer = 0f;

        gunAudio.Play ();

        gunLight.enabled = true;

        gunParticles.Stop ();
        gunParticles.Play ();

        gunLine.enabled = true;
        gunLine.SetPosition (0, ray.origin);

        shootRay.origin = ray.origin;
        //shootRay.direction = transform.forward;
        shootRay.direction = ray.direction;

        if(Physics.Raycast (shootRay, out shootHit, range, shootableMask))
        {
            EnemyHealth enemyHealth = shootHit.collider.GetComponent <EnemyHealth> ();
            if(enemyHealth != null)
            {
                enemyHealth.TakeDamage (damagePerShot, shootHit.point);
            }
            gunLine.SetPosition (1, shootHit.point);
        }
        else
        {
            gunLine.SetPosition (1, shootRay.origin + shootRay.direction * range);
        }
    }
}

Playing

The hand controller needs to be flat when the game starts. Once the scene appears, it can be used to aim and shoot at the attackers. Only the rotation of the glove is calculated from the IMU, though the game object rotates on the edge of a sphere for natural-feeling movement.

Oculus Rift

The game is displayed on an Oculus Rift (OR), which is plugged into the PC, and interfaces with the Unity code using a C# wrapper. Head movements while wearing the OR control the camera in the game.

Demo

Here is the full game being played