NYVR Hackaton: Follow-up... a work in progress - Chris LoBello

Chris LoBello

​
​IIn July, I and my team (ETC) participated in the NYVR Hackaton - which was hosted at NYU ITP.

My main goal for the Hackathon was to build a haptic feedback pointing device for VR. This blog post outlines what I worked on, my approach, and some follow-up notes from after that awesome hackaton. Hopefully, some of you will find this useful.

This hackathon was only 2 days, so I had to simplify what I wanted to make in that time. I figured only one orientation sensor should be enough to get a good pointing device working. The orientation sensor that I chose to use was the BNO055 from Adafruit. Check out the information for it here:

http://www.adafruit.com/product/2472

The best part about the BNO055 is that you can get the quaternion data out of it directly. Getting the quaternion data is incredibly useful since all rotations in Unity3D are done with quaternions. The BNO055 was connected to an Arduino Due. It uses the i2c interface and the Wire library to work. So the SDA and SCL pins from the BNO055 were connected to the Arduino D20(SDA) and D21(SCL) pins respectively.

The other pins on the BNO055 besides 5V and ground weren't used due to the speed in which I was trying to get this project done. An Arduino Uno was added which had an MP3 Shield from Sparkfun connected to it. The serial1 connection of the Due was connected to the serial connections of the Uno. You can find it and more information about it here:

https://www.sparkfun.com/products/12660

On the 2gb microSD card of the MP3 player Shield I uploaded a bunch of random light saber noises that I found online. I wasn't really concerned with what the noises were at this point, just that I had a variety of them to work with. The second Arduino was used because the MP3 Shield also uses the SPI interface to run the MP3 Shield, and the pinout is totally different for the SPI pins on the Due. I didn't have time to get it all integrated so it was just easier to add another Arduino. The output from the MP3 player shield was piped into an Adafruit 20W stereo amplifier. This is a super powerful amp for its size, you can find out all about it here:

https://learn.adafruit.com/adafruit-20w-stereo-audio-amplifier-class-d-max9744

The stereo amp was running two surface transducers. Surface transducers are basically speakers with the speaker cone removed, and a good mounting point put in its place. The surface transducers I choose for this were the Sparkfun small surface transducers. They are really loud for their size if you have them mounted solidly to a surface. These transducers are very fragile though. The best thing to do with them is to solder your speaker wire onto the connections and then cover that connection in hot glue. If you plan on handling them after they are connected your going to have the terminals rip out and it'll never work again. You can find the surface transducers here:

https://www.sparkfun.com/products/10917

This whole thing had to communicated to either a computer or smartphone wirelessly. The natural choice for that would be to use Bluetooth. Sparkfun makes an incredibly easy to use Bluetooth module called the BlueSMiRF. I went with the BlueSMiRF Gold module for this application, though the Silver edition would work just as well. You can find information on getting the BlueSMiRF bluetooth module setup here:

https://www.sparkfun.com/products/12582

All of this stuff wired together was powered by a Turnigy 3 cell LiPo pack that I pulled out of my quadcopter. Its giant and had much more power then I'd ever need for this project. At full charge a 3 cell Lipo pack gives out 4.2volts per cell which gives me 12.6 volts output. This was perfect to run the 20W amplifier, since this LiPo pack has a high C rating and will not be over burdened by sudden current draw from the amplifier. The Arduino Uno and Arduino Due were also connected to the LiPo and ran perfectly due to the internal voltage regulator on the Arduinos. The nicest part of using this LiPo pack was to connect to the Arduino through the balance charging port on the battery. When connecting through the balance charging port you can just plug standard jumper wires into the port, without having to do some hackery with alligator clips on the main output of the battery. You can get the LiPo pack here:

http://www.amazon.com/Turnigy-2200mAh-20C-Lipo-Pack/dp/B0072AEY5I/ref=sr_1_1?ie=UTF8&qid=1443729735&sr=8-1&keywords=3+cell+lipo

Here is a quick breakdown of how all of this stuff is connected. The Arduino Due is getting orientation data from the BNO055 through the I2C interface. It is also handling bluetooth communication with the BlueSMiRF. The Due is controlling the Arduino Uno with the MP3 Shield on it. The MP3 Shield is outputting sounds into the 20W amplifier and that amplifier is connected to two small surface transducers. The Arduino Uno is running the FilePlay example from here:

https://github.com/madsci1016/Sparkfun-MP3-Player-Shield-Arduino-Library/tree/master/SFEMP3Shield

Lets see how this all ties together on the Arduino Due code.

10/08/15 12:04:50 Untitled

1  // (c)2015 Evolving Technologies Corporation.    2  #include // for easy communication between Arduino and PC over bluetooth    3  #include    4  #include    5  #include #include    6  #include    7  #include <utility/imumaths.h>    8       9  #define BNO055_SAMPLERATE_DELAY_MS (100)   10      11  Adafruit_BNO055 bno = Adafruit_BNO055(55);   12      13  const char s0[] PROGMEM = "?"; const char help0[] PROGMEM = "Prints this help menu.";   14  const char s1[] PROGMEM = "dataPush"; const char help1[] PROGMEM = "dataPush 1, dataPush 0, determines whether to push data or not";   15  const char s2[] PROGMEM = "mp3rand"; const char help2[] PROGMEM = "mp3rand just plays random mp3s at a 10 second interval";   16      17  const FuncEntry_t functionTable[] PROGMEM = {   18  // String, help, Function   19  {s0, help0, printHelp },   20  {s1, help1, dataPush },   21  {s2, help2, mp3Rand }   22  };   23      24  int funcTableLength = (sizeof functionTable / sizeof functionTable[0]); //number of elements in the function table   25      26  int pushData;   27      28  MsgParser myParser; //this creates our parser   29      30  void setup()   31  {   32  Serial.begin(115200);//pluggin serial monitor from the Due board over usb   33  Serial1.begin(115200);// for talking to the Arduino Uno with the MP3 Sheild on it   34  Serial.println("Ready for action!");   35      36  myParser.setTable(functionTable, funcTableLength); //tell the parser to use our lookup table   37      38  Serial.println("Orientation Sensor Test");   39      40  /* Initialise the BNO055 sensor */   41  if (!bno.begin())   42  {   43  /* There was a problem detecting the BNO055 ... check your connections */   44  Serial.print("Ooops, no BNO055 detected ... Check your wiring or I2C ADDR!");   45  }   46      47  delay(1000);   48  /* Display some basic information on this sensor */   49  displaySensorDetails();   50  bno.setExtCrystalUse(true);   51  printHelp();   52  }   53      54  void loop()   55  {   56  while ( Serial.available() ) myParser.processByte(Serial.read () );   57      58  if (pushData == 1)   59  {   60  //HPRJSON();   61  bnoQuaternion();   62  bnoXYZ();   63  bnoEuler();   64  Serial.println("");   65  }   66  }   67      68  void displaySensorDetails(void)   69  {   70  sensor_t sensor;   71  bno.getSensor(&sensor);   72  Serial.println("------------------------------------");   73  Serial.print ("Sensor: "); Serial.println(sensor.name);   74  Serial.print ("Driver Ver: "); Serial.println(sensor.version);   75  Serial.print ("Unique ID: "); Serial.println(sensor.sensor_id);   76  Serial.print ("Max Value: "); Serial.print(sensor.max_value); Serial1.println(" xxx");   77  Serial.print ("Min Value: "); Serial.print(sensor.min_value); Serial1.println(" xxx");   78  Serial.print ("Resolution: "); Serial.print(sensor.resolution); Serial1.println(" xxx");   79  Serial.println("------------------------------------");   80  Serial.println("");   81  delay(500);   82  }   83      84  void bnoXYZ()   85  {   86  /* Get a new sensor event */   87  sensors_event_t event;   88  bno.getEvent(&event);   89      90  /* Display the floating point data */   91  Serial.print("xyzOrientation, ");   92  Serial.print(event.orientation.x, 2);   93  Serial.print(", ");   94  Serial.print(event.orientation.y, 2);   95  Serial.print(", ");   96  Serial.print(event.orientation.z, 2);   97  Serial.print(", ");   98  //delay(BNO055_SAMPLERATE_DELAY_MS);   99  }  100     101  void bnoQuaternion()  102  {  103  // Possible vector values can be:  104  // - VECTOR_ACCELEROMETER - m/s^2  105  // - VECTOR_MAGNETOMETER - uT  106  // - VECTOR_GYROSCOPE - rad/s  107  // - VECTOR_EULER - degrees  108  // - VECTOR_LINEARACCEL - m/s^2  109  // - VECTOR_GRAVITY - m/s^2  110  imu::Vector euler = bno.getVector(Adafruit_BNO055::VECTOR_EULER);  111  /* Display the floating point data */  112  // Quaternion data  113  imu::Quaternion quat = bno.getQuat();  114  Serial.print("qWXYZcsv, ");  115  Serial.print(quat.w(), 6);  116  Serial.print(", ");  117  Serial.print(quat.y(), 6);  118  Serial.print(", ");  119  Serial.print(quat.x(), 6);  120  Serial.print(", ");  121  Serial.print(quat.z(), 6);  122  Serial.print(",");  123  }  124     125  void bnoEuler()  126  {  127  // Possible vector values can be:  128  // - VECTOR_ACCELEROMETER - m/s^2  129  // - VECTOR_MAGNETOMETER - uT  130  // - VECTOR_GYROSCOPE - rad/s  131  // - VECTOR_EULER - degrees  132  // - VECTOR_LINEARACCEL - m/s^2  133  // - VECTOR_GRAVITY - m/s^2  134     135  imu::Vector euler = bno.getVector(Adafruit_BNO055::VECTOR_EULER);  136     137  /* Display the floating point data */  138  Serial.print("eulerXYZ, ");  139  Serial.print(euler.x());  140  Serial.print(", ");  141  Serial.print(euler.y());  142  Serial.print(", ");  143  Serial.print(euler.z());  144  Serial.print(",");  145     146  }  147     148  //MsgParser helper functions  149  void printHelp()  150  {  151  //For each command that the parser knows about...  152  for ( int i = 0; i < myParser.numCmds(); i++)  153  {  154  Serial.print(myParser.cmdString(i)); //print the command name  155  Serial.print(" - ");  156  Serial.print(myParser.cmdDesc(i)); //print the command description  157  Serial.println("");  158  }  159  }  160     161  void commandNotFound(uint8_t* pCmd, uint16_t length)  162  {  163  Serial.print("Command not found: ");  164  Serial.write(pCmd, length); //print out what command was not found  165  Serial.println(); //print out a new line  166  }  167     168  void dataPush()  169  {  170  pushData = myParser.getInt();  171  }  172     173  void mp3Rand()  174  {  175  int mp3Number;  176     177  mp3Number = random(0,50); //get the first number passed in  178     179  Serial1.println("s");  180  Serial1.println(mp3Number);  181  }

If we connect to the BlueSMiRF over your serial this is what you'll be greeted with when you start it up.

---

Ready for action!
Orientation Sensor Test
------------------------------------
Sensor: BNO055
Driver Ver: 1
Unique ID: 55
Max Value: 0.00 xxx
Min Value: 0.00 xxx
Resolution: 0.01 xxx
------------------------------------</code>

? - Prints this help menu.
dataPush - dataPush 1, dataPush 0, determines whether to push data or not
mp3rand - mp3rand, plays a random MP3
qWXYZcsv, 0.993957, 0.098633, 0.048218, 0.000061,xyzOrientation, 359.94, -11.25, -5.56, eulerXYZ, 359.94, -11.25, -5.56,
qWXYZcsv, 0.993957, 0.098633, 0.048218, 0.000061,xyzOrientation, 359.94, -11.25, -5.56, eulerXYZ, 359.94, -11.25, -5.56,
qWXYZcsv, 0.993957, 0.098633, 0.048218, 0.000061,xyzOrientation, 359.94, -11.25, -5.56, eulerXYZ, 359.94, -11.25, -5.56,
qWXYZcsv, 0.993957, 0.098633, 0.048218, 0.000061,xyzOrientation, 359.94, -11.25, -5.56, eulerXYZ, 359.94, -11.25, -5.56,
qWXYZcsv, 0.993957, 0.098633, 0.048218, 0.000061,xyzOrientation, 359.94, -11.25, -5.56, eulerXYZ, 359.94, -11.25, -5.56,

---

First it checks that the BNO055 is working and outputs some mostly unnecessary sensor information. Then it prints out the help menu. When the help menu is up I sent in the command "dataPush 1" followed by a carriage return. This then set pushData to 1 and caused it to start reading and outputting the orientation data from the BNO055. The orientation data first is the quaternion data in format WXYZ, then we have it in XYZ, then as its Euler angles. XYZ and Euler are equivalent.

Nows the tricky part. Getting Unity3D to get that orientation data and do something useful with it. Startup Unity and make a new empty project. Make a cylinder game object and place it anywhere within the scene. Then go into build settings &gt; player settings&gt; other settings&gt; optimization&gt;api compatibility level, and set that to ".Net 2.0" . We need it set to .Net 2.0 not subset so that we have full access to the serial ports on the computer.

I made a new script called bnoRotator.cs and attached that to the cylinder. Here it is:

10/08/15 12:23:18 Untitled

1  // (c)2015 Evolving Technologies Corporation.   2  using UnityEngine;   3  using System.Collections;   4  using System.IO.Ports;   5      6  public class bnoRotator : MonoBehaviour {   7      8  public static SerialPort sp = new SerialPort("/dev/cu.bluesmirf-SPP",115200);//This connects to the bluesmirf serial port   9     10  // -----------------------  11  public string message2; // data string from read line.  12  public string[] values; // parsed values from the incoming stream of data.  13     14  public static float flag01_x = 1f;  15  public static float flag02_y = 1f;  16  public static float flag03_z = 1f;  17     18  // -----------------------  19  void Awake() {  20  HideChildren();  21  }  22     23  void Start () {  24     25  OpenConnection();  26  sp.Write ("dataPush 1");  27  sp.Write ("\r");  28  }  29     30  void Update () {  31     32  // Read the data.  33  message2 = sp.ReadLine();  34  print(message2);  35     36  // Parse the data.  37  values = message2.Split(',');  38     39  // Procss the data.  40  // -----------------------  41  // This should really be handled by a separate function.  42     43  try {  44     45  print (  46  (Mathf.Sign(float.Parse(values[6]))>0?"+":"-")+""+  47  (Mathf.Sign(float.Parse(values[7]))>0?"+":"-")+""+  48  (Mathf.Sign(float.Parse(values[8]))>0?"+":"-")+""+  49     50  " v1: " + values[6] + ", v2: " + values[7] + ", v3: " + values[8]  51     52  );  53     54  float x_roll_banking = Mathf.Round(float.Parse(values[8]) * -1f * flag01_x) / 1f;  55  float y_head_yaw = Mathf.Round(float.Parse(values[6]) * 1f * flag02_y) / 1f;  56  float z_pitch = Mathf.Round(float.Parse(values[7]) * -1f * flag03_z) / 1f;  57     58  if (Mathf.Abs(z_pitch) >= 15f) {  59  sp.Write("mp3Rand");  60  sp.Write ("\r");  61  }  62     63  transform.localEulerAngles = new Vector3(x_roll_banking, y_head_yaw, z_pitch);  64     65  } catch{print("error");}  66     67  }  68     69  public void OpenConnection() {  70  if (sp != null) {  71  if (sp.IsOpen) {  72  sp.Close ();  73  print ("Closing port, because it was already open!");  74  } else {  75  sp.Open (); // opens the connection  76  sp.ReadTimeout = 500; // sets the timeout value before reporting error  77  print ("Port Opened!");  78  message2 = "Port Opened!";  79  }  80  } else {  81  if (sp.IsOpen) {  82  print ("Port is already open");  83  } else {  84  print ("Port == null");  85  }  86  }  87  }  88     89  void OnApplicationQuit()  90  {  91  sp.Close();  92  }  93     94  }