Wearables in FRC

[Michael Corsetto]

Originally Posted by marshall

For the past few seasons some of our students have engaged in some research projects around wearable technology. Both haptic feedback for the drivers as well as connecting our drive team to sensors for data acquisition and analysis.

We use the rumble in our xbox 360 controller to notify the operator when we have a ball in the intake. The LED's on the robot change color too, but sometimes the LED's are hard to see across the field (aka Drawbridge).

-Mike

[marshall]

Originally Posted by Michael Corsetto

We use the rumble in our xbox 360 controller to notify the operator when we have a ball in the intake. The LED's on the robot change color too, but sometimes the LED's are hard to see across the field (aka Drawbridge).

-Mike

We've done that too. We'd like to do more but it's difficult at the moment though we have some ideas about how to make it happen.

Fun fact, in 2014 we were beta testing the new control system and one of the promised features was an update to enable rumble on controllers. It didn't come out when anticipated so we filed a bug report/request to get it added with a reason of "our driver is blind". I don't recall the answer we received but it was pretty funny at the time.

[udpatil]

Oh man, have I been waiting for a thread like this for a while. So this year, the programming team for 540 used wearables for the drive team itself. We used the Vuzix m100 smart glasses which run on Android as a substitute for the driver station's dashboard. Essentially, information such as whether we had a ball in the intake or if we were aligned to shoot were sent to the Vuzix smart glasses and were displayed as lights in the Android app we developed. This implementation was AFAIK legal because it used a wired connection (port forward from laptop to smart glasses using ADB interface through a USB cable) and the secondary driver didn't put it on until after autonomous ended (took him like one whole second). I'm super excited to have led this project and hopefully our team will use such technology in future years to provide intuitive data to the drivers! Don't hesitate to PM me about how the smart glasses were implemented because I'd love to explain it and hopefully increase the usage of cool new technology in FRC

EDIT: For anyone wondering what this looked like to the driver, imagine holding your phone in landscape mode almost an arms length away in the top right of your FOV. It was very natural for the secondary driver to glance up to check if we were aligned to shoot or if we successfully picked up a ball into the intake

[messer5740]

Could you use a FatShark RC viewer to replace a monitor to view video feed from the robot? Here is a lik to check out the products:
http://www.fatshark.com

[udpatil]

Originally Posted by messer5740

Could you use a FatShark RC viewer to replace a monitor to view video feed from the robot? Here is a lik to check out the products:
http://www.fatshark.com

I don't think those would be allowed. The main reason is that it prevents the drivers from viewing the field which is a safety issue. We ran into a similar problem last year when developing with the Oculus Rift DK2.

[messer5740]

Originally Posted by udpatil

I don't think those would be allowed. The main reason is that it prevents the drivers from viewing the field which is a safety issue. We ran into a similar problem last year when developing with the Oculus Rift DK2.

Ok I can see that. Maybe smart glasses?

[marshall]

Originally Posted by udpatil

Don't hesitate to PM me about how the smart glasses were implemented because I'd love to explain it and hopefully increase the usage of cool new technology in FRC

EDIT: For anyone wondering what this looked like to the driver, imagine holding your phone in landscape mode almost an arms length away in the top right of your FOV. It was very natural for the secondary driver to glance up to check if we were aligned to shoot or if we successfully picked up a ball into the intake

I think we're all going to need more details and some pictures.

[plnyyanks]

Somebody proposed a feature to TBA to integrate with Pebble time recently. Is this kind of integration something we should pursue (maybe in tandem with more features for Android wear)?

Does somebody want to help build it (because my time is finite, sadly)?

[udpatil]

This is what it looked like when the driver was wearing it, with this picture being taken before the match: https://drive.google.com/file/d/0B_S...QwSVBFekU/view

Here you can see the secondary driver wearing it during the match as well:
https://drive.google.com/file/d/0B_S...YtUGFxOVE/view

So, time for some technical talk
We hosted network tables on our robot to which both the robot's sensors (infrared, pressure, gyro, etc.) and data from the coprocessor (raspberry pi with pi camera tracking with OpenCV). This data was updated throughout the match, and we had a jar file running on the driver station laptop using the windows networktables library to access the network tables on the robot. When setting up the Vuzix smart glasses, an ADB port forward was used to connect a port on the laptop to the smart glasses. The jar on the laptop created a socket at the same port and the Android app created using Android Studio read from the appropriate TCP ports. Then, the data was received by the smart glasses and depending on the values that the glasses received, lights were colored or uncolored on the android app.

Lemme know if any other details are requested

[gblake]

Originally Posted by ASD20

Did you hear that at a driver's meetings? I never heard anything about it at my competitions and I never was confronted about wearing one. Since no one said anything, I just did not USE my watch on the field as the rules say rather than not WEAR it.

Just wear a wide sweatband that you pull down to cover the watch ...

[marshall]

Originally Posted by udpatil

This is what it looked like when the driver was wearing it, with this picture being taken before the match: https://drive.google.com/file/d/0B_S...QwSVBFekU/view

Here you can see the secondary driver wearing it during the match as well:
https://drive.google.com/file/d/0B_S...YtUGFxOVE/view

So, time for some technical talk
We hosted network tables on our robot to which both the robot's sensors (infrared, pressure, gyro, etc.) and data from the coprocessor (raspberry pi with pi camera tracking with OpenCV). This data was updated throughout the match, and we had a jar file running on the driver station laptop using the windows networktables library to access the network tables on the robot. When setting up the Vuzix smart glasses, an ADB port forward was used to connect a port on the laptop to the smart glasses. The jar on the laptop created a socket at the same port and the Android app created using Android Studio read from the appropriate TCP ports. Then, the data was received by the smart glasses and depending on the values that the glasses received, lights were colored or uncolored on the android app.

Lemme know if any other details are requested

Did the drivers connect a USB cable before each match or did they not wear glasses before the match or attach it? Did you have any issues with this with the refs? How did you guys come about the Vuzix and what's the story behind it?

[udpatil]

Did the drivers connect a USB cable before each match or did they not wear glasses before the match or attach it? Did you have any issues with this with the refs? How did you guys come about the Vuzix and what's the story behind it?

Yes, the smart glasses were connected via USB before the match began because the most time required to setup the glasses was when running the jar file. We talked to the refs at our competition and they said as long as the operator isn't wearing them while they are attached during autonomous we are fine. So, we just leave them on the driver station during autonomous.

The operator is wearing safety glasses and the vuzix smart glasses came with their own safety glasses and mount (how convenient). As you can see in that second picture (linked again: https://drive.google.com/file/d/0B_S...YtUGFxOVE/view), the operator puts the vuzix smart glasses on while pushing the original safety glasses above his head so that he can use the smart glasses while still being safe by wearing safety glasses.

The story behind Vuzix:
Last year (2015 Recycle Rush), I contacted Oculus to see if they would be willing to donate a VR headset so that our programming team would be able to experiment and create new, creative ways to control the robot and view it's camera feed. We even had the Oculus Rift DK2 working with stereoscopic camera feeds from two Axis cameras, but we weren't able to use it because it obscured the operator's view (safety issue).
So, this year (2016 Stronghold), I began researching augmented reality as an alternative to the virtual reality we experimented with last year. I found the Vuzix m100 as a good solution because they used Android (which I know how to program for) and had a preexisting safety glass mount. I was able to get in contact with the CEO of Vuzix (praise LinkedIn) and he directed me to one of his associates who was able to get us the m100 smart glasses with the condition that we could do some PR stuff for them (send pictures and describe what were accomplishing etc). Because we were able to get the smart glasses for free, we had the resources to experiment with this implementation of wearables which we thought worked out wonderfully.

[marshall]

Originally Posted by udpatil

Yes, the smart glasses were connected via USB before the match began because the most time required to setup the glasses was when running the jar file. We talked to the refs at our competition and they said as long as the operator isn't wearing them while they are attached during autonomous we are fine. So, we just leave them on the driver station during autonomous.

The operator is wearing safety glasses and the vuzix smart glasses came with their own safety glasses and mount (how convenient). As you can see in that second picture (linked again: https://drive.google.com/file/d/0B_S...YtUGFxOVE/view), the operator puts the vuzix smart glasses on while pushing the original safety glasses above his head so that he can use the smart glasses while still being safe by wearing safety glasses.

The story behind Vuzix:
Last year (2015 Recycle Rush), I contacted Oculus to see if they would be willing to donate a VR headset so that our programming team would be able to experiment and create new, creative ways to control the robot and view it's camera feed. We even had the Oculus Rift DK2 working with stereoscopic camera feeds from two Axis cameras, but we weren't able to use it because it obscured the operator's view (safety issue).
So, this year (2016 Stronghold), I began researching augmented reality as an alternative to the virtual reality we experimented with last year. I found the Vuzix m100 as a good solution because they used Android (which I know how to program for) and had a preexisting safety glass mount. I was able to get in contact with the CEO of Vuzix (praise LinkedIn) and he directed me to one of his associates who was able to get us the m100 smart glasses with the condition that we could do some PR stuff for them (send pictures and describe what were accomplishing etc). Because we were able to get the smart glasses for free, we had the resources to experiment with this implementation of wearables which we thought worked out wonderfully.

Very cool and great use of LinkedIn!

[Karibou]

Originally Posted by udpatil

(praise LinkedIn)

This is literally the only successful LinkedIn interaction that I've heard of. And here I was, thinking it was just something to update when I'm really trying to procrastinate at work...

But seriously, that's a great story and a really neat use of a) networking and b) technology. Great job!