360 Degree Video... What do you guys think?

[TheMilkman01]

We got our first 360 Degree video at champs and recorded this bit with Team 5024 Raider Robotics.

That got me thinking: are there any practical uses for these types of cameras in FIRST beyond experiencing competitions in virtual reality? If we could get a live feed and protection, how far-fetched is the idea that these cameras, with a VR headset, might become the driver's robot camera?

Granted, they are pretty expensive right now and the quality isn't that great for the price, but the way technology moves so fast, you never know.

[EmileH]

I'd doubt that with the current FMS limitations of 7mbps max bandwidth that it could be used for driver viewing, but in the future if the technology allows, it could be really cool to drive with a VR headset. There's the possibility of only sending back the part of the video feed that the driver headset is aimed at and using on-robot software to move and crop the image to only what the driver needs to see, to save bandwidth.

TL;dr- it's a great idea in concept but would take lots of software to make it work well for driver vision.

[Eric Scheuing]

I think that there are too many factors here to make a call about whether it would be good or not. I know that our team's driver spends most of the match watching the field instead of the camera feed from the robot. I would be concerned with using a headset because it would mean either only watching from the robot's point of view, and not seeing a majority of the field at any one time, or taking his hands off the controller momentarily to remove the headset, which could also be an issue in the heat of competition.

I would love to see the stream/youtube uploads in 360 though. It would allow me to watch exactly what I want to watch during each match.

EDIT:

Originally Posted by EmileH

I'd doubt that with the current FMS limitations of 7mbps max bandwidth that it could be used for driver viewing

Also this.

[AlexanderTheOK]

I've done this before outside of FIRST. Bottom line is that to get anything NOT nausea inducing over MJPEG you need about 25 mbps. That's two 640 by 480 streams. There would need to be some serious changes to the FMS before it was possible to do.

From personal experience, though, it's fun as all heck to drive a robot FPV with full stereo and 360 vision.

[Jaci]

There are some technologies that can make this possible over FMS with a few clever tricks.

1st is only rendering what you have to. Think about it, your eyes don't have a 360 field of view, so why would you send the entire 360 video? Only send the data you have to, along with a bit extra to account for if the user moves between frame timings. This is how facebook does its 360 video.

2nd is compression. I've had success with LZ4 sending high-quality image frames over the network in small sizes. With some tuning and a good enough processor (the Jetson TK1 is on FIRST Choice now, why not take advantage of those CUDA cores?), it is very possible to send a 720p+ image over FMS and have it decoded on the other end with a framerate well beyond anything that you will get with an AXIS camera or similar.

Is it practical? Nope. Is it possible? Yup.

[lurker]

This is off topic but shout out to 5024, they played some of the most intelligent and effective defense in Stronghold. Awesome team.

[pandamonium]

Why the focus on using this in a match. Isn't this potentially the best tool for driver practice and post match debrief ever?

[Lobão]

Very cool idea
Don´t know much about it to participate on the discussion. If it is technically possible, it will be an awesome idea.

I´m just curious to know which camera are you using.

[TheMilkman01]

Originally Posted by Lobão

I´m just curious to know which camera are you using.

I believe we were using the Ricoh Theta 360 Degree Camera. Hope this helps

[KPSch]

Wouldn't it be awesome for matches to be streamed or recorded in 360? Oh heck, even 180 degrees from a camera on the sidelines. You could watch matches with Google Cardboard or that Viewmaster thing.

[generaldgibson]

Perhaps a more efficient way to do this is to have something like the Microsoft Hololens, where you can still see the field as you would without the headset, but you can also toggle on and off an overlay and have a sort of pop-up view of the robot's POV. Just an idea, but given the price of the Hololens/stage where the technology is at right now, it would probably be a long time before this is possible. Still, just a thought

[dirtbikerxz]

Things that need to happen for this to work (I think):

• FMS has to allow for a 1080p stream at 90 fps (minimum..to not have motion sickness)

• There has to be absolutely no lag between the camera and the headset (that means that the camera has to have no long from just the movement of the and also the movement of the headset)

• You have to be capable of mounting a camera on the bot (especially in this game) that will be unobstructed to view in all angles

• Have the money for the cost of the camera, headset (I have a Vive... and let me tell you.. it is not cheap), and laptop that would be powerful enough to run it

• I doubt the RoboRio is capable of handling this sort of work... might be wrong)...

• Safety concerns with a Driver moving rather erratically especially since they wont be able to see their own surroundings, and are also tethered to a cable that is connected to the driver station... (I personally have tipped my desktop twice while dodging bullets in my Vive)

And probably a lot more, but even if all of this happened, I would not want to use this. This year our bot had a camera, and we also had a pole camera. I almost never looked at my bot's camera, and I only looked at my pole camera maybe 5 times a regional, when my view was obstructed by a drawbridge. Cameras simply don't give me the whole picture that is crucial to being a good driver. Being able to know where the next ball is on the other side of the field, know which bots are coming to play defense on you, and how you can avoid them, or even looking at the opponent alliances human player station, so you can know when their human player is about to roll a ball, and you can intercept it.... all things that a camera could not have shown.

[dirtbikerxz]

Originally Posted by generaldgibson

Perhaps a more efficient way to do this is to have something like the Microsoft Hololens, where you can still see the field as you would without the headset, but you can also toggle on and off an overlay and have a sort of pop-up view of the robot's POV. Just an idea, but given the price of the Hololens/stage where the technology is at right now, it would probably be a long time before this is possible. Still, just a thought

I agree. (If all the bandwidth issues and etc were fixed), I would love something like this. It was really annoying this year, when I had to break my attention away from the field, and stare at the computer screen to look at our pole camera. Being able to just glance into the corner of your peripheral view would be so much easier and quicker.

[ctt956]

This is awesome! I can see how it would be good for tours of the pits. And what if it could be combined with a Leap Motion Controller AND a VR headset? One question about the Theta, can you put an SD card(or flash drive or other media) in it to expand the storage? The website says it only has enough storage for a few minutes of video...

[TheMilkman01]

Originally Posted by dirtbikerxz

Things that need to happen for this to work (I think):

• FMS has to allow for a 1080p stream at 90 fps (minimum..to not have motion sickness)

• There has to be absolutely no lag between the camera and the headset (that means that the camera has to have no long from just the movement of the and also the movement of the headset)

• You have to be capable of mounting a camera on the bot (especially in this game) that will be unobstructed to view in all angles

• Have the money for the cost of the camera, headset (I have a Vive... and let me tell you.. it is not cheap), and laptop that would be powerful enough to run it

• I doubt the RoboRio is capable of handling this sort of work... might be wrong)...

• Safety concerns with a Driver moving rather erratically especially since they wont be able to see their own surroundings, and are also tethered to a cable that is connected to the driver station... (I personally have tipped my desktop twice while dodging bullets in my Vive)

And probably a lot more, but even if all of this happened, I would not want to use this.

The practicality just isn't there right now, and may never be, but driving a robot with a VR headset for demos or even in free time sounds like a really fun idea. If it ever could be implemented, I'm sure it wouldn't be the primary driver or coach. But here's another idea: having a VR headset that an audience member can wear and experience a match from the perspective of the robot. That would be sick and definitely put spectating at a whole other level.

Originally Posted by ctt956

This is awesome! I can see how it would be good for tours of the pits. And what if it could be combined with a Leap Motion Controller AND a VR headset? One question about the Theta, can you put an SD card(or flash drive or other media) in it to expand the storage? The website says it only has enough storage for a few minutes of video...

Website says 25 minutes long of continuous shooting. Not total storage. It has 8 GB of memory and – I quote from the website – a total video recording time of "(L) approx. 65 minutes, (M) approx. 175 minutes." I assume the L and M stand for Large and Medium, but that's just guesswork there. If your not satisfied with this model I'd suggest looking for other brands and options too, there's some good stuff out there – just gotta budget effectively