Streaming video from TX1 to driver station via NetworkTables

Anon10W1z · February 16, 2017, 3:11pm

Hi Chief,

Our team has successfully implemented a method of streaming video from a coprocessor to the driver station. We first convert our OpenCV frame into a JPEG image via the PIL library, and we also resize the image to decrease bandwidth consumption. We then use BytesIO to write the image to a byte] and then use the putRaw method of NetworkTables to send it over the network. On the Driver Station laptop, we have programmed a separate application to receive, decode, and display incoming frames. We have capped the frames sent per second in the code on the Jetson to 20 FPS. We measured the size of each byte array, and it is about 5 to 10 kilobytes per image, and at a rate of 20 FPS our max bandwidth consumption is about 1.6 megabits per second, far less than the 6.9 megabits per second limit. Are there any other concerns or issues with our methodology?

solomondg · February 16, 2017, 3:43pm

Should be fine, can’t see anything wrong with it.

Peter_Johnson · February 16, 2017, 5:28pm

Anon10W1z:

Our team has successfully implemented a method of streaming video from a coprocessor to the driver station. We first convert our OpenCV frame into a JPEG image via the PIL library, and we also resize the image to decrease bandwidth consumption. We then use BytesIO to write the image to a byte] and then use the putRaw method of NetworkTables to send it over the network. On the Driver Station laptop, we have programmed a separate application to receive, decode, and display incoming frames. We have capped the frames sent per second in the code on the Jetson to 20 FPS. We measured the size of each byte array, and it is about 5 to 10 kilobytes per image, and at a rate of 20 FPS our max bandwidth consumption is about 1.6 megabits per second, far less than the 6.9 megabits per second limit. Are there any other concerns or issues with our methodology?

While it’s impressive that it works, NetworkTables isn’t really designed for this. In particular, it will send a full copy of the image to every NetworkTables client. You will also see significantly increased latency versus other methods because the image needs to traverse from the coprocessor to the roboRIO and from the roboRIO to all other clients, at each step incurring a delay as the NT dispatcher runs only every 100 ms by default (so you’ll see at least 0.2 seconds of latency you wouldn’t with other methods). Also, if you have more than one dashboard application running on your driver station (even if it’s not actually viewing video), each one will be separately sent the image, linearly increasing the bandwidth consumption.

There are many posts on CD regarding how to do this much more easily using standard protocols like MJPEG over HTTP directly from the coprocessor to the driver station, which also would have avoided the need for developing a custom application to display the video stream. There’s information on this on screensteps as well, e.g. http://wpilib.screenstepslive.com/s/4485/m/24194/l/682949-vision-processing-on-an-arm-coprocessor and http://wpilib.screenstepslive.com/s/4485/m/24194/l/687863-off-board-vision-processing-in-java

Anon10W1z · February 16, 2017, 6:38pm

Peter Johnson:

While it’s impressive that it works, NetworkTables isn’t really designed for this. In particular, it will send a full copy of the image to every NetworkTables client. You will also see significantly increased latency versus other methods because the image needs to traverse from the coprocessor to the roboRIO and from the roboRIO to all other clients, at each step incurring a delay as the NT dispatcher runs only every 100 ms by default (so you’ll see at least 0.2 seconds of latency you wouldn’t with other methods). Also, if you have more than one dashboard application running on your driver station (even if it’s not actually viewing video), each one will be separately sent the image, linearly increasing the bandwidth consumption.

There are many posts on CD regarding how to do this much more easily using standard protocols like MJPEG over HTTP directly from the coprocessor to the driver station, which also would have avoided the need for developing a custom application to display the video stream. There’s information on this on screensteps as well, e.g. http://wpilib.screenstepslive.com/s/4485/m/24194/l/682949-vision-processing-on-an-arm-coprocessor and http://wpilib.screenstepslive.com/s/4485/m/24194/l/687863-off-board-vision-processing-in-java

Thank you for the informative reply! We did these shenanigans as a last-resort due to some issues with getting MJPEG working, but I guess I can take more stabs at it. I do agree it’s a bit shady what we have going.