Low Latency Camera Discussion

[technogeek6952]

Because of the large obstructions in our field of view, we (team 4068) are thinking about using a camera to aid the drivers in tasks such as lining up the gear with the peg.

We've done a fair bit with cameras and driver visibility in the past, but it's always had a significant lag time (at least one or more seconds) between when something would move and the driver would see it on the camera. This made it hard for our drivers to use, and they mostly ignored the camera, in favor of lining things up by eye from the driver station.

I'd like to know if anyone has any insights as to how to get a low-latency (ideally only a couple hundred milliseconds) camera stream to the drivers.

We're currently thinking of skipping all of the code FIRST uses for their default camera stream, and just send raw data from the camera (compressed to save bandwidth) to the driver station over one of the UDP ports they give us for our own use. We would make a companion application on the driver station to quickly draw the frames sent over UDP. We're not sure if that would do enough to reduce latency to be worth our time, though.

Has anyone done any analysis of what contributes to latency on the camera stream? Would removing the WPILib code remove much latency? Or is more time added by some other factor? What success have other teams had in removing latency in the past?

PS: We us Java, in case that's important.

[aeastet]

Take a look at this:

https://www.chiefdelphi.com/forums/s...highlight=NavX

The SF2 libraries are written for Java, C++ and LabVIEW. We have used them with teams that have frame rates as low as 5 FPS with success.

I hope this helps,

Tim

[Joe Ross]

You didn't mention what programming language you use or what resolution you had issues with in the past.

For C++ and Java, the WPILib CameraServer class has been completely rewritten to significantly optimize it. We get lower latency and much lower CPU usage compared to last year. It also sends the data as a standard mjpeg stream, so you can open it in a web browser. This is helpful because it can help determine if it's your dashboard, or something else that is causing the latency. You should give it a try this year before trying to implement your own.

In general, you want to use the smallest resolution that gets you the results you need. For a driver camera, we've been able to get away with 160x120 in the past. That requires pushing 16x less pixels then 640x480.

[slibert]

Originally Posted by Joe Ross

You didn't mention what programming language you use or what resolution you had issues with in the past.

For C++ and Java, the WPILib CameraServer class has been completely rewritten to significantly optimize it. We get lower latency and much lower CPU usage compared to last year. It also sends the data as a standard mjpeg stream, so you can open it in a web browser. This is helpful because it can help determine if it's your dashboard, or something else that is causing the latency. You should give it a try this year before trying to implement your own.

In general, you want to use the smallest resolution that gets you the results you need. For a driver camera, we've been able to get away with 160x120 in the past. That requires pushing 16x less pixels then 640x480.

Just wanted to echo Joe's sentiment here. An issue I've seen at competitions is that the bandwidth (between driver station and robot) is sometimes limited due to various environmental issues that arise during a real-world competition. FIRST has done a lot of work to improve things here, but the wireless link between the robot is susceptible to disruption from other signals (e.g., cell phones). Again this is usually not a problem but in addition to keeping your video bandwidth on the low side I'd encourage you to think through what happens if this doesn't work due to some sort of temporary disruption. I think it will likely be fine 99% of the time, but there are (likely rare) times when it may not, so think carefully before making this a feature that your robot has to have to play the game. Be sure to have a workaround in case it doesn't work during a particular match.

[BradAMiller]

Just to add one comment about the CameraServer class... Please try it early and let us know if you are having any issues including latency. I understand that the environment in your shop might not be completely the same as the field, but it would help you understand the basic performance and let you tune parameters to make it work for you. And, you can give us feedback while there's time to fix bugs or do small optimizations before the competitions start.

If you are just using the camera for getting a remote view you can try an ethernet camera connected to a switch on your robot. Then you can view the http stream on your driver station computer and not go through the roboRIO at all. However, we expect that the hit on the CPU will be minimal just for moving images through without processing.

[Joe Ross]

Originally Posted by BradAMiller

If you are just using the camera for getting a remote view you can try an ethernet camera connected to a switch on your robot. Then you can view the http stream on your driver station computer and not go through the roboRIO at all.

Be cautious with this approach also. A lot of cheap IP cameras are designed for security usage, and a second of latency isn't an issue for security, so they may have poor latency.