Patched Widget for USB Camera in SmartDashboard

[RyanCahoon]

Hi all-
After hearing of and experiencing the issue with streaming the USB Camera to the SmartDashboard, I investigated the issue some, and present here what I feel is an optimal solution.

The problem appears to be that the frames being sent from the camera driver are being extracted from an MJPEG stream, and thus are missing a part of the JPEG header called the Huffman Table.

As previously noted, one solution is to use the code from the Intermediate Vision template, which works because it re-encodes the JPEG before sending it to the driver station. However, this means that the RoboRIO has to decode, then encode the JPEG image for each frame the camera sends. Also, the Intermediate Vision example places the image processing in the main control loop, which can interfere with robot control functions. Ideally, we would be able to use the CameraServer class, which doesn't process the image data and runs in separate threads from the main robot loop, reducing latency for both the camera stream and robot control.

Tl;Dr: I've created a modified version of the "USB Webcam Viewer" applet that repairs the images coming from the USB camera. Download it here, and extract it in your user directory (usually C:\Users\<username>), then add the "USB Webcam Viewer (Compat)" in SmartDashboard. More instructions here.

You can download the source here.

It reinserts the Huffman Table at the correct place in the image data before decoding the JPEG for display. This lets the driver station computer (which is likely much faster than the RoboRIO) do the work of repairing the image data, and does it much more efficiently than having to re-encode the entire image.

A slightly more efficient solution might be to use a JPEG decoder that's agnostic to the Huffman Table being missing, but the Java JPEG decoder is so entwined with the AWT framework that replacing it in a manner that wouldn't lead to more inefficiencies would be extremely difficult. In addition, the presented solution requires only a handful of memory reads and two block memory copies, so it should be very efficient.

Thanks to @TheChewanater on Reddit for posting the original code to the USB Camera Viewer widget. The algorithm to fix the JPEG header was adapted from here.

[Javaru]

Ryan,

Many, many thanks for your post. Your fix is working like a charm. You saved us a lot of headaches during crunch weekend.

Regards,
Mark

[eddie12390]

Does there need to be anything in the code that is running on the robot? The solution from the Intermediate Vision example is working correctly but when using the new widget I just get red box stating "Error: Connection refused: Connect".

[KevinWang]

Thankyou so much, works really well.

[RyanCahoon]

Originally Posted by eddie12390

Does there need to be anything in the code that is running on the robot? The solution from the Intermediate Vision example is working correctly but when using the new widget I just get red box stating "Error: Connection refused: Connect".

Hi Eddie,
The only thing that needs to be running is the Camera Server, the code in Simple Vision is what it was intended to work with.

I actually haven't tried running with Intermediate Vision code, but I didn't change anything about the networking code, so I'm not sure what the issue might be. If you're running Intermediate Vision or otherwise post-processing the image before sending it to the drive station, there's no benefit to running my version of the applet, though, since the robot would already be fixing the image as described in my OP. If you're taking this approach, I would recommend the vanilla USB Webcam Viewer.

Best,

[eddie12390]

Originally Posted by RyanCahoon

Hi Eddie,
The only thing that needs to be running is the Camera Server, the code in Simple Vision is what it was intended to work with.

I actually haven't tried running with Intermediate Vision code, but I didn't change anything about the networking code, so I'm not sure what the issue might be. If you're running Intermediate Vision or otherwise post-processing the image before sending it to the drive station, there's no benefit to running my version of the applet, though, since the robot would already be fixing the image as described in my OP. If you're taking this approach, I would recommend the vanilla USB Webcam Viewer.

Best,

I wasn't using the Intermediate Vision code, I had left out starting the automatic capture on the CameraServer.

[iLloyd]

We are using the Microsoft LifeCam and your patched widget. Sometimes, the camera feed will appear and sometimes it won't. It's not very consistent, but the light is always on the camera. We are using the Simple Vision sample in our C++ code. It would be awesome if you could help us so the camera feed is consistent.

[RyanCahoon]

Originally Posted by iLloyd

Sometimes, the camera feed will appear and sometimes it won't.

When the image doesn't show, does a red box appear at the bottom of the widget with an error? Also, if my part of the code encountered an error, there should be files placed at C:\Users\<user name>\SmartDashboard\extensions\ named error.txt and frame.jpg. Can you upload them?

[ThomasClark]

Originally Posted by RyanCahoon

Thanks to @TheChewanater on Reddit for posting the original code to the USB Camera Viewer widget. The algorithm to fix the JPEG header was adapted from here.

"TheChewanater on Reddit" here. Do you mind if we adapt this for the official SmartDashboard distribution? We've been trying to solve this exact problem for a while now, and it looks like you've got it down.

[RyanCahoon]

Originally Posted by ThomasClark

Do you mind if we adapt this for the official SmartDashboard distribution?

Be my guest. That would be great.

[SndMndBdy]

What would have gone wrong if you weren't using this patch? We were using a Logitech USB Camera, and the the smart dashboard extension generally worked fine but occasionally didn't show anything at all. Might the problem with the missing Huffman Table have caused that symptom?

[eddie12390]

Originally Posted by SndMndBdy

What would have gone wrong if you weren't using this patch? We were using a Logitech USB Camera, and the the smart dashboard extension generally worked fine but occasionally didn't show anything at all. Might the problem with the missing Huffman Table have caused that symptom?

It would have never worked correctly and the widget in the SmartDashboard would have a red bar with a message stating that the frame was missing a Huffman Table.

[Joe Ross]

I found that the resolution property was being saved to the xml file, but on startup, the camera was always set to 640x480. My patch for that is here: https://usfirst.collab.net/gerrit/#/c/844/

[RyanCahoon]

Originally Posted by Joe Ross

I found that the resolution property was being saved to the xml file, but on startup, the camera was always set to 640x480. My patch for that is here: https://usfirst.collab.net/gerrit/#/c/844/

Thanks, Joe