We have been having intermittent problems at competition where our Raspberry Pi vision co-processor stops updating Network Tables, rendering our semi-autonomous “go place this game piece” command not work. Here is our setup and the symptoms:
Raspberry Pi 3 running the latest version of the FRCVision library
Two USB cameras plugged into the rPi, one with ring lights and one just for driver assist
rPi is connected to the radio via ethernet, along with the RoboRIO
rPi is running two simultaneous vision threads, which do some manipulation of the OpenCV mats (render contours, draw crosshairs, etc.), broadcast the two camera streams, and update some network table values to send the center of the vision targets and an estimated distance to the Rio.
Both the streams and NT values seem to work fairly reliably in testing, when tethered, etc. Our lead programmer seems to remember a few times that the NT froze when testing in our build space, but we were not able to reproduce this tethered after our last competition.
During several (more than half) of our competition matches at our last Regional, we would no longer see the NT values coming from the pi updated on the driver station, and the Rio’s code would act as if it was not getting data.
When this happens, we still continue to see the camera streams from the rPi, so we know it’s still on, connected to the network, and running the vision threads. Only the Network Tables values are frozen.
We know the NT values are actually frozen and not just a glitch in the processing pipeline because we have a simple iteration counter in the NT that we update every pass through the vision pipeline. This happily counts up while things are working (and on the practice field, in the pit, etc.) but freezes when the problem occurs.
Does anyone have any idea what could cause the rPi to lose the ability to refresh the Network Table server on the rio?
I was able to reproduce the problem today on a practice robot. It seems to always happen shortly after powering everything on at the main breaker, like the Pi is trying to connect to the NT server before it exists then freezes once it does.
That sounds very unusual. The underlying ntcore library is used in a lot of places including dashboards, so the reconnect implementation is tested a lot. It’s clear that at least the initial handshake is working, as you’re getting the stream values? Can you connect to the FRCVision web dashboard and take a look at the console?
Yes, we get the stream values and the values we are writing to the NT seemingly at the time everything connects, but it is frozen. The stream continues fine so we know the pi is still running and the NT update and vision code is consecutive on the Pi so the NT values should be updating. If we reboot the Pi from the dashboard it seems to then run indefinitely without issue.
We can check the pi console and try swapping out the radio at our next meeting. Thanks again!
Is the stream provided by the image processing, or is it unprocessed frames provided directly by CameraServer? I’m wondering if your vision code is hanging up, but CameraServer is still running to provide the unprocessed frames. You could double-check this by having your vision code also print to console, and then viewing the console on the web dashboard. Can you provide a link to your code?
With similar setup, we had the same issue of NetworkTable values frozen but stream working fine only with competition router. It was resolved by setting static ip of the RaspberryPi on the webdashboard.
It is definitely sending processed images (contours, etc.) to the driver station, so we know the image pipeline is running. The code is just a modified version of the Java multicameraserver example with our vision thread added on. The code to establish the NT client is straight from the sample.
Thanks for the suggestion. We tried setting the pi to a static IP at competition but then couldn’t get the driver station to connect via frcvision.local. We were smart enough to leave the competition radio out of the bag after out last competition so we can fiddle with it more before Detroit.
It should automatically try all URLs published by CameraServer, at least one of which should be the IP. But if it’s not working, you could try one of the Java dashboards (SmartDashboard or Shuffleboard) or a web browser to view the stream instead.
We were having trouble with what I thought was noise or voltage dips occasionally causing the PI to stop steaming video. We’d have to complete reboot the PI and the robot to reconnect. Instead of powering the PI from a regulator connected to the PDB, we added a battery pack (phone charger) to power the PI. This worked great and we’ve not seen the PI disconnect in this way again.
However, we did start seeing another problem. If the PI was booted significantly after the boot of the roborio and radio, the camera stream would never connect to the dashboard. We were not able to ping the PI. It did not seem to communicate through the radio. It did communicate to the roborio though because we could see the network tables being updated with the vision target data.
If the PI was booted significantly before the boot of the roborio and radio, the camera stream would connect to the dashboard but the network tables would never connect and update the vision target data. The vision processing was working and finding targets because our annotation drawing on the mjpeg stream showed the location of the target.
Our solution was to boot the roborio, radio, and PI at the same time every time we start. E.g. we unplug the USB cable from the battery pack to the PI and plug it in and then immediately power up the bot. This worked for us every time in our last competition without any disconnects or loss of mjpeg stream.