Recently we’ve been having issues with Photon Vision on a Raspberry Pi not being able to connect to networktables. I haven’t found any consistence with the issue yet. Sometimes a reboot will fix it, sometimes not. If it’s working and we restart our code on the roborio it will sometimes not connect. Once I had the feeds streaming to shuffle board just fine, but then the driver station wasn’t seeing the robot, so I closed it out and reopened it. As soon as the driver station connected to the robot, I lost connection to the Pi. I can still ssh into the pi from the driver station laptop and then from there ssh into the roborio.
I’ve also tried swapping out hardware(Radio, Switch, Cables, and Robot). No change.
Originally I had the pi set to a static ip via Photon at (10.31.60.3). I tried changing the ip to (10.31.60.57), which I believe is the default for Photon. It seems to be better but still not consistent. Another thing I did was to enable “Wait for Network at Boot” on the pi.
I’m at a loss right now and have one day till our first competition. Does anyone have any other ideas?
Here are some of my logs:
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Searching for rios
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Trying 127.0.0.1
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Didn't match RIO IP
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Trying 10.31.60.3
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Maybe found 10.31.60.3
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Looking up 10.31.60.2
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Pinging 10.31.60.2
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Was able to connect to 10.31.60.2
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Trying 169.254.20.176
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Didn't match RIO IP
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.local
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.lan
[2021-01-11 02:24:03] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.frc-field.local
[2021-01-11 02:24:23] [WebServer - SocketHandler] [INFO] New websocket connection from /10.31.60.183:52667
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Searching for rios
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Trying 127.0.0.1
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Didn't match RIO IP
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Trying 10.31.60.3
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Maybe found 10.31.60.3
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Looking up 10.31.60.2
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Pinging 10.31.60.2
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Was able to connect to 10.31.60.2
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Trying 169.254.20.176
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Didn't match RIO IP
[2021-01-11 02:24:24] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.local
[2021-01-11 02:24:24] [WebServer - Server] [DEBUG] Handled HTTP GET request from 10.31.60.183 in 48.0788ms
[2021-01-11 02:24:29] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.lan
[2021-01-11 02:24:29] [General - NetworkTablesManager] [INFO] Looking up roboRIO-3160-FRC.frc-field.local
[2021-01-11 02:24:38] [General - NetworkTablesManager] [ERROR] NT Connection has failed! Will retry in background.
The fact that opening driver station caused it to lose connection makes me think that maybe it’s not a photon issue, more an issue the the roborio networktables server.
In the logs when it’s trying different ip addresses, I noticed that at one point it tries its own address. Is that what it’s supposed to do?
I had this issue too. I talked drivers about just restarting the pi while setting up. Maybe even turning the robot on then powering the pi. And then checking in the shuffleboard camera server to make sure it appears.
I’ll see you all in Oklahoma. So if we have more answers we can share there.
Power cycling the the raspberry pi at least worked once for us, and was easier. I haven’t tried just restarting photonvision from the dashboard to see if that is an option (would be best case – might be able to make it fix itself mid-match for example).
We’ve heard some reports of people seeing improved results after setting the roborio to static IP rather than DHCP (which is weird, that shouldn’t affect this, but worth a try?) Also, v2022.1.6 makes the roborio finder only run when you click into the settings tab, which might also help? We’re still trying to find a definite root-cause.
Our roborio has been set to a static ip. I’ll check out the new version.
We had a competition this last weekend. Before we left, everything robot-wise was working except for the issue of photon not connecting to networktables consistently. Once we got there we were having a heck of a time getting our driverstation to connect to networktables in the pit via ethernet connection to a switch. Photon seemed to be connecting more reliably but started to take awhile to come up. Long story short, we ended up configuring a new radio and putting it on the bot. After that everything connected consistently and in a few seconds. Didn’t have any problems for the rest of competition.
I haven’t been able to test the connections since we got home, but will test them soon and let you know.
Thanks for all the work and support you guys have put in to Photon Vision. It’s really coming along. We appreciate it!
We’re noticing that during testing we often will have photonvision not connecting to network tables.
The reproduction steps are essentially, have photon vision running and working. Re-deploy code N-number of times (typically it was happening within about 10-25 deploys), then see the robot code fail with the error can’t find the camera tables.
Restarting photon vision in the UI seems to fix the issue fairly consistently.
Is the robot code putting out an error from NT about not being able to bind the server address? That means the NT server isn’t starting on the robot code side and thus PhotonVision has nothing to connect to (nor dashboards). The robot code templates are designed to not start the NT server until the previous robot program has been killed, but potentially there’s something starting the NT server early or something keeping the previous robot program from getting killed in a timely manner. Note that rebooting the Rio will never have this issue, it can only happen on deploy.
We saw a potential improvement by disabling the roborio finder, specifically on Gloworm devices. Whatever’s going on isn’t consistent either, because 5940 saw no issues with 2022.1.6 and LL2+ at MBR this weekend. You’re welcome to give this JAR a try though:
Just an update on how our system has been running. For the last few weeks we haven’t had any connection issues with photonvision. The feeds have actually been coming up faster than the driverstation.
For anyone who it may help, here is our current setup.
Hardware Setup:
Raspberry Pi 3b+
PiCam v1.0
Microsoft Lifcam 3000 HD
DC to DC converter(Outputs 5v with an input of 8-14v)
12v fan for cooling(A Pi will slow down the running processes until it gets to an aceptable operating temperature if overheated. )
First port on radio runs to network switch
Roborio, Raspberry Pi, and Driverstaion(during competition) are all connected to the switch
Software Setup:
Photonvision v2022.1.6
Wait For Network at Boot is enabled via raspi-config
Static IP set for 10.te.am.13
Looking back I would conclude most of our connection issues were radio related. Using a newly imaged radio at our first competition helped a lot. Since then we’ve been running everything off of the first radio port without issues.
We were having similar issues at our last regional, but for some reason it was happening with both Photon and LL. Rebooting the rio after boot from DS solved it pretty well, wish we had a more permanent fix. Most FTA’s just want to get matches going as fast as possible.