In our last competition (Israel International Offseason),
we had a problem where our bot disconnected for a second or two and then reconnect, repeatedly for a short time.
This happened for about half of our games.
It is not happening in our practice field at home.
After the competition at home we reconfigured our radio and tried to play with the bot, for hours, including bumping into things, and a “full match”.
We looked into our bandwidth and it is looking good around 8000-12000 bytes per second which the maximum is about 524288 bytes/sec (4 Megabits/sec).
Our ethernet cable management is pretty straightforward,
from the Rio we have one ethernet cable to a Brainbox Switch (SW-005), from the switch we have four cables running out:
FRC stock Router
raspberry pi 3 (we use it for a color sensor as I2C is crushing the kernal :/)
We are running out of ideas for the problem as all of our tests are coming back negative.
If someone here has encountered this problem and can give us some insights into the solution we would be grateful for the help.
It’s hard to tell from screenshots, but it looks like you’re remaining in contact with the radio, but losing contact with the roboRIO. I think your disconnection times are too short to be a roboRIO reboot. Possibly you are losing power to your switch. How is it powered?
I notice your battery voltage is dipping. Is it possible that your bad games were all with the same battery? You might want to check that your battery terminals and wiring are solid. Also, next time you have a bad match, check for hotspots in the power wiring, from the battery terminals to the switch.
Oh, and probably not your problem, but I see loop overruns in robotPeriodic. You should check on that. What is it doing?
as for the switch, it is the brainbox sw-005, it is wired directly to the PDH.
it can handle a range of input power: +5VDC to +30VDC.
in addition most of the drops happen after the connection is restored, just as the bot start moving again…
it happened with many different batteries,
but we will check…
About the robotPeriodic, we are running a loop overrun that controls the autonoumes (the driving part), so it will not affect the teleop part, we close it after the autonoumes. Also we had the same part of the code in 2019, 2020 and we didn’t disconnect like this.
The packet loss in that log is through the roof- consistently 50%, with disconnection events coinciding with spikes of 80-100% loss. Additionally, latency is 40ms (two full DS packet cycles) throughout the log.
Were other robots having issues? Did you talk to the FTA at the event about the issues? Does this happen at home when operating wirelessly?
The high packet loss happened to other teams though the disconnection issues seemed to be mostly to us.
At our practice field, the robot is working just fine! Not even one disconnection over several hours of running.
Not sure why the switch would only brown out at matches but maybe you are digging deeper into your battery supply and a few may have issues. Another thing to check - often teams only use laptop ethernet during matches. Try that connection at home and see if makes a difference.
When my team acquired this brainbox wired this way AND used static IP addressing for all devices our mysterious disconnects at competitions dropped essentially to zero. But I don’t remember high packet loss being associated with our disconnects as they might be in your case.
To verify: all your Ethernet devices go to the Brainbox and the radio’s cable from the Brainbox is in radio port 18-24v POE. The radio 802.3af POE port is empty.
We’ve been powering the radio with POE 12 v and last season 18v POE and that fixed radio power disconnects. (18v POE into the radio only and not routed through switches, usually.)
We also participated in ISO this year,
The whole network worked in our workshop but in the competition and only during the games we experienced network problems, we analyzed it a bit and didn’t come to too many conclusions but in one of the games before the game started we couldn’t connect to the radio and I checked it five minutes before that in the pit and the network worked so we came to the conclusion that the problem is with the ethernet cable On the driver station and after we changed the cable we were able to connect to the robot, I can’t be sure that the problem is the same for you but it is probably a problem with the wired network that the competition provided.
The only camera we have on the robot is the Limelight which also transmits video to the dashboard,
I don’t think it is hitting the FMS bandwidth limits although I cannot confirm it.
As you said the problem seems to be roboRio unavailable can it be caused by hitting the bandwidth limit?
Unfortunately, the shop is closed right now so I can’t send you any photos, but I’ll make sure one of my colleagues sends a photo tomorrow morning.
Thank you very much
How much data are you sending to SmartDashboard and NetworkTables? If you’re sending a lot of data you could reach bandwidth limits. Optimize your code to only update NetworkTable values when they need to be updated.
Also, strip out stuff that isn’t really necessary to be seen during the match and move it to a log file so you can still see it later.