Hi,
In our last competition (Israel International Offseason),
we had a problem where our bot disconnected for a second or two and then reconnect, repeatedly for a short time.
This happened for about half of our games.
It is not happening in our practice field at home.
After the competition at home we reconfigured our radio and tried to play with the bot, for hours, including bumping into things, and a “full match”.
We looked into our bandwidth and it is looking good around 8000-12000 bytes per second which the maximum is about 524288 bytes/sec (4 Megabits/sec).
Our ethernet cable management is pretty straightforward,
from the Rio we have one ethernet cable to a Brainbox Switch (SW-005), from the switch we have four cables running out:
Rio (obviously)
FRC stock Router
raspberry pi 3 (we use it for a color sensor as I2C is crushing the kernal :/)
Limeligt 2+
We are running out of ideas for the problem as all of our tests are coming back negative.
If someone here has encountered this problem and can give us some insights into the solution we would be grateful for the help.
It’s hard to tell from screenshots, but it looks like you’re remaining in contact with the radio, but losing contact with the roboRIO. I think your disconnection times are too short to be a roboRIO reboot. Possibly you are losing power to your switch. How is it powered?
I notice your battery voltage is dipping. Is it possible that your bad games were all with the same battery? You might want to check that your battery terminals and wiring are solid. Also, next time you have a bad match, check for hotspots in the power wiring, from the battery terminals to the switch.
Oh, and probably not your problem, but I see loop overruns in robotPeriodic. You should check on that. What is it doing?
as for the switch, it is the brainbox sw-005, it is wired directly to the PDH.
it can handle a range of input power: +5VDC to +30VDC.
in addition most of the drops happen after the connection is restored, just as the bot start moving again…
it happened with many different batteries,
but we will check…
thanks
About the robotPeriodic, we are running a loop overrun that controls the autonoumes (the driving part), so it will not affect the teleop part, we close it after the autonoumes. Also we had the same part of the code in 2019, 2020 and we didn’t disconnect like this.
The packet loss in that log is through the roof- consistently 50%, with disconnection events coinciding with spikes of 80-100% loss. Additionally, latency is 40ms (two full DS packet cycles) throughout the log.
Were other robots having issues? Did you talk to the FTA at the event about the issues? Does this happen at home when operating wirelessly?
The high packet loss happened to other teams though the disconnection issues seemed to be mostly to us.
At our practice field, the robot is working just fine! Not even one disconnection over several hours of running.
Not sure why the switch would only brown out at matches but maybe you are digging deeper into your battery supply and a few may have issues. Another thing to check - often teams only use laptop ethernet during matches. Try that connection at home and see if makes a difference.
When my team acquired this brainbox wired this way AND used static IP addressing for all devices our mysterious disconnects at competitions dropped essentially to zero. But I don’t remember high packet loss being associated with our disconnects as they might be in your case.
To verify: all your Ethernet devices go to the Brainbox and the radio’s cable from the Brainbox is in radio port 18-24v POE. The radio 802.3af POE port is empty.
We’ve been powering the radio with POE 12 v and last season 18v POE and that fixed radio power disconnects. (18v POE into the radio only and not routed through switches, usually.)
During teleop, you had 19 disconnection events, none during auto. The disconnections seems to last about a second.
There are no “Time since robot boot” messages, which suggests that the roboRIO did not reboot.
All the ping results show “link-bad, DS radio(.4)-bad, robot radio(.1)-GOOD, roboRIO(.2)-bad, FMS-GOOD”. This suggests that the DriverStation can contact the robot radio, but not the roboRIO.
I also see “Info Communications Timeout: NN Input Voltage Brownouts: 0.”, with NN going up to 31.
Packet loss is about 45% throughout, peaking to 100% (or more!). The peaks are often synchronized with the disconnections.
Latency is steady at about 40ms.
Laptop CPU remains at about 15% CPU.
CAN utilization is steady at about 75%.
Battery voltage is around 10.5v, dipping to 8v at points.
I see a lot of messages throughout: CS: camera: Connecting to USB camera on /dev/video0
There’s a few messages about “HAL: CAN Receive has Timed Out”, which seems to be from Climb.getTilted().
Do you have cameras on the robot network transmitting to the Driver Station? Is it possible that they are hitting bandwidth limits under FMS?
Do you want to share photos of your Ethernet wiring and switch power?
We also participated in ISO this year,
The whole network worked in our workshop but in the competition and only during the games we experienced network problems, we analyzed it a bit and didn’t come to too many conclusions but in one of the games before the game started we couldn’t connect to the radio and I checked it five minutes before that in the pit and the network worked so we came to the conclusion that the problem is with the ethernet cable On the driver station and after we changed the cable we were able to connect to the robot, I can’t be sure that the problem is the same for you but it is probably a problem with the wired network that the competition provided.
I know you were able to connect to the radio and it’s probably not the same problem, but I would recommend double checking the cables in the driver station next time
The only camera we have on the robot is the Limelight which also transmits video to the dashboard,
I don’t think it is hitting the FMS bandwidth limits although I cannot confirm it.
As you said the problem seems to be roboRio unavailable can it be caused by hitting the bandwidth limit?
Unfortunately, the shop is closed right now so I can’t send you any photos, but I’ll make sure one of my colleagues sends a photo tomorrow morning.
Thank you very much
I wouldn’t have thought so, but robots continue to surprise me, and I don’t know if you were using an official FMS or one of the various off-season versions.
How much data are you sending to SmartDashboard and NetworkTables? If you’re sending a lot of data you could reach bandwidth limits. Optimize your code to only update NetworkTable values when they need to be updated.
Also, strip out stuff that isn’t really necessary to be seen during the match and move it to a log file so you can still see it later.
