We’re having an intermittent radio to RoboRIO drop out during impacts in which the robot losses connection for ~ 5seconds. For troubleshooting we’ve replaced the Ethernet cable, improved strain relief, and added hot glue. This has improved the frequency and level of impact causing the drop, but not eliminated it.
I’ve not experienced this exact failure before and it seems unusual for an Ethernet connection to be impact / vibration sensitive. Inspecting other robots I don’t seem the same level of strain relief we’re implementing.
Any thoughts, similar experiences, and solutions that worked?
We don’t believe it’s power related as recovery from those failures requires a much longer recovery time.
Try wiggling each Ethernet connection to figure out which thing it is when powered on, if you can do that we can probably give more troubleshooting steps.
Getting a CSA in-person would be best, but I can help you with some basic troubleshooting remotely while you wait.
Do all of your ethernet cables have locking pins? They often break off.
How long exactly does the comms drop last? Could you upload a Driver Station log that includes an occurrence of this?
Put a glob of hot glue on each ethernet plug on the opposite side the retainer clip is on. You should be able to rip off the glue with pliers if you need to unplug the cable to work on something. Make sure each cable is zip tied to something solid so the plug is not getting tugged on during robot motion.
Replace the Ethernet cable and POE injector with new ones.
You’re right in that it’s not power related, either a radio reboot or RIO reboot would take much longer. From the timing, it sounds like a contact is momentarily disconnecting during impact either at the cable to radio, cable to RIO, or cable to POE cable connection. I suspect the time delay you are seeing is the link renegotiating speed and TX/RX direction on reconnection.
I cannot stress enough how easy and crucial hot glue is to FRC connections. They effectively shock mount the cable. Not just ethernet cables, do all your wiring that goes into the RoboRio, ethernet plugs, pwm, CAN… anything that involves just pushing it in and no lever mechanism. If something needs to come out, it’s easy enough to pull off, but stays in if you get hit in a match.
The addition of redundant power with a barrel connector in addition to the PoE is that if the PoE loses connection for a millisecond, the radio doesn’t reboot entirely, it will come back within a few seconds rather than a minute. Hot glue that thing in as well.
I generally think pit scouting is not that useful (why ask how many cycles they can do, just watch matches to see what they actually do), but one of my major exceptions to that is seeing how a team has their radio connected. Do they have redundant power in addition to the PoE connector, and is it hot glued? If it isn’t, I suggest to them that they do it (and offer a hot glue gun), and if they don’t take the advice, I sit back and wait for them to disconnect mid-match, and then remind them that they can borrow our hot glue gun if they need it.
I’m glad it sorted you out, and I hope other teams see this thread too and start hot gluing their connectors. My kids refused to do it until they saw 973 did it on their robot…
Update; Hot glue masked the true issue as it came back after 1 high impact match. The team reviewed the Drivers Station logs (thanks CSA) and identified a radio connection error specifically calling out a RoboRIO fault. Replacing the RoboRIO has resolved the issue (three matches without a fault).
We are having a similar issue, can you tell us specifically what the fault is that found on the RoboRIO? Did you try re-imaging the RoboRIO to see if it would fix the issue? Thanks!
We had a spare RoboRio we went directly to swapping hardware and did not need to re-image. Additionally, there was evidence it was mechanical (Eg it occurred on impact) thus we focused our troubleshooting on physical hardware and connections.
Our learnings and guidance for you is:
Your on-site CSA or and another experienced team will be your most effective path to a fix.
Use your logs and show them to the event CSA to root cause.
Have video of your robot faulting to show the CSA as you discuss the failure.
Work to replicate the error in you pit. If you can do this it speeds up root cause identification dramatically.
We’ve also found the symptoms of short (5-10 sec) loss off comms can be due to a lot of things!
Poor Ethernet connection in robot wiring
Damaged RoboRio connection.
Robot power brown outs
Using too much bandwidth resulting in the field stopping comms
Faulty Drivers computer Ethernet connection to the field.
In a case where the roborio has a damaged Ethernet port, a USB-Ethernet adapter can also serve as a simple replacement, if you have a USB port to spare
Thanks! That is helpful and we have checked most of those. We were having some intermittent brown-outs primarily due to not using a fully charged battery. We think/hope the primary cause of our issue was a damaged breaker going to our ethernet switch that temporarily disconnected the ethernet switch during hard impacts.
We appreciate the list of other things to try and backup options.
Last year at Bayou Regional, we had broken solder connections on a factory motherboard of a ethernet switch. The hard hits were enough to temporarily break the electrical connection for a few ms. Didn’t find the problem until after the competition when we opened up the radio and switch.
