What is the deal here. Is there really that many teams with robo rio issues…
41 Likes
It feels like a joke watching playoffs of the best 256 teams in the world when every third match, at least one of the robots stops for the entirety or a significant portion of the match. I don’t want to believe it’s field issues but with the frequency of robots going out it’s hard to maintain that.
30 Likes
CAN needs to go. I’d be willing to bet that’s the culprit in so many cases. We had broken CAN wires in 3 seperate places after being totally fine leaving our last match yesterday. It’s an awful system and I think we can do better.
15 Likes
I’m especially surprised by, I think, 2 robots on Archimedes that went limp while the RSL is still flashing. Don’t expect something like a whole-bus CAN bus failure that often. Maybe a CANivore coming unplugged? Weird.
5 Likes
Do you have a better system in mind?
19 Likes
I think this season the crossing over the charge station has been especially rough on the electrical connections and devices.
12 Likes
We were one of those dead robots. Our last two matches had radio disconnects on the field. The first time, it rebooted on the field in about 40 seconds, then we got 3 more seconds of driving before another disconnect. These reboots came with significant packet loss across the alliance. We thought it may have been a loose wire on the VRM, so we stayed in for the last match.
In the last match, our auto started, then the radio disconnected again. This time it took roughly 100 seconds to reboot. We drove a little further then lost connection again.
Our alliance partner then lost a battery. About 30 seconds after they were powered off, we were able to move again.
The inspection the CSAs, FTA and the team that happened immediately after the match didn’t find anything that would cause those types of problems. The only hunch we have is a bad WAGO. Our radio was able to power on again immediately after the match, so we’re still a little confused about it.
I’d be curious to know what other first hand experiences are.
6 Likes
Redundant CAN to protect against a broken loop? Some way of integrating large portions of the control system? Bigger CAN wires with standard connectors? Surely this is a solved problem elsewhere.
2 Likes
This season has definitely been a turning point in robustness standards for competitive play. Brushless swerve in high speed collisions and shockingly rough impacts, combined with falls from the charge station and the longest horizontal extensions we’ve seen in decades getting banged around when acquiring game pieces, transporting them, and scoring them.
Moving forward, teams are going to need to change parts of their engineering philosophies to be prepared for the updated rigors of play.
Alongside this, our control system needs an entire overhaul for reliability, but that’s another deal.
37 Likes
Theres so many single points of failure on an FRC bot. With how hard some hits are, it’s not that surprising to have issues, especially this late in the season.
One thing thats not usually visible from a spectator standpoint is every time a robot is dead, both FTAs and CSAs really do try to get to a root cause. Theres never a case where a dead robot just gets ignored.
27 Likes
Maybe if the CANivore had 3 more busses for half the price, then you could only have partial loss if something happens. I can’t think of a better protocol to have so much real-time data with few wires. Maybe a good first step is less awful connectors and sleeved cabling
Edit: There are products like this https://www.traquair.com/store/can/repeaters/can-repeater-crep-s8c/ that could create a not-bad star topology.
3 Likes
At very least we need a legal $30-$50 CANivore (not necessarily with CAN FD).
I’d be interested to try to quantify the number of dead robots that are caused by different issues. I think the following are some of the main causes of robot death:
- Bad main power connection.
- Main breaker switched off due to contact with robot/field.
- CAN issues.
- Radio loses power (due to power connection or the spontaneous robot caused by the case).
- Bad RoboRIO power connection.
- Bad ethernet connection between RoboRIO and radio.
- Damage to individual control system components (for example, from swarf, contact, or defective components).
Both of our robot deaths this season were caused by a bad main power connection. We were also concerned about our Radio to RoboRIO ethernet connection since both ports are old and would intermittently disconnect with certain ethernet cables. Figuring out which of these issues are most frequent across all teams is the first step to fixing them. There are things that can be done about a lot of these.
- Main breaker without screw terminals. Better online documentation about best practices would probably also help.
- Better education about how to protect your main breaker. A breaker shield in the KOP would be nice. Also, instruction to inspectors to not tell teams to place their unprotected main breaker facing outwards.
- Cheaper USB-CAN interface to isolate issues. Consider switching from CAN.
- Different radio (as has been suggested to death).
- Remove screw terminals from RoboRIO (already possible for teams to do this I believe).
- Hard to fix systematically.
- Hard to fix systematically.
Not an definitive answer but this thread seems relevant.
6 Likes
Does anyone know of any of the teams that lost radio power had an Rev radio power module? I saw @CoreyBrown mention wires on the VRM. I don’t really like the VRM connectors they seem to fail more than Ethernet.
1 Like
I actually honestly don’t see how more can busses solve any issues, other then only making certain subsystems go down if a failure occurs. Like you couldn’t have a can bus per subsystem, but instead would probably split the robot into 2 or 3 distinct subsystems. Generally in FRC, loss of 1 subsystem, especially at high levels of play, really likely isn’t any better then the loss of the entire robot.
2 Likes
My dream product would be an enclosed navx-MXP with like 8 non-FD CAN busses, and maybe a few I2Cs for good measure.
1 Like
Imo we can stop pretending that the can bus is a serial bus. If it’s wired up in parallel correctly, most of the complaints that teams have disappear.
8 Likes
Except the terminating resistor is supposed to go on the longest branch (often on an arm outside your frame perimeter). I suppose the solution is to extend the connection to the PDP/PDH so that it’s the longest branch.
2 Likes
This is impossible.
Now, isolated CAN bus repeaters could start to make it possible…maybe?
EDIT: with less cheek, I will say that star CAN can help a lot but only if your busses are short. It can cause a lot of mysterious issues if it’s not handled well, and because it’s entirely empirical and out of spec, that varies from team to team and year to year.
4 Likes
That is legitimately one reason I had for not purchasing the CANivore: we already have enough stupid issues, I’d rather run slower and forget about signal integrity a bit more. Maybe with a bunch of busses running at like 250kbaud, you could use a star topology and have, maybe, a star point in each swerve module, with their own separate busses.
2 Likes
CAN is suitable and used for both steering and braking in automobiles. It is more than robust enough for FRC.
What you are describing “3 wires broke” is not a CAN problem. It’s a wiring problem. Out of curiosity, where did the breaks occur and did you work out why?
18 Likes