I’m posting this here and contacting CTRE separately, but I wanted to ask if anyone has ever had similar issues before especially now that CAN devices are becoming more standard.
Over the weekend we had two separate incidents of completely “random” CAN device failures. We had a Falcon 500 motor and a CANCoder on entirely separate occasions completely knock out an entire CAN bus.
Our robot has 2 CAN Bus device chains no bigger than 12 devices each. All CAN devices and power lines are securely fastened together to prevent disconnections. We use anderson power-poles for both power connections and CAN connections. We’ve been doing this on multiple robots going back to 2019 with little to no problems.
In instance one we had a falcon motor that would show on the CTRE Phoenix tuner via the roboRIO but when I checked the CAN bus from the end of the chain at the REV PDP using their software client I couldn’t see the motor in their CAN comms screen… I found that odd so I asked the students to unhook that one motor from the chain and lo and behold the entire bus worked again. In fact something I found odd was in the REV CAN diagnostics screen the BUS was showing multiple copies of each CAN device for some reason, but then once the affected motor was removed from the chain the screen refreshed and only showed one of each device in the CAN network. The motor was barely used at all at that point so there is no way it was damaged from use. We replaced the motor with a spare and all was well.
But wait… there’s more! 2 matches later the CAN Bus dies again and this time using the Phoenix tuner the CANivore and by moving a terminal resistor around I find that the CANCoder has somehow died… again no abuse to speak of just failed without a known cause.
I’m pretty frustrated that some of the best performing motors we can use in FRC have some of the worst electrical troubleshooting. To my knowledge my only option was to unplug and try each motor one-two at a time with a terminating resistor.
Sidenote I can’t for the life of me understand why a CAN Bus wired series chain is the recommended BUS layout. I hated fixing christmas lights as a kid that were wired in series that got damaged in storage… why is it that our critical controls backbone of our robot has to be wired that way as well. This experience has me really wanting to learn about how to do STAR topology even if it isn’t recommended just due to the better fault tolerance.
Just added a CANivore, seeing a similar problem where we get the CTR:CAN Frame Not received/too-stale message we used to get due to over-utilization at seemingly random times. Was really hoping CANivore was the solution, but this has thrown us for a loop… Wiring seems good, drivetrain will drive sometimes, but then this behavior also happens.
Edit: Drive will not drive properly, modules only turn some times. Some other info: DriverStation has version infos for TalonFx’s as “inconsistent”, whereas version info for Pigeon2 and CANCoders were correct.
Edit2: More info: I threw in a function that checks for power failures using ctre’s hasResetOccurred() and checked to see if a power fail matched up with the stale status frame. Nope, it doesn’t.
Yep. Thinking our next step may be checking for a short within one of the motors as outlined in this post: CTRE CANivore CAN Bus not working - #10 by bruingineer? But I don’t know how that would lead to the observed behavior…
i would first check the solder joints on the CANcoder, one might be cold soldered and noisey. I like Anderson Connectors but would not recommend them for CAN. The CAN signal is such that it might not breakdown the oxides on the silver plated connectors and produce noise and or intermittent connections. I much prefer the Molex latching connectors from Digikey. (Search for my previous posts with the Digikey part numbers) They use the same pins as PWM so most teams have the pins and a crimper.
First, make sure the CANivore’s internal bus termination is enabled in Phoenix Tuner, and double check the other end of the bus is properly terminated (NOT with the PDP, as it is CAN 2.0).
From there, you can do the first set of steps in that post, as that should rule out issues with out-of-date firmware, duplicate IDs, bad wiring, etc. If you encounter a problematic device (CAN blips red), connect it to the CANivore by itself (with a terminating resistor) and check if the CAN issue still reproduces. If it does, then I would contact CTRE Support.
I’ll also add on that if the CAN LED on the CANivore is SOLID RED (not strobing/blinking), then the CANivore may not be receiving enough voltage for CAN to operate properly. You can confirm this in Tuner in the 5V Supply column – any value ~4.6 V or less is bad. If that is the case, try connecting V+ and V- to a 12V slot on the VRM – or disconnecting it if it’s currently connected – and see if the issue goes away.
If you still can’t figure out the issue, I would contact CTRE Support, as they may have other troubleshooting steps not mentioned here.
Edit: somehow got the V+/V- input voltage wrong, should be 12 V
Some more info from tests today:
The chassis I had been working on was taken for driver practice, minus the CANivore, worked fine. Started work on a bare, clean, untouched chassis, just wired, and got it set up with the same CANivore-using chassis code I had used on the old chassis. At first it seemed the problem had gone away, however, I still occasionally (at a much less frequent rate) see the problem where I get a CTR: CAN frame not received/too-stale error for some of the CANcoders. This does not cause the code to stop working, and happens randomly as far as I can tell. Although I am encouraged by the lower frequency of errors and that the code still works, I am concerned that the issue may not be entirely electrical, or if it is, then there is some electrical problem we missed.
Edit: I am holding out on doing the full-blown electrical test so that driver team gets practice, and I assumed today’s test would prove conclusively if it was a software or hardware problem. That was not the case, as the problem improved but was not completely solved. I may go through the full process tomorrow, unless I find a software fix/ clear electrical answer that I am missing.