Can-Bus Reboot?


#1

At the MARC competition, we had a very interesting issue. It started out as a momentary “blip” in the can bus during a match. We lost all control of the robot momentarily and the canbus usage as reported by driver station logs dropped to zero.

We could not reproduce the error after the match. A couple matches later, it happened again. This time the ‘blip’ was longer. But again, we couldn’t reproduce it. However, it appeared to happen when we ran our hilo motors (two talon srx’s driving 2 775s with a versa encoder slice in the mix).

The third time I was field-side when it happened, and it was much longer. I got to the robot and noticed that all the speed controller lights were blinking red. This lasted 30-45 seconds.

Back in the pits, we were able to finally reproduce the issue. We noticed a mechanical bind in the hi-lo that appeared to cause the current on the 775’s to spike. When that happened, the can bus would drop. We had already checked all our can wiring, and re-terminated our resistor at the end. We even tried utilizing the resistor in the PD board as well with no change in behavior.

We unplugged the breaker from one of the two hilo 775’s and thought we had the problem solved, as the hilo went up and down (slowly) without killing the bus. Back on the field though, the first time we lifted the hilo, the robot died. It never regained can bus. In the pits afterwards, we were able to duplicate this. Solid flashing red. Unfortunately the last hilo 775 was buried under the hilo itself and could not be switched out quickly, and it was also the one that was connected to the hilo encoder. So it was game over.

Has anyone seen this before, where a single Talon SRX is capable of somehow taking down the can network when it runs?


#2

Even if you shorted the CAN bus, it would only cause a few CAN errors before restarting the bus- unless it was permanently internally shorted. I’d send the bad Talon to CTRE ASAP so they can take a look at it. Most of their stuff relies on a good CAN bus so I’m sure they’re be interested.


#3

Haven’t ever seen that, but it’s friday night so I’ll flail about in the dark for the fun of it.

Have you tried swapping the SRX’s to different units to see if one happens to be defective in some way? Seems pretty remote as a possibility, but would be interesting to see if you managed to create a faulty unit somehow…

Another improbable explanation is maybe the high change in current induces weird voltage spikes if you happened to run the CAN bus wires in parallel with the power supply for the mechanism?


#4

OK, at the risk of asking a dumb question: What’s a hilo?


#5

Not dumb at all, just different Terminology.

Hilo is an elevator. We call all the fork lifts at work hilos, short for high low. No idea how wide spread out is.


#6

Thanks, that’s what I inferred from context, but with such a strange issue, it seemed better to ask than assume.

On the “bad device” hypothesis, swapping out the devices one at a time sounds like the only way to check. Other things equal, I’d start with the SRXs and move to the PDP and finally RIO because this is the order in which current draw is most likely to affect device operation.

On the “current/CAN crosstalk” hypothesis: Are your CAN wires twisted? This is the primary means of reducing the area of your antenna loop, which reduces crosstalk both in and out. Do your CAN wires run a long distance alongside the power wires and/or motor wires (or worse, wrap around them)? If so, you may want to try re-routing the CAN farther away from the power lines. Twisting the motor and power wires as well to cut radiation can’t hurt, but isn’t as likely to have much effect because the SRX chop frequency is about ten octaves below the CAN bus frequency.


#7

Our can wires are twisted and run perpendicular to our power wires for the most part.


#8

We found that if there is a error in the talon can cause a break down. Is all your can run in a loop or is it independent from each other because if your talon is busted it can cause issue in the loop and completely kill the robot so maybe try replacing the talon and if it keeps happening you can check for a short in the system we had that problem while working on our robot that a connector poked through were we plug the can in to make the loop. If none of that works then I will try to ask a mentor about it. Hopefully this helps you guys out.


#9

We have checked through the wiring as best we could a number of times. Unfortunately due to some revisions during the season a portion of our electrical board became utterly inaccessible. That portion requires the removal of the entire electrical board, or our entire hi-lo. Due to Murphy’s Law, that’s exactly where the offending talon is located.

CAN bus has definitely been a learning experience this year. From the bus utilization to how the Roborio CPU usage affects it, it’s not as “it just works” as it might seem. But I like to think that we’ve found all the ways we can mess it up.