Odd scenario for us, though I’ve seen similar oddities in the forum.
Last Saturday everything worked fine on the robot, all VictorSPXs and TalonSRXs working, no warning lights anywhere.
This Friday’s meeting we turned on the robot and EVERY Victor and Talon was doing the left/right slow blink red. PCM status light slow blinking… PDP status lights slow blinking… every indication that the CAN bus was kaput. I had the few students at the meeting work on finishing mechanical stuff and I dug into some researching, which turned up no identical faults.
Trying to connect to the robot using Phoenix Tuner fails.
Today the students went through ALL wires, checking connections at the PDP, RoboRio, PCM, the CAN splitters we use (mindsensors) and any ferrule crimps. No issues.
We even swapped to a different RoboRio and still had the same issue.
Ran out of time at today’s meeting before we could try swapping the PDP…
Anyone have ideas were else we should be looking for an issue?
We also have a similar randomly occuring issue where we see in the DS log that the CAN frames are stale and none of our TalonFXs or TalonSRXs work, and it’s also resolved as we reboot the bot. We’ve checked wiring and it all looks good.
We have had some weird experiences where one bad talon/victor will take down the whole can loop, not just things after it. The best way to troubleshoot this is to start breaking you loop into smaller parts and test those individually. We cut the loop in half and find the broken half, then split that. Should be able to isolate the problem rather quickly.
If your CAN bus is separable via some sort of plugs or other connections, make a terminator using a 120 ohm resistor to keep the buss loaded properly (in practice most anything from 100-150 ohms works). If you’re using the PWM-style connectors, you might want to make a male and a female pinned version for your bag of tricks.
That way you can break the bus at some point and still have it properly terminated so that isn’t causing another problem.
Since the CAN bus is a parallel bus, there really isn’t a notion “after” something. The yellow and green wires aren’t “input” or “output” - they’re actually connected together inside of a device like a Talon for example. So if a device is misbehaving and pulling the bus signal one or or the other, it’s bad for everyone because everyone sees that.
Breaking the loop is absolutely the way to do it. Since the Rio is the master, working your way out from that is the way to go.
If there is a break in the wire inside the insulation that cannot be seen from the outside or a bad solder joint on the circuit board, there will be no signal past that point. Most of the devices used in FRC with integral wires do not have strain reliefs where the wires come through the enclosure. With a lot of flexing, the conductors can break at this point. It is easiest to detect by gently pulling on the wire and seeing if it stretches (it should not) and if the diameter gets smaller at some point (where the conductor is broken).
I worked with one of our techs to fix a differential probe last week and found that the problem was the wire was broken at the input plug.
Absolutely correct - if there is a break, there definitely is an “after” the break!
Having seen a 6ga cable with an internal break on our robot a couple of seasons ago, it’s a thing for sure.
Within our team, people have the mental picture of “in” and “out” of a device and I have a damaged controller on the bench to show it’s just a continuous length or wire (or it should be ) - that kind of drove my not that clear response. If you have a device or wiring problem that pulls the bus one way or the other, everyone gets hit until that device gets isolated.
If you’ve got CAN trouble, in my experience look at connectors first; I’ll pull the bus apart some and stick in a terminator but the approach of a binary search is the same no matter what the fault.
But @philso point is super on target, it’s not that hard, especially for a team that’s been around a while and has older controllers, to get damage right at the point where the wires enter the case.
I try to get all the controllers tested each season and tagged as such but that’s no guarantee - sometimes non-working ones can get fixed but most of the time it’s just to get them out of the eco system.
We use some mounts that can be screwed or double sided taped into place (what we usually do) which have a slot for 2 tywraps so you can change a controller without messing with its mount - that can help if you have an actual malfunctioning device if you need to swap for a replacement.
Single and Dual for SparkMax and TalonSRX - it’s probably not an original idea - I think I saw something similar…
One non-wiring thing that also helps, is do all your configuration in your code - if you do anything non-standard to the controller for whatever purposes it serves, configure it via the code - each and every time. This means you can replace that controller pretty quickly with nothing but a set CAN ID.
My original reply wasn’t as clear as it could have been - that’s the difference between folks sitting down talking and shooting a quick response out into the dark ether…
Found the failure. One (1!) of the many motor controllers had an extension wire that was NOT yellow and green. Someone disconnected that one extension and reconnected it in reverse.
Whole system went down.
