Falcons and Talons giving random config errors in begin.vi

Labview team running 5 Falcon FX motors and several Talons and Victors on the robot.
When running RobotMain.vi and even on deployed code we randomly get configuration errors.
Example of what I’m seeing is in the attached picture.
Everything is the latest. Latest firmwares, latest LabView, CAN IDs are not in conflict and so on.
Anyone seeing this as well?
image

Also this only happens when Begin.vi is run.

The above screen snip indicates that your Talon FX at CAN ID #25 didn’t respond to the configuration command.

I’d suggest seeing if that Talon FX is visible on the CAN bus using Phoenix Tuner. If that Talon FX is not visible in Phoenix Tuner, check the power to the appropriate Talon FX, check the CAN bus wiring to it, and confirm that there aren’t multiple Talon devices with that same CAN ID.

if the problem is intermittent (i.e. it works sometimes, but not always), look for a partially connected or loose connection on a CAN bus cable.

It is visible in the tuner. There are no conflicting id’s, and CAN wire was just all gone through again to eliminate issues. All connections were soldered and shrink wrapped by a qualified electronics engineer to be sure it wasn’t a connection issue.

This is an intermittent issue for sure. Happens 3 or 4 out of 10 times when running code from the laptop and maybe 5% of the time when code is deployed…

I’m not an expert on this subject, but I’m wondering if any of the following might help. (Not necessarily in any order…)

  1. Is the temporary phoenix tuner server still running when executing the robot code. (If so, suggest rebooting the roborio.). I wonder if phoenix tuner and the robot code has issues with each other.
  2. Is the firmware updated on the all the CAN devices.
  3. What does the CAN statistics look like on the driver station? I think all the counters other than % usage value should be zero.
  4. The CAN bus has the correct termination resistor at each end. (Built in resistors on roborio and PDB. The PDB has to be enabled with a jumper.)
  5. All the CAN Id’s are unique and all devices are visible when running phoenix tuner.
  6. Lights on each device appear to be appropriate.
  7. You can “blink” each device.
  1. If there is a timeout for the call, is it set too fast (or slow).

#1 No
#2 Yes
#3 ~70% utilization, all other counts are 0
We do have “Comm Errors” but I can’t find any documentation on what those are/mean
#4 - Hardware team on 3572 will need to answer, I can only assume yes since they are very competent.
#5 Yes
#6 Yes
#7 Yes

Don’t think the CTRE configall.vi has a timeout but I’ll check.

Number 4 is a solid yes as well

From the above and other responses you’ve posted, it sounds like you’ve been doing all the right things to address possible root causes.

Intermittents are always hard to track down. When the error message appears, is it always for the Talon FX at CAN ID #25?

Finding out what the timeout is set to is also a good idea. I seem to recall that CTRE typically uses 1000msec in their examples.

It’s happening on only about 5 CAN IDs of our Falcon/Talons right now.
So one time I might get an error on CAN 25 like above but the next time it might be CAN 6 then the next time CAN 7. We have about 10 CTRE speed controllers on the robot but I’m only seeing it on the few. And only one error at a time. We never get two errors on two different CAN IDs at one time.

I open the controller in Begin.vi then the next vi is CTRE’s “ConfigAll.vi” this error must be in response to that “ConfigAll”.
In the above the error talks about not setting ConfigClosedLoopRamp but the next time the error occurs it might be “couldn’t configure kI” or “couldn’t configure kP” or about any other setting on “ConfigAll”.

Have you tried reading the error in begin.vi and retrying if there is an error?

The retry thing above sounds like a good suggestion.

The suggestions here are a little more odd, and less likely to be happening, but maybe will help bring forth other ideas…

  1. Can you easily organize the open and configuration calls so that they happen sequentially, then see if the error happens in the same place all the time? I doubt this would correct anything, but it might help to indicate if one particular device is the issue, or indicate it isn’t related to a single device.

  2. Does 70% CAN bus utilization sound high? One of the teams I help has 11 talons/victors and a PCM, PDB with a utilization rate of about 50%. (A couple of those devices are followers.)

  3. After BEGIN finishes are you trying to continually re-write configuration, or PID parameters? I don’t know if this would be an issue, but it might contribute to high utilization rate.

  4. Any chance there is induced signals on the CAN bus. Are the CAN bus wires sufficiently twisted? Is there a little segregation between CAN wires and motor controller output wiring?

  5. Any chance one of the devices on the CAN bus is bad… (Hard to find. Probably have to remove one at a time… or pull the breaker.)

Anyway, hope you find the issue.

Thanks, we will check through those items once we regain traction from having our season shut down. Once we know what the future holds we will know whether to chase our gremlins or simply plan for next year. Luckily we have a lot of time on our hands to debug if we choose to do so.