SPARK MAX follower with lower CAN ID than leader causes ~4 stutters/sec until power-cycled?

Technical Control System
1

After several painful days of debugging, we believe we have isolated the cause of some strange behavior we’ve been seeing with our dual NEO direct-drive shooter. It appears that if a SPARK MAX follower has a lower CAN ID than it’s leader, they will enter a strange state that will cause them to pause for a split second ~4 times second. I speculate that the 220ms watchdog on the spark max is being triggered but don’t have any way to know. The most frustrating aspect of the problem is that just switching which CAN ID is the leader vs follower is not sufficient to fix it. A full power-cycle is required, with code using the correct order deployed before the power-cycle.

We have reproduced the problem with the SPARK MAX example follower code on one minimal test bed and also with our own code on our robot with an entirely separate set of parts.

For a complete replication, please see this video.

Is this a known issue?

EDIT: added link to code

1 Like
2

Definitely strange behavior. I don’t have the example code in front of me. Can you post it to github and share the link?

In the meantime, can you confirm whether there’s a restoreFactoryDefaults() for those CANSparkMax objects? I suspect there’s some different configuration for master vs follower.

1 Like
3

The example code is here, and restoreFactoryDefaults() is called for both controllers.

4

We had the same problem today running two neo 550s, and our circumstances were different, but suspiciously similar to yours:
First, the follower did NOT have a lower ID than the leader: the leader had CAN ID 13, and the follower had CAN ID 14 (which, suspiciously, was the same ID as the problematic follower in your video)
Second, we also had the follower invert with respect to the master.

Just like in your video, the follower would flash green, as it should, but regularly (we didn’t measure but 4/sec sounds right) will flash blue (we are using break mode) and fight the leader spark for a moment before starting to behave again.

Instead of fiddling with it and testing further as you have, we just stopped configuring the second one as a follower and manually set it to the same output in the code, at which point it started working perfectly.

Did you try other CAN ID numbers?

5

we had a similar issue. It turned out to be a flaky power connection to one of the Spark Max. After a power event the parameters would reload the defaults and the leader follower wouldn’t play nice together.

There was a post recently about updating your default parameters to overcome this issue, but fixing your power wiring is a better approach. This turned out to be a fairly handy debugging tool to help us identify the wiring issue.

6

@balink, although we have not tried different CAN IDs on the testbed shown in the video, we are able to reproduce the problem on our full robot (with its code) which has somewhat different CAN IDs. In that case, (leader=1, follower=14) works as expected, but (leader=14, follower=1) has the same stuttering behavior.

@JohnD, the fact that we’ve seen the issue on 2 entirely different setups makes a flaky power connection a less likely culprit. But perhaps we’re just incredibly unlucky.

I’d love to have outside confirmation or refutation of the proposed cause. Anyone else having this problem? Or is anyone successfully having a higher CAN ID SPARK MAX follow a lower CAN ID SPARK MAX?

Also, fwiw, I did report the issue to REV immediately before reporting it here. I haven’t heard back yet, but it has only been 1 business day and I’m sure they are swamped at this point in the season. Presumably @Will_Toth is watching this thread and would appreciate any additional datapoints that other teams can provide.

7

I suspect there is something else going on here, and I have been working to try and reproduce this today.

One data point I can give you is that the team I mentor with is for sure using both configurations, following a higher CAN ID, and following a lower CAN ID.

Drivetrain:

CAN ID 3 is leader
CAN ID 2 is follower
CAN ID 1 is follower

CAN ID 6 is leader
CAN ID 5 is follower
CAN ID 4 is follower

Shooter:

CAN ID 10 is leader
CAN ID 11 is follower

Additionally our automated integration test setup runs both configurations and has not failed in this way.

8

Thanks. Could it be specific to having the follower be inverted relative to the leader? If your shooter is direct drive, it would be inverted where the drivetrain would not. The leader < follower condition on the shooter might be what is preventing it from showing up there. Just guessing.

9

We are having an eerily similar issue with an inverted follower on our shooter stuttering and fighting the motor (we also did not count the time between stutters but the 4/sec sounds close) and have instead been using this code to manually set the inverted output:
shooter2.set(-shooter1.getAppliedOutput());

10

@RobotNerd1, do you happen to know what the CAN IDs of the leader vs follower were when you were experiencing the problem?

11

Leader 12, Follower 13

Tonight I’ll swap the roles of the motors and report our findings.

12

Thank you all for your patience, and thanks @brettle for the very detailed description. I believe I’ve been able to recreate and fix the issue. It still does not appear to be related to the order of CAN ID, but rather an interaction of the restoreFactoryDefaults() command with the act of switching the leader and follower. This would also explain why my own team’s code did not cause this issue, as well as why it is not common. Luckily there is an easy work-around and I would like to confirm that this fixes your issue as well.

After setting restoreFactoryDefaults(), add the following to explicitly tell the lead controller to unfollow:

m_leadMotor.follow(ExternalFollower.kFollowerDisabled, 0);

So using the example code:

  @Override
  public void robotInit() {
    m_leadMotor = new CANSparkMax(leadDeviceID, MotorType.kBrushless);
    m_followMotor = new CANSparkMax(followDeviceID, MotorType.kBrushless);

    m_leadMotor.restoreFactoryDefaults();
    m_followMotor.restoreFactoryDefaults();

    m_leadMotor.follow(ExternalFollower.kFollowerDisabled, 0);
    m_followMotor.follow(m_leadMotor);
  }
2 Likes
13

FWIW - We have 15 Spark Max and #6 following #15 and #4 following #45 and haven’t’ had any issues.

14

I can confirm that adding m_leadMotor.follow(ExternalFollower.kFollowerDisabled, 0) fixes the problem.

I was also able to use this fix to partially reproduce the problem with leader=14 and follower=15 (i.e. the opposite config as before). I first ran the code with leader=15 and follower=14 with the extra line. No problem (as expected thanks to the fix). For the rest of this description, I ran without the extra line to see if I could reproduce the problem starting from this configuration. I ran with leader=14 and follower=15 and that did in fact cause the stuttering. I then switched back to leader=15 and follower=14 and that did not fix it (again as expected). So far, this is consistent with the idea that the behavior is not dependent on CAN ID or particular motor controllers. However, if I then run with leader=14 and follower=15 and then power-cycle, then the problem went away. This is in contrast to the original configuration where a power-cycle was not (and still is not) sufficient to fix the problem. Is it possible that the leader/follower state is consistently initialized at power-up (or during the call to restoreFactoryDefaults()) to different values on different motor controllers?

15

Fyi, I was able to reproduce the original problem on both sides of our drivetrain. That confirms that the problem is not specific to inverted followers and also means that we have seen the problem in 4 out of the 4 places we’ve tested. So at least for us, this is not a rare problem. I have also confirmed that the fix works in all of those places.

16

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.