REV Spark Flex CAN Timeout. Periodic Status 2

ISiple · January 9, 2025, 10:15pm

We are running swerve drive with Spark Flex and NEO along with Pigeon2. The Spark Flex drive motors seem to run into this issue after about 30 seconds with the robot being enabled. We have checked CAN wiring and it seems to be good, along with reducing number of motors in CAN loop. What else would cause this error?

jan · January 9, 2025, 10:53pm

We are looking into this issue right now. Do you notice anything else when the error comes up? e.g. an impact in performance?

ISiple · January 9, 2025, 11:05pm

When it happens my odometry thread stays locked and odometry doesn’t update. I can hear the vortex move slightly when it happens.

Nigelientje · January 9, 2025, 11:22pm

Are you using a template for your odometry thread? Ours only updates when a method to a Spark returns kOk.

We have been seeing the same issue. After some time, our SPARKs return kTimeout instead of kOk, meaning we will not receive any updates.

ISiple · January 9, 2025, 11:27pm

Yup, that’s exactly what’s happening to us. Did you just get rid of that for now?

Nigelientje · January 9, 2025, 11:29pm

We put every SPARK (both MAX and Flex) we have back to factory settings and that seemed to help initially, but I don’t think it solved this issue.

I assumed it was an electrical issue until I encountered this thread.
If needed, I can do some additional testing.

ISiple · January 9, 2025, 11:56pm

Did the same thing and experienced same thing. If you find anything let me know! I’m going to continue to test.

ISiple · January 10, 2025, 12:16am

I commented out the line that checks the status and haven’t had an issue since. I don’t like the workaround, but for now it’ll work.

khillans · January 10, 2025, 12:49pm

We also encountered this problem noticed reflating the firmware seemed to get rid of the issue for a while we’re also able to avoid the issue going at lower current limits for the flexes. but the current limits had to be set to 10 amps and even then this could sometime occur. We also had a weird behavior where when we placed the drive train onto the ground while on caused 3 flexes to error

jwt388 · January 11, 2025, 2:13am

We’ve been seeing this issue on our swerve prototype running YAGSL with SparkFlex/Neo Vortex drive. The errors almost always come in pairs, 2 seconds apart and there is a big drop in CPU for a second in between. We’ve also been trying to track down why CPU% is so much higher in 2025 than in 2024, but so much has changed across all of the libraries. I have a JFR dump if anyone is interested. The red dots are the timeouts.

nstrike · January 11, 2025, 2:19am

jfr-1736559923330.zip (167.6 KB)

Here is the JVM profile of it @jan

khillans · January 11, 2025, 3:10am

The team down graded the rev library back to the beta and flashed all the flexes back to beta release 9 and the error seems to still occur albeit more infrequently. we also ruled out the shake being the cause and current limits as we’ve pushed the drive train to max accel and speed and haven’t seen it occur. To be clear the motors still drive fine but our odometry gets locked and it seems like we can’t get what we want out of the motors to run our odometry.

jan · January 11, 2025, 4:34am

From that log, it looks like the Flex is actually dropping off of the CAN bus given that there are slight dips in CAN utilization that line up with the timeouts, which makes me lean towards it being an issue in the firmware. Can you check if there is a “has reset” fault on the Flex that times out if it happens again?

The CPU usage is interesting, as I don’t actually think REVLib is doing anything drastically different in either case of timeout/no timeout.

jan · January 11, 2025, 4:44am

Also I believe I have not seen any cases with this happening with MAX, unless someone can prove me wrong, but I also haven’t gotten a chance to confirm with hardware recently. When I was talking to @Jochem about this issue, he mentioned it would seem to be limited to Flex.

Nigelientje · January 11, 2025, 6:17am

Every time we had a motor reporting kTimeout instead of kOk, it also had a has reset sticky fault.

jwt388 · January 11, 2025, 12:53pm

I hooked up the Rev HW client. Over a period of 10 minutes I had 6 pairs of timeouts and I don’t see any faults or warnings. All were on Flex. We also have 4 Max on the bot and I haven’t seen any timeouts on those.

jwt388 · January 11, 2025, 12:55pm

@nstrike Are you doing anything in the library for SparkFlex that might affect how faults are reported when there is a timeout?

nstrike · January 11, 2025, 12:59pm

Absolutely! This

github.com

BroncBotz3481/YAGSL-Example/blob/dev/src/main/java/swervelib/motors/SparkFlexSwerve.java#L115-L130


      
           * Run the configuration until it succeeds or times out.
           *
           * @param config Lambda supplier returning the error state.
           */
          private void configureSparkFlex(Supplier<REVLibError> config)
          {
            for (int i = 0; i < maximumRetries; i++)
            {
              if (config.get() == REVLibError.kOk)
              {
                return;
              }
              Timer.delay(configDelay);
            }
            failureConfiguring.set(true);
          }

The delay is 5ms

jwt388 · January 11, 2025, 1:06pm

And this gets called when you do pid.setReference() every frame. I forgot that “configure” means something different in 2025 RevLib. Maybe I should enable open loop to see what happens since that calls setVoltage().

OpenLoop doesn’t seem to make any difference with the timeouts or CPU. Motor speed behavior changes as it should.

nstrike · January 11, 2025, 2:34pm

You can try to remove the configure wrapper from setReference and see what happens. It shouldn’t be that big of a deal since it’s called every user loop