Ok don't panic! SysID is "half" going away!

Our team is super appreciative of WPILIB and all the work that they do. Without the cool infrastructures and tools like command-based programming, and vast libraries that actually make “talking” to the components of our robot feasible, I’m sure our team would not have been as successful as we were. I say this so that no shade is thrown at the wonderful WPILIB team.

Soooooo, when we heard that SysID is moving half into the robot code and half an offline tool we started to panic. You see, we are a Java team, and java with it’s garbage collection issues and time jitter issues (these are things I know very little about), will record your characterization tests in a noisy manner such that your constants would not be calculated as cleanly as you would like.

With that said, I’d love to hear what your team is planning to do about it. Are you planning on switching to C++, writing your own separate standalone tool or “don’t care”?

10 Likes

I am a bit panicked by this, though glitchy I found sysid very useful. It looks like I will have to learn something new which isn’t so bad but it will be during build season when I have a lot of other things to do. I hope the data logging and calculation instructions are clear and we can get stuff done. Recalc should be a good start to our feed forward for mechanisms. I think the data logger approach will make swerve drive characterization a little better.

4 Likes

That’s my stance at least.

It’s always possible to write the math in a way that accounts for the actual time between samples (rather than assume a fixed sample rate).

Tyler’s a sharp guy, I trust that either he and team will either account for this, or provide clear warnings if there are restrictions, and a way to measure the impact of inconsistent timing.

6 Likes

If you want really accurate data you’ll have to run in C++. It won’t be that hard to do.

Accounting for the noise in the math can only get you so far, but also be aware that not all mechanisms accelerate fast enough for any of this to matter. Use common sense, and know the update rate of your main loop (it’s 50hz by default).

3 Likes

If it’s just reading data, moving it to a faster loop on another thread shouldn’t be difficult on almost any project I guess

We haven’t figured out how yet. We can deal with a consistent delay, but nondeterministic delay horribly breaks the tool’s assumptions.

The SysId plots I’ve seen for Java data collection are really bad. One of them even had an R² of -1 (< 0 means it was worse than guessing the average everywhere). My advice is “avoid using Java for data collection if you can”. That could mean vendor-side logging in native code, roboRIO-side DMA in Java, or users writing their robot code in C++.

Another far-fetched option would be supporting a language like C# that actually gives you control over when the garbage collector is allowed to run (note that C# is not a supported language at the moment, though it would fix a lot of problems like this; syntactically, it’s basically Java but better). No-op Java GC didn’t fix it by the way; it ran out of RAM immediately.

4 Likes

To get a good fit, you need deterministic data sampling. The issue is even separate threads are subjected to the whims of the garbage collector pauses.

I’ve had teams say their Java data collection and custom OLS fit “gave fine feedforward gains”, but I don’t think they actually used SysId to measure how accurate the fit was. My educated guess is the feedforward was super inaccurate, but their subsystem was so overpowered that feedback took care of it.

2 Likes

I’d love to see another standalone example c++ code for each type of mechanism any team could deploy after entering controller ids - but for now we’ll just keep a 2023 rio and radio around and hook them up whenever we need to characterize.

3 Likes

I wouldn’t bet on it. They’d run into the same maintenance nightmare SysId did (see WPILib 2024 Beta - #15 by Peter_Johnson for more info on that).

1 Like

It can also be the case that the mechanism is so underpowered that it accelerates slowly compared to the loop timing. Less common now that brushless motors are so common, but old 4CIM drives definitely could be characterized this way in Java.

1 Like

I think the easier solution would be a RIO 2 SD card flashed to 2023 and a long ethernet cable.

I don’t think that’s the best way forward for sysid. While a stand-alone tool does resolve the timing issues, I think a log based analysis tool is far more useful for teams. The ability to tune feed-forward models asynchronously to robot operation would allow teams to set up logging and basic control loops, then while drive practice is being done, they could download logs and tune the feed-forward models.

3 Likes

CTRE has already added data logging support (to Phoenix logs that can be converted to WPI logs that SysId will be able to read). It shouldn’t be hard for REV to add something not dissimilar to it. Also, in the worst case I think WPILib can provide something that will work even for vendors that don’t provide support by providing a C++ thread that runs at a fixed rate and a way to call arbitrary JNI functions from it.

In short, I think there are avenues in various stages of development to get the required data in ways that are not dependent on Java timing.

In other words:
image

13 Likes

As a point of clarification when talking about data quality — concerns about noise and non-deterministic timing are most relevant for acceleration (kA, not kS and kV). There are lots of mechanisms (including many drivetrains) where kA is largely unnecessary and often unused by teams anyway. 6328 hasn’t used measured kA values in years, so definitely try data collection in Java for the particular system you care about as a starting point. There are cases where the difference in data quality matters, but that’s far from every mechanism.

3 Likes

They can make Kv bad as well. Observe this arm data collected with Java.

One of the use cases SysId was trying to enable was running one routine and not having to tune anything to get optimal feedforward and feedback gains. Inaccurate Ka breaks the feedback calculations as well as physics sim (tho few teams probably use it, unfortunately).

1 Like

How does the team justify this as an acceptable solution for your product, when 93.5% of your “customers” (teams) are not using C++? (source: WPIlib data reporting published by FIRST).

What does the team working on WPILib consider when making decisions like this? From my understanding, this answer is effectively “if you use Java, we don’t care how accurate the data you get is, and it’s your fault for using java.”

Is this a priority that the team has? Is this a priority that REV has? Or are Java/REV users simply “SOL” for this season?

2 Likes

It’s between this and the tool not working. There is flatly not the bandwidth to support the generated robot program moving forward.

It will be easier to learn how to write a simple test routine in C++ than to wrestle with the error swamp of the generated executable. Test routines in Java will still work for many mechanisms.

5 Likes

What if teams make standalone programs just for tuning? So you don’t entirely switch away from Java, but just make a simple program to spin motor for analysis?

1 Like

Elaborating on this, I’ve seen more cases of SysId breaking than working as intended. It didn’t work for my team (3512) at all during the 2021 at-home challenges. I also commonly got help requests from teams incorrectly configuring the project generator (it was very error-prone).

2 Likes

Sure, my point is that it depends on the mechanism and the requirements of the control loop. I don’t doubt those results, but I could just as well dig up dozens of examples of velocity FF calculations we’ve done from Java with >0.95 R2 values. Some mechanisms require more precise data collection (including kA values) and some teams prefer the alternative feedback gain calculation, but that doesn’t apply to everyone. Basically, I don’t think there’s much reason for panic here.

1 Like

We do care. That’s why we made SysId deploy C++ robot programs for data collection. It turned out to not be maintainable. I don’t see a way to fix the data quality issue that doesn’t involve native code at some level.

2 Likes