AdvantageKit 2023 Beta: Log replay framework, now with WPILOG/NT4 integration

jonahb55 · November 13, 2022, 4:07am

Last year, 6328 introduced our AdvantageKit logging framework as part of our #OpenAlliance efforts. Whether you’ve never heard of AdvantageKit or are already using it yourself (thank you ), we have some exciting updates to share. Feel free to skip to “What’s New For 2023?” if you’re already familiar with AdvantageKit and log replay.

What is AdvantageKit?

Most logging frameworks in FRC (like WPILib’s built-in data logging) are designed to record a limited number of fields that are explicitly logged by the robot code. AdvantageKit goes beyond this traditional approach by recording all of the data flowing into the robot code, not just its outputs. By isolating hardware interaction in subsystems, log data can be replayed in simulation. Forgot to log a field? Run a replay and inspect the code more closely. Want to retune an odometry pipeline? Run a replay and see how it would have affected your last match. Input capture and log replay completely redefine the types of problems logging can solve.

Does this feel a little abstract? Let’s look at an example. Below is some log data from 6328’s 2022 robot, showing the flywheel setpoint during a match. It’s set automatically based on the distance to the target, but it seems we forgot to log the measured distance . So how can we check that the calculation is working?

Let’s just log the new data with this call…

Logger.getInstance().recordOutput("TargetDistanceMeters", latestPose.getTranslation().getDistance(FieldConstants.hubCenter));

…run the code in replay…

…and enjoy our newly logged data:

We can confirm that the flywheel setpoint is increasing along with distance, so the calculation is working correctly. Using replay, we’re able to make that conclusion with complete certainty even though we’re using data that was never logged by the robot on the field.

Below are just a few more examples of how we’ve used log replay in the 2022 season. Solving any one of these issues would likely have taken multiple hours of testing on the real robot, but log replay took care of them with almost no time required for testing.

While working between shop meetings, we were able to debug a subtle issue with the code’s vision pipeline by recording extra outputs from a log saved from a previous testing session. More details here.
During an event, we retuned our odometry pipeline to more aggressively adjust based on vision data (tuning that requires a normal match environment with other robots). We tested the change using replay and deployed it in the next match, and saw much more reliable odometry data for the remainder of the event. More details here.
After an event, we determined that our hood was not zeroing currently during match conditions. This involved replaying the matches with a manually adjusted hood position and comparing the quality of the vision data processed with different angles (the robot’s Limelight was mounted to the hood). More details here.

For more detailed examples of AdvantageKit in action, check out the “What is AdvantageKit?” page in the documentation.

What’s New For 2023?

We’ve introduced a variety of new features for AdvantageKit’s 2023 update. Here are some of the most important changes:

WPILOG and NT4 have replaced our old RLOG format for logging and live data streaming, allowing AdvantageKit to interface with WPILib tools like Glass and Sapphire. AdvantageScope’s 2023 update adds support for WPILOG and NT4 (link at the end of this post).
Console output (STDOUT and STDERR) is now captured automatically by AdvantageKit, and can be displayed in AdvantageScope’s new console view.
The @AutoLog annotation automatically defines the toLog and fromLog methods for subsystem inputs, reducing the amount of boilerplate code.
New methods allow for convenient output logging of WPILib classes like Pose2d, Pose3d, and SwerveModuleState.
More fields and customizability has been added to the built-in system stats and power distribution logging.
Example projects are now provided with each AdvantageKit release: a skeleton project with just the basics, and a command-based example project with multiple subsystems.

The first 2023 beta of AdvantageKit has now been released. For information on getting started, check out the AdvantageKit documentation. We would welcome community feedback, questions, and suggestions as we continue our development effort.

We also just released the 2023 beta of AdvantageScope, our data visualization tool! More details here.

-Jonah

GIF

jonahb55 · December 11, 2022, 3:13pm

AdvantageKit 2023 Beta 2

We just released the second beta of AdvantageKit for 2023, which supports WPILib 2023.1.1-beta-6 (among other minor changes). The full changelog is available here. To update an existing project:

Click “WPILib: Manager Vendor Libraries” in VSCode, then “Check for updates (online)”.
Update the AutoLog annotation processor version to “2.0.0-beta-2” in “build.gradle”.
Update the WPILib version to “2023.1.1-beta-6” at the top of “build.gradle” (all robot code must be built with the same WPILib version as AdvantageKit).

hage-043 · January 8, 2023, 10:15pm

Hi Jonah,

I’m very impressed by the functionality of AdvantageKit and appreciate all of the work you guys have put into it. I have two questions about it though.

This is my team’s first year logging so we aren’t aware of the limit to the amount of data that can be logged without running into issues. Looking through 6328’s 2022 code, all data coming from motors is logged in addition to other things like OI. If we weren’t to log some of the extraneous information we don’t care all that much about, and started logging some of the “outputs” that you guys run through the replay framework, could you give us an estimate of how many fields we could log without running into problems? (We are a little hesitant to switch to the IO class stuff our first year).
After trying to model our swerve code around the AdvantageKit framework, we noticed that you guys have an GyroIOInputsAutoLogged class that seemingly appears out of thin air in your code. When we initially setup the project we made it as a copy of the command-based template you guys provide, and after checking the AutoLog plugin in the build.gradle, and looking for anything that stood out, we were unable to find where this class exists. For example, these two lines provide errors saying no such thing exists.

import org.team5557.subsystems.gyro.GyroIOInputsAutoLogged;

private final GyroIOInputsAutoLogged gyroInputs = new GyroIOInputsAutoLogged();

Also, since I don’t know where this class even is, I’m not sure what it does either. Any explanation would be much appreciated. Thank You!!

tecnicraze · January 8, 2023, 10:45pm

The GyroIOInputsAutoLogged class is generated by the @AutoLog annotation on the GyroIO.GyroIOInputs class.

hage-043 · January 8, 2023, 11:32pm

Thats what I assumed but the following code doesn’t seem to do that.

package org.team5557.subsystems.gyro;

import org.littletonrobotics.junction.AutoLog;

@java.lang.SuppressWarnings({"java:S1104"})
public interface GyroIO {

  @AutoLog
  public static class GyroIOInputs {
    public double azimuthDeg = 0.0;
    public double azimuthVelocityDegPerSec = 0.0;

    public double pitchDeg = 0.0;
    public double pitchVelocityDegPerSec = 0.0;
  }

  /**
   * Updates the set of loggable inputs.
   *
   * @param inputs the inputs to update
   */
  public default void updateInputs(GyroIOInputs inputs) {}
}

Is there any import or plugin I am missing? Do you have to run something from terminal to actually access the AutoLog plugin?

This is what the dependencies section of my build.gradle looks like

dependencies {
    implementation wpi.java.deps.wpilib()
    implementation wpi.java.vendor.java()

    roborioDebug wpi.java.deps.wpilibJniDebug(wpi.platforms.roborio)
    roborioDebug wpi.java.vendor.jniDebug(wpi.platforms.roborio)

    roborioRelease wpi.java.deps.wpilibJniRelease(wpi.platforms.roborio)
    roborioRelease wpi.java.vendor.jniRelease(wpi.platforms.roborio)

    nativeDebug wpi.java.deps.wpilibJniDebug(wpi.platforms.desktop)
    nativeDebug wpi.java.vendor.jniDebug(wpi.platforms.desktop)
    simulationDebug wpi.sim.enableDebug()

    nativeRelease wpi.java.deps.wpilibJniRelease(wpi.platforms.desktop)
    nativeRelease wpi.java.vendor.jniRelease(wpi.platforms.desktop)
    simulationRelease wpi.sim.enableRelease()

    testImplementation 'org.junit.jupiter:junit-jupiter-api:5.4.2'
    testImplementation 'org.junit.jupiter:junit-jupiter-params:5.4.2'
    testRuntimeOnly 'org.junit.jupiter:junit-jupiter-engine:5.4.2'

    annotationProcessor "org.littletonrobotics.akit.junction:junction-autolog:2.0.0"

    implementation "gov.nist.math:jama:1.0.3"
}

michaelyu · January 8, 2023, 11:33pm

The AutoLog code is generated when you build the code. It should generate the files to a build folder (might be .build which would be hidden)

hage-043 · January 8, 2023, 11:41pm

Yup… building the code will do it! Thanks everyone for your help.

jonahb55 · January 8, 2023, 11:44pm

I’m glad you got the issue with auto logging resolved. For your other question, I wouldn’t be concerned about logging too many fields. For reference, our 2022 code logged >200 fields and had no issues. The overhead from logging each cycle was ~1ms, and we haven’t observed that the number of fields logged has a significant effect on that (within reason). AdvantageKit automatically records the cycle times for user code and logging code under “/RealOutputs/LoggedRobot/”, so you can always double check that you’re getting the performance you expect.

hage-043 · January 9, 2023, 12:06am

Thats awesome! Such a cool tool.

jonahb55 · January 12, 2023, 5:40pm

AdvantageKit v2.1.0: Mechanisms!

AdvantageKit v2.1.0 is now available, with support for logging the state of a Mechanism2d object as an output field:

Mechanism2d mechanism = new Mechanism2d(3, 3);
Logger.getInstance().recordOutput("MyMechanism", mechanism);

The mechanism data can be visualized using AdvantageScope v2.1.0 or WPILib’s built in tools (Glass, SimGUI, etc).

LightspeedLazer · January 21, 2023, 4:39am

Hey Jonah,
Our team is experimenting with multithreading to mitigate the effects of packet loss, and I was wondering how does AdvantageKit handle logging the same key multiple times with different values between Driverstation packets? The code below suggests that WPILOGWriter will append all the updated data to the same table, with the same id and timestamp. This means that something probably very spooky happens when you try to simulate from a log, making multithreading with AdvantageKit a no-go.

public class Logger {
  ...
  /**
   * Periodic method to be called before robotInit and each loop cycle. Updates
   * timestamp and globally logged data.
   */
  void periodicBeforeUser() {
    if (running) {
      ...
      // Get next entry
      if (replaySource == null) {
        entry.setTimestamp(conduit.getTimestamp());
      }
      ...
    }
  ...
  }
...
}

public class WPILOGWriter implements LogDataReceiver {
  ...
  public void putTable(LogTable table) {
    ...
    // Append data
    if (appendData) {
      int id = entryIDs.get(field.getKey());
      switch (field.getValue().type) {
        case Raw:
          log.appendRaw(id, field.getValue().getRaw(), table.getTimestamp());
          break;
        case Boolean:
          log.appendBoolean(id, field.getValue().getBoolean(), table.getTimestamp());
          break;
        case Integer:
          log.appendInteger(id, field.getValue().getInteger(), table.getTimestamp());
        ...
      }
      ...
    }
    ...
  }
  ...
}

This is just my initial assumption based on a quick skim through the logging code, so I thought I’d ask what the actual answer is before making a decision on how to use multithreading.

jonahb55 · January 21, 2023, 5:17am

AdvantageKit requires that the main loop be singled threaded, and each field can only have a single value per cycle. This is required for replay to work correctly because the state of the code at each cycle must be predictable. If the code was interacting with another thread, their interactions would not be captured by the log and thus the behavior in replay would not match the real robot (like if a background task took a different length of time to execute on the robot or in the simulator).

Multithreading in robot code tends to only be useful in very niche cases, so we’ve rarely found this to be a hinderance. If threads are truly required, they just need to exist within an IO implementation. This way all of the data flowing from the thread to the main loop is logged and can be replayed (without the thread needing to run during replay).

Could you elaborate on the problem you’re trying to solve? If you’re dealing with packet loss, I suspect that multithreading is not the correct solution.

LightspeedLazer · January 21, 2023, 3:24pm

Thanks for the quick response! Essentially, we decided to copy 254’s idea to run the autonomous code and subsystem code in a separate thread, it would allow the autonomous to run cleanly without packet losses effecting the accuracy of the robot. Running a separate thread also allows us to run the auto faster than default, increasing the accuracy to something that couldn’t be achieved with the normal autonomousPeriodic() frequency.

jonahb55 · January 21, 2023, 3:41pm

It sounds like you’re dealing with two different concerns — packet loss and auto accuracy. Packet loss never affects the execution of your code. WPILib used to include IterativeRobot that only ran a loop cycle when a DS packet was received, but the current classes (TimedRobot, LoggedRobot, etc) run with a constant period regardless of any DS packets. If too many DS packets are missed the RIO’s watchdog will disable motor outputs, but that safety measure is applied at a lower level than the robot code and will affect all outputs regardless of how they were set.

As for accuracy, are you trying to resolve a specific issue? The dynamics of a drivetrain are generally slow enough that a 20ms loop cycle is more than fast enough to develop very accurate autos. Typically we’ve found that accuracy issues are a result of suboptimal tuning rather than a limit of the loop period. Of course, you always have the option of running the main loop at a faster rate but that shouldn’t be necessary for most systems.

LightspeedLazer · January 21, 2023, 4:12pm

Interesting, I was told that TimedRobot (and by extension LoggedRobot) ran periodics only when a DS packet was received. Thanks for the clarification! As for the autonomous accuracy, it was a nice side-effect of running a separate thread, and never the main goal of the thread. However, in the hypothetical situation that a subsystem has limited time to achieve a goal and requires more clock cycles than the base 20ms can provide in that timeframe, would addPeriodic allow for faster cycles while keeping logger synchrony (idea is that WPILib will know about the separate “thread” and run logger things every cycle no matter the “thread” it’s in)? Regardless, I appreciate your help and the fact that I don’t have to deal with thread synchronization!

jonahb55 · January 22, 2023, 3:06am

addPeriodic is also unsupported by AdvantageKit — LoggedRobot doesn’t have that method. In order for the extra periodic cycles to be useful, they would have to be able to read and write log data independently from the surrounding cycles. Essentially, every callback would carry the logging overhead of a full cycle. Overall, we felt that the use case was niche enough that it wasn’t worth the complexity and performance overhead. The closest equivalent is using a Notifier in an IO implementation, which we have used successfully in the past.

In general, our strategy is always to start with the simplest solution and build complexity only as required. In FRC, there are very few use cases that truly require multithreading or extra periodic callbacks. Also keep in mind that having the occasional loop overrun is rarely a cause for concern — you can use the cycle durations from AdvantageKit to debug issues if necessary. I’d recommend that you only look at adding complexity like multithreading if you start to see a measurable performance impact that can’t be resolved by other means, which in our experience is very rare.