Experiencing Delays in Robot Loops

My team noticed some serious delays in controls in a few matches at the FiM Championship. It was most prevalent in our first quarterfinals match. After the match, we reviewed the Driver Station logs and found that the robot started looping only once every ~0.5 seconds. With little to go on and not much time to fix any problems, we went out for the second match and told our drive team to restart the robot code if it happened again. It occurred twice that match, and restarting the code did indeed mitigate the issue.

We’ve pulled the logs for the matches where delays occurred. There was only one match where we saw delays and it fixed itself temporarily before happening again later. In every other match it persisted. The delays in the loop also continued after teleop ended and the robot was disabled. When the robot is disabled, we only read inputs from sensors/network tables/driver station and write to SmartDashboard.

In total, it occurred in three matches: Qualification 38, Elimination 2, and Elimination 6.

Our code is written in Java with based on the WPILib TimedRobot. If it was something occurring within our code, we’d expect to see log messages for overrun loops, but there aren’t any.

Our drive team doesn’t remember the issue occurring at previous events. We won’t be able to pull logs for the previous events until we unpack our crate at champs.

The only major change we made in between events was adding a Color sensor. We did have it plugged directly into the Roborio I2C, which we have since learned is related to a know issue causing lockups. We plan to move this to the navX2 at Champs, but couldn’t find anything talking about consistent 0.5s delays.

The only other change we made was adding additional auto routines. We specifically added one before playoffs to better fit with our alliance. The Trajectories get loaded into memory at robot init. As a last ditch effort, we deleted some routines that were deprecated before our third quarterfinals match. We were able to play that match and the two semifinals match without experiencing the same issue. Our thought was that the additional trajectories were just enough memory to cause pressure. We aren’t sure this was actually the case. The robot logs show that there is 252 mb available. The match after changes, that only went up to 253 mb.

We are wondering if anyone has experienced a similar problem and may have a solution.

Below is a screenshot of the driver station logs from of the one of the matches in question. I’ve also included the related logs as an attachment.

Problematic Log Files.zip (240.5 KB)

Several teams have reported long delays in reading from either I2C port. The known issues page on WPILib has some links to alternatives such as using a Pi Pico (it’s not listed yet, but another team wrote a Teensy library as well).

I second @Peter_Johnson’s advice about I2C, but I also note that your CAN bus utilization is hitting 100%. At this level you will experience delays and missing data. This generally comes from one or more of the following causes:

  • Physical noise on CAN bus: loose connectors, missing termination, pinched wires, interference.

  • Status congestion. CAN bus devices by default report status fairly frequently. For example, each Talon FX (including the Falcon 500) is estimated to use 4.6% of the CAN bus by default. You can mitigate this by increasing the status frame period for motors that are less time-critical, such as followers and power-control (REV, CTRE).

  • Query congestion. Some API calls cause a CAN bus query and wait for a response. If you’re fetching device data just to report on SmartDashboard, I believe you would be better doing this in a separate thread, possibly to run at a lower frequency. (But see also Peter’s remarks on this subject).

Looking in your logs, I see that you’re spending a lot of time in teleopPeriodic and LiveWindow.updateValues. I’d have a close look at those. I also see some blips for autonomousInit, which I’m guessing is related to trajectories.

2 Likes

The canbus utilization noise is another known issue, as shown on the link @Peter_Johnson provided. At least from the picture in OP’s post, the canbus utilization is likely to be closer to 60-75%

1 Like

Thanks everyone for taking a look. As planned, we moved the color sensor from the roboRIO I2C to the MXP I2C before champs and have not experienced major delays since. We still had some loops missing, but never more than one or two at a time. Much better than 0.5s delays. Ideally, we would move it to a coprocessor as suggested.

1 Like

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.