Is 70% CAN Bus Utilization Bad?

According to the Driver Station Log viewer we have a very consistent 70% CAN Bus utilization. Even after removing a dozen or so SmartDashboard() reporting motor speed/position/voltages that generated CAN info requests and making sure we didn’t make the same CAN request more than once per periodic() loop, the utilization is the same. There are a few CAN motor follower pairs: left drivetrain pair, right drivetrain pair, and the flywheel motor pair.

So is 70% typical? is it High?

Are there any tools for monitoring what devices are creating what CAN bus traffic?

2 Likes

I found this thread, but it doesn’t exactly answer my question.

There was a mention of reducing the frequency of status frames? About to google what that means…

With how can works, it’s actually the device that sends status packets. When you call a function to get a value from the device, the roborio is actually just reading the last value the device sent. So reducing the number of reads in your code will actually do nothing to lower can utilization.

Each vendor has an api that lets you lower the period the status frames are sent at. So if there is certain data not needed as often, you can make that data come less often.

70% utilization isn’t really a problem at all. It’s perfectly fine for a can bus to be at that level of utilization.

5 Likes

From our experience, 70% is fairly typical, when it gets over 80% we tend to see issues.

70-80% isn’t absurdly high, but some caution to higher intervals / higher rates for some mechanisms is needed; turning off unneeded status frames, and just being aware of what’s in each group for each respective CAN motor controller / CAN sensor can help you reduce utilization. I wrote a thing about this a while back to better summarize my own notes on this type of thing.

7 Likes

70% may also mean more latency and latency variation, which you may care about in some (probably rare) cases.

These numbers (and higher) becoming increasingly common tell me the next hardware iteration really needs to support multiple CAN busses.

OK, Thank you.

I’ve just read about how you can lower the status frame rate on some CAN devices. This would lower the overall CAN bus utilization right?

I found this SPARK MAX User's Manual that discusses update frequency and how to change them on Spark Max controllers. If I’m understanding this correctly, I can safely lower the update frequency on devices like follow motors and devices I don’t need to poll for velocity, position, voltage, sensors, etc. And it seems like I want a keep the default poll interval for lead motors and motors I’m getting data like flywheel RPM and drive train position and velocity.

Do lead/follow motor pairs communicate directly over CAN to each other or is information proxied through the RoboRio?

Is there any spec available on how the ID/Priority (or other early-message) bits can be manipulated from user space?

In particular, I’d expect in non-FRC scenarios that I’d put my control messages (and critical sensor feedback) on higher priory messages to keep jitter and latency down as much as possible for critical systems (ex: drivetrain leader motors). However, for stuff like PDP or PCM, I’d want them on much lower priority messages.

Would manipulating the CAN ID for devices accomplish this?

There is not. That’s one of the flaws with FRC’s CAN addressing scheme: the vendor packet priorities are essentially ordered by vendor ID, so all of CTRE’s PDP messages preempt REV’s motor controller messages, as an example (I forget the exact vendor assignments).


It’s a legacy partitioning scheme that originated from the CAN Jaguar; existing vendors have had to conform to it. REV, CTRE, and some other people were looking into improving this at one point, but idk if it made it past the draft proposal stage.

No, because the device ID comes last in the arbitration.

4 Likes

eeew. Ah well

At least as I read it, the Device Type table seems laid out in a very reasonable way (looks like roboRIO = most important, followed by motors and relays, with PDP and PCM much further down the list).

As you mentioned, definitely interesting that all CTRE devices get prioritized before any REV devices. Not like it really should make much of a difference… but I suppose on a CAN-heavy mixed system of SparkMax’s and Falcons, it’s interesting the Falcons would get priority for similar messages.

Does this happen for sure? As I read the Device Type, I’d assume the PDP would pick device type 8 (prioritizing it below any motor controller). I might be misreading though?

I think you’re correct.

1 Like

At some point, I imagine there will be a disruptive migration to something like BroadR-Reach. There is probably a non-disruptive path to higher rate CAN, and multiple CAN buses could also help a lot. But getting into compliance with CANopen, reworking the header used with FRC, etc. tend to favor a migration where there are be old and new buses for some number of years, and devices gradually change over. At that point, may as well get all the disruptive changes in at once.

The trick is to get the timing right, minimizing the number of times this is done and winding up on the right choice (CANopen FD, BroadR-Reach, something else?). Other than possibly multiple CAN buses, I don’t think anything like this will happen at least until the next major control system refresh (i.e. after anything currently in the works). There’s a big ecosystem of vendors and devices, and big inventories of hardware in use. In fact, it could be that this is very far out, or never happens. But I do think something will be needed fairly soon – even if that is only techniques for reducing status frame rates becoming pretty common.

2 Likes

BroadR definitely has some very unique advantages for this application. I don’t personally feel they yet outweigh the negatives for FRC (especially related to cost - both of actual components, and the supplier rework required).

An Open CAN protocol with more user control available would be awesome, especially if you could find a packet structure to coexist with with the existing standard. Enthusiastic teams could fully migrate right away, and teams that don’t care wouldn’t notice.

FRC Can is already running on the fast 1M standard. I’d be worried a bit too about updating to the 5M CAN - even if all hardware could support it, I worry the minimum-viable wiring quality would be out of reach for many beginner teams (and cause lots of non-reproducible failures).

My immediate “solution” instinct is to ask NI to make the guts of something like this into a form factor that mates to the expansion port on the RIO (and RIO 2) and produces an extra CAN bus. With the right software support, teams would be able to have two independent CAN buses to help split their load.

1 Like

There’s also a point to be made that a lot of the data that’s currently on FRC CAN busses really shouldn’t be. CAN is originally an automotive standard designed for short, relatively infrequent, safety-critical messages; this is why each frame payload can be a maximum of 8 bytes. For things like setting the velocity setpoint of a Talon, that’s great. For reading high-resolution quaternions from an IMU at a fast rate, CAN isn’t ideal.

7 Likes

I tested reducing the frame rate for status update on follow motors and non-monitored motors. I reduced the frame rate for 6 motors on our robot: 2 drive follow motors, 1 flywheel follow motor, and to a lesser degree on 3 other motors that are not position or velocity controlled. The end result reduced CAN bus utilization from 70% to 59%. No effect on robot performance.

This may be reducing the number of occasional CAN stale and/or missed update warnings. However, I just noticed that the majority of these messages occur when pushing code and enabling or disabling. Something that you can only tell by opening the log viewer. I need to take a closer look at our older logs to see if this is the case prior to this change. At the moment we’re in the middle of trying to get our At Home Challenges filmed!

I assume you’ve seen this thread?

2 Likes

I had not. Thanks for the pointer @nuttle. We may have wasted an afternoon fixing a “broken” CAN wire that was just a software problem.

1 Like

CTRE’s CANivore seems to be pushing for CAN FD. I think next year we will probably get a CANivore and run the drivetrain off of it (8 falcons, 4 cancoders, and a pigeon2). Those are the noisiest items, everything else on traditional can, but that is for next year as we don’t have a canivore n this year.

2 Likes

last night we got or CANivore up and running. 10 falcons and 4 cancoders. 20-25% bus utilization. we are going to try increasing the update rate of the cancoders to see how it handles it… so far we are very impressed. total bus length is probably around 7 feet. we were very particular about maintaining twisted pair as close to all connections as possible . no issues so far