Unexplained packet loss

What could cause tons of packet loss at around 1.25 minute intervals? We have tried:

*Reimaging and updating the Rios
*Updating radio/hotspot
*trying the connection from another laptop
*fully reinstalling frc uilities
*trying two different radios and rios
*connecting to the rio with empty code
*closing all other programs on the pc before connecting
*going out of range of other networks

We still get around 5 seconds of very intense(50-80%) packet loss every 1 minute 10 seconds, but before that, everything is perfect.

1 Like

Is this at a competition or in your lab?

Do you have some sort of coprocessor on your robot (Raspberry Pi etc)?

I’d ask for your code, but if it really occurs without any code loaded onto your RIO then that’s not worth anything.

Does it happen while the robot is sitting still, or only in match play?

What connections have you tried? Is this wireless, tethered over ethernet, tethered over USB? I see your listed team hasn’t competed yet, so I won’t ask about the field.

Are there any other devices on your robot network? Coprocessor, camera, etc.?

What software do you have on your Driverstation laptop? Some programs commonly used in FRC try to contact external servers to check for updates at predetermined intervals. This has been known to cause issues like you are seeing.

3 Likes

We have a pi on the robot, but disconnecting it did nothing. It happens even if the robot is just sitting there while disabled.

You might want to learn about Wireshark, which is a network sniffer. You could install it on your DS, turn it on and then watch the traffic. Correlate the packet loss time with what is in the packet capture, and that might tell you what is causing the problem. This is assuming that the loss is caused by a packet storm of some kind.

If you’re within a school, then there can be conflicts with the school network.
If you are hosting an AP (e.g., the robot radio in home setup), then there may be devices, for instance phones, that are pinging it periodically.

When you tried a different PC, was it another school PC or a different person’s personal PC?

I’d suggest looking at Task Manager to see what processes are running and turning off anything that isn’t part of the basic Windows OS. You can always turn it back on later, but try eliminating all extraneous software.
Turn off firewalls, anti-virus, defender, etc.
Turn off 3rd party updaters.

1 Like

To shed some more light on the issue:

  1. As Liam stated, the packet loss happens approximately every 1.25 minutes.
  2. The CPU usage is very consistent until the packet loss period
  3. When the packet loss occurs, the CPU usage drops immediately to about 50% of what it normally is
  4. It is steady at the reduced CPU for the for a very short period and then immediately returns to its normal usage.

We had this issue last year, although the bursts were every 15 secs or so. We found it was triggered by Wing IDE, the Python IDE we used for robot code. We switched to VS Code this year, specifically to avoid the packet loss storms. I fired up Wireshark a few times, but didn’t find anything conclusive. No network traffic coming to/from Wing, but it was like clockwork every time it was running alongside the DS.

If anyone tracks down the cause of this I would love to hear it.

That sounds like something is waiting on disk I/O or a network call that’s running synchronously somehow.

A possible scenario is you don’t have enough memory and as memory is consumed you are paging to disk.

We may be misunderstanding something. Our understanding is that the data that we are seeing in the log viewer, including the packet information, is from the RoboRio. We are thinking that we are seeing the RoboRio CPU % fall and then go back up. Is this not correct?

I thought you were talking about the CPU usage on the driver’s station. There’s two ends of the conversation for packets to be getting dropped.

What’s happening to the CPU on the driver’s station when this happens?

Question: Is your radio mounted close to any motors?

Sorry, No this is all from looking at the Log Viewer. I don’t know what the CPU is doing on the laptop. We have been focused on the RoboRio side of things.If the laptop CPU is good, what could be causing the issue no the robot side?

No, they are mounted ~10" above them.

We just have VS Code, CTRE stuff, and the various utilites packaged with the update suite.

If you are using iterative robot, the lossy network would cause the robot CPU% to drop, as the iterative cycle is triggered by packets from the driver station.

It seems most likely that your packet loss would be caused either by a process running on the laptop (e.g. a virus scanner) which pops up every 70 seconds, or something else on your network bumping in. If you switch to tethered networking and disconnect all co-processors, the problem will temporarily go away if it’s on the network. If this “fixes” it, add each co-processor back to see if it is the issue. If you add all back and it’s still gone, you have a wireless issue (perhaps interference from another computer). If the issue did not go away, look for processes running on your driver station computer which jump in resource usage at the same time as your slowdowns.

Just checked, we are using an iterative robot and now everything make sense. I was basing my troubleshooting on using a timed robot.

We will look into things and post what we find.

Which radio are you using?

We had a similar problem last year at home when communicating via radio,

It ended up being specific to the wireless radio in the laptop; we turned off the onboard one and put in a USB wireless dongle, and the periodic packet drops went away.

YMMV.