Dropped packets between ports on radio

Tracking down a problem we saw at St Joseph District last week.

The team has an onboard coprocessor, so to debug the whole shooting match in the pits, we left the RoboRIO connected to the radio, connected the second port on the radio on a switch in the pits, plugged the DS and coprocessor into the switch. Traffic from the RIO goes into one port on the radio, out the other, through the pit switch, and to the DS.

(Can we please please please get a 3 or 4 port radio for FRC? Pretty please?)

In this configuration, we were seeing 50-75% packet loss indicated in the DS.

I had the team leave their radio out of the bag, and we put it on the bench today. Plugging the RoboRIO into “it’s” port on the radio and the DS into the other port has the same results; tons of packet loss.

Putting just a cable between the DS and the RIO is much better, though we still see packet drops.

I used iperf3 to move a a few hundred meg of UDP packets between the ports, and saw no packet drops.

Why would the DS/RoboRIO software indicate packet drops on a path that does not seem to actually be dropping packets? Is it that sensitive to the possible additional latency between the radio ports?

Most likely the issue is using the radio as a network switch. I am confused by the troubleshooting steps you took. What was the initial problem the team was having?

Putting just a cable between the DS and the RIO is much better, though we still see packet drops.

This is the more confusing symptom. What size packet drops are you seeing? Also, did you try multiple cables?

We had the same problem and we resolved it by not using the radio as a switch. We put a switch on board the robot instead and would plug into the switch anytime we needed to tether.

Better yet, if you aren’t using a coprocessor, plug into your RIO using the handy dandy USB port!

Edited since @prensing brings up a very good point.

Very hard to see packets from a coprocessor if you are not on the Ethernet side.

1 Like

I am confused why using the radio as a 2-port switch (which it is!?!) would cause packet loss. Anyone have an idea of why this would happen?

Its not actually a true switch, its 2 ports bridge in software. Non FRC use for this is one port is WAN and the other is LAN.

1 Like

I agree the issue is using the radio as a network switch, but it’s supposed to be able to do that.

The initial problem was seeing the packet drops when tuning video in the pits.

We were seeing 50-75% packet loss.

Multiple cables tried. Cables were fine when used between DS and RIO.

I put together a test rig with PCs running iperf3 on each port, and was able to shovel a few hundred meg through there without iperf showing any packet drops.

How were you measuring the packet loss?

What was the bandwidth looking like during that time? Also, were control packets were being dropped or just vision? It seems like a few separate things are going on which are all mixing together:

  • The radio is not behaving like a true network switch. My understanding (though there are better people to ask about this) is that it isn’t a network switch to begin with, but the control system treats it as such.

  • The vision packets are overloading the bandwidth cap. When too many packets are sent, the RIO prioritizes control packets which makes the other packets drop and gives you those huge numbers. On the field, you are capped at 4 mbps.

  • Maybe there are some other code issue causing dropped packets when connected directly to the RIO

So then I think the best bet is to use a network switch in between the RIO and radio and turn the vision off. That will at least tell us if it’s a vision problem or not.

We saw similar behavior at Midwest. 75% packet loss while tethered in the pit via Ethernet, nothing on the field, and nothing while tethered via USB.

This is my understanding as well.

Edit: We “solved” it by using USB. :frowning:

Is it possible that the radio is imposing the bandwidth limit on all traffic though it, not just over WiFi? Clearly everyone is thinking of this as a WiFi limit, because that is the intent, but maybe it is limiting all ports (or even the sum of the ports).

We had similar packet loss issues whilst in our home practice field, but never at the tournament.

This aligns with… some … of our observations. We saw a limit on downloading log files from the RIO over SCP, much lower than what we had at the shop. Still, the scenarios we saw the large packet loss in should have been only driver station - no extra telemetry or camera streaming. Definitely should have been well under the bandwidth limit.

with the test I was running yesterday, there was NO vision going. Just control packets.

Probably not code. If I cable directly from DS to RIO without the radio in the middle, packet drop falls way off.

My team had a similar issue week one. Ultimately it came down to the switch we were using. As soon as we removed it all the packet loss went away