Large amounts of packetloss

After going through the DS logs, I have realized that we have had huge amounts of packetloss all throughout Alpena #1, and continue to have the issue.

The only reason I did not realize it sooner is because our focus was the cargo ship which does not involve our elevator raising.

While practicing the rocket for Alpena #2, it was noticable that our lift motors would shut off, the signal light would stay solid, and pretty much all functionality was lost during the packetloss.

I could feel it in the drivetrain at Alpena 1, but assumed it was our brown out issue.

Here is a log file from Qualification Match 63:

In terms of cameras / bandwidth, all cameras are streamed from a raspberry pi.

We have 3 cameras, but only 2 of them are streamed at once. One of them draws ~1mpbps while the other draws slightly less than 2mbps.

Considering the cap is 4, I can’t see why it would be a bandwidth issuelogs.zip (321.9 KB)

I’m no expert at this topic myself but I’ve heard people mention that the positioning of the radio can affect signal strength. We have never had this issue before, but some teams have mentioned that placing the radio near motors can affect connectivity due to noise.

By the looks of your DS log, the issue is due to your battery voltage dropping too low, there are a couple of brownouts recorded in the log (indicated by those brown/orange dots at the top). For a short period there the RoboRio would have disabled all motor outputs until the voltage came back up.
How often did you observe driving problems? Was it complete dropouts for a second or couple seconds (likely brownouts), or jittery movement (likely packet loss or really bad brownouts)?

You are correct about your bandwidth utilisation, in that it shouldn’t exceed 4mbps, however if it did, I believe the radio prioritises robot control communication over other data.

@0x54796C6572 is correct in that the placement of the radio can affect the quality and strength of the signal. Such as enclosing the radio in a wire box (yes I have seen this, no it didn’t work), or around motors.

WPILib has more information regarding RoboRio brownouts.
https://wpilib.screenstepslive.com/s/4485/m/24166/l/289498

3 Likes

Seconding this - when we’ve experienced serious packet loss, it typically corresponds to large voltage drops - it seems like you’re experiencing huge voltage drops, dropping down almost to 6V at one point. I’d check to make sure your competition batteries are fully charged, and make sure all the connections from your battery to the PDP are secure.

If you graph brownouts on the log graph, do they correspond to voltage drops?

2 Likes

Another thing to look into is that your roboRIO CPU is pretty consistently high. How is your DS CPU during the match? Do you have anything other than the DS app itself open when you are in the alliance station? This would include Labview/VSCode as well as they both keep the CPU busy even when they don’t have focus.

Here is a graph of one of our elimination rounds at Alpena #1

I graphed only voltage, brownouts, packetloss, and roborioCPU.

It’s hard to tell if the packetloss is caused by voltage. You can see that before the match starts (indicated by the green line at the top) the battery voltage stays constant, but there is still packetloss to be seen.

We always go into matches with fully charged batteries.

What could be causing our voltage to dip so low?

In terms of running mechanisms, we have an arm which stalls at ~.5v (TalonSRX), an elevator which stalls at ~1.5v (2 talonsrx’s), and our drivetrain is powered by 6 NEOs.

The log also lists 47 total brownouts during this match.

I don’t know if there is a way to check the logs for this, but I don’t think we run into any laptop resource problems.

When driving down the hallway to rebag the robot last night, I noticed that the LEDs on our SparkMAX’s would return to magenta for maybe 1/5 of a second.

I’d describe it as jittery rather than huge a huge drop.

Do you think it could be our 6 NEOs? We noticed that when switching directions in high gear the robot would rapidly brownout, but would not in low gear.

(To clarify the test. we had the robot up on blocks so the wheels were not contacting the ground. In high gear, we would drive them forward at 100%, immediately driving at -100% caused them to rapidly brownout.)

Could you graph your pdp total current draw? That could provide some indication of what is causing the voltage drops. If you are driving at full power and operating your mechanisms that could pull some high amps and drop the voltage.
Do you have a battery beak or similar with which you can check the quality/health of the battery?

So jittery motion could be caused by brownouts under hard acceleration. The spark Max’s turning magenta would indicate that they are being disabled by the Rio, a sign of a brown out rather than packet loss.

Here is a graph of total PDP draw vs Voltage

The number correlated with the highest spike is ~200

If you know what pdp channels your mechanism and drivetrain are plugged into you can look at what is drawing a low of current. Although I’m gonna guess it’s the drivetrain, that’s the usual culprit.
Do you get the jittery motion when the robot is not under high acceleration? Ie moving slowly.
If that is the case and it is the draw from the NEOs, you can try implementing current limiting on the Spark Max and/or a ramp rate to prevent high acceleration.

We are moving from 6 to 4 and putting 2 on our level 3 hab mechanism.

Which ports on the PDP correspond to which channels?

Edit: Looked up a pic, we’re good.

Edit 2: We have one NEO on each side of our drivetrain plugged into a 30a breaker. Not sure if this could be causing any issues?

Here is a graph of only the NEOs. Keep in mind there are 6 of them.

The arm appears to draw a steady 10a throughout the match.

I’m unsure of which ports the elevator uses.

I would have your entire drivetrain on 40A breakers, having the breaker trip during a match would be sub-optimal. However if anything that would prevent sustained high current draw and voltage drop, so shouldn’t be causing an issue (until you trip it) [Note, I don’t actually think our team has ever tripped a breaker, so I’m not 100% sure what applications need what breaker].
And yes the ports on the PDP are numbered.
Even if you reduce the number of motors on your drivetrain, I would still suggest adding a ramp rate to smooth out the output, and prevent large acceleration.
The recorded brownouts likely match up with jittery motion.

I will do all of these things to prevent brownouts, but I’m still not sold that this will fix the issue that I had originally posted about.

Essentially, when raising our lift, sometimes it would drop packets and output 0 voltage. This was without moving the robot, so the neos should have had 0 amps going to them.

Even without moving anything on our robot, the signal light would stop periodically, and always correlated to drop packets.

Assuming that current draw causing brownouts is not causing the packetloss, what else would you target as the culprit?

I assume this packet loss shown is with wireless communication with the radio (due to a qual match and FMS detected), what is the packet loss like when you are tethered to the robot using ethernet or USB? Is the packet loss any different if you use a different computer as the DS?

Looking through the logs in between matches, there is still a good amount of packetloss (20-30% sometimes. Not 100% sure if the DS log indicates total packets dropped or packetloss %)

Also, what would you call a safe voltage to drop to? 10v, 9v?

The DS records packetloss %.
Regarding a safe voltage to drop to depends on the situation. You defiantly don’t want to be anywhere near the Rio brown out threshold though. When demonstrating our robots we change the battery when it hits around 9v-9.7v. In conpetition the robot is likely to draw high currents breifly and as such i would expect the voltage to drop. I’d feel good if it was above 9v or 10v maybe briefly 8.5v (wouldn’t want to see it below that), however I can’t check our DS logs at the moment as it is in a shipping crate coming back from our regional! (At least I think that’s where it is…)

If you still have high packet loss tethered to the robot, it is unlikely to be a wifi problem (of course it could still be a problem, just a different one), if you have high loss on ethernet it could be a problem with the ethernet adapter on the laptop, or the radio, or the Rio (or maybe an ethernet cable?). I would check it on USB. (At our event we could never connect tethered on ethernet, had to use USB all the time, probably a different issue though.)
Perhaps try re imaging the Rio and radio? Or testing with a different laptop.
If 2017 we had major problems with our driverstation (which had a Celeron in it) not being able to keep up (max CPU) and was dropping packets all over the place to the point where we couldn’t control the robot!

A loose connection to the battery can create high resistance through the battery, lowering your voltage to the point where you have frequent power issues - I would carefully inspect all of the connections that go through your battery, breaker, and PDP - we had this issue once, which led to huge comms and frequent and unexplained jitteriness and brownouts.

1 Like