Lag Issues On Field

[tinybob20]

Last week at the Finger Lakes Regional, our team experienced serious lag issues. We were not able to replicate this with either our tether, or wirelessly (back at home). our network usage was about 1.5Mbps consistently. Looking at our logs from the games, we were not able to see any packet loss nor a abnormally high ping. We are using Java, and run our drive system in a separate thread. we were originally using the smart dashboard, but after losing camera feed, we went back to the default. Our camera was set to 320 x 240 at 15fps, with 70% compression. We were just wondering if anyone else had this issue, and if anyone was able to fix it.

[gerberduffy]

We've had field communication issues for a number of years now. We found that the root of the problem most of the time is spawning too many threads that ate up processor time, or logging output to stdout/files, which kills flash memory and halts the FPGA (cRIO).

In the old version of our software, every robot sensor/component spawned its own thread, and we suffered often from serious comm issues, even at home. To solve this, we put all sensor polls into a single main-event thread that polls ALL sensors a few times a second, and that knocked out nearly all of our problems.

It's odd that you never had problems at home, do you have any custom hardware, or manually set the IP's of something? You could have an IP collision with another device that the FMS is looking for.

[Greg McKaskle]

If you post the log files or images of them I can help look at them.

Responsive driving requires that all elements of the communication chain are not introducing lag. Conversely, if you have lag, any element of the chain could be causing it, and it is generally not easy to identify which link is the issue.

The chain starts at the DS. The DS reads the joysticks and sends their value out to the robot. If the drive computer has high CPU, it can introduce some jitter or lag into the control packet timing.

The next link depends on the venue. It can be a cable, or a managed switch, or wireless. But skipping a few robust elements that you generally don't control anyway, the next link is the DLink, then the robot.

On the robot, your code needs to process the packet, respond, update drive motors, and produce a status packet.

If the Robot CPU usage is high, it can introduce lag between when it reads the joystick and when it responds. Even if the CPU usage isn't high, if the user code doesn't utilize a packet due to a delay in its code, it feels like lag. Sleep statements are generally the cause of this form of lag.

Back to those elements we generally don't control. We can't control, but we do influence them, and that is where the camera settings and other network usage come into play. The system this year implements QOS and bandwidth limiting. My explanation for this is that the communication on the field travels in lanes, like on a highway. Each robot has a couple lanes it can use for traffic to and from the robot. Your robot can no longer use the lanes of other robots. QOS is a bit like an HOV lane, or a flashy light that tells other traffic to get out of the way -- perishable cargo. If you put too many cars or trucks on the road, you cause your own traffic jam. QOS does a pretty amazing job of keeping the drive code from lagging. But flashy lights only go so far, and we aren't on the autobahn anymore.

Greg McKaskle

[tinybob20]

Looking at our log files, It seems that we max our cpu a couple of times during the match, but at otherwise remains at around 60-70%. We only use about two threads, the main one and one for our drive. At our competition, we talked to one of the FTAs, and according to them, our network usage seemed normal, but a little on the high end, lower than some, higher than others.

[Greg McKaskle]

It looks from your log file that you are still using the older DS. I'd recommend updating to the later 2/12 or 2/13 version so that you'll have trip time and lost packet info in your logs as well.

Greg McKaskle