Communication Issues?

[xraymypanda]

San Diego had one really weird match. Match 88, every robot misses in autonomous. About 10 seconds into teleop every robot started to lag and the d's would flash from no connection to teleop enabled. I had my driver get his hands off the controls, but other teams we spinning due to lag. We had to reset and turned our robots on one at a time. Of course my team was the suspicious one, with our full resolution camera, smart Dashboard, and c++ so the field csa came to stand behind our station and made us very nervous by joking that we look like the problem. But no problems happened in the replay of the match and nothing happened for my team prior to or after that match.

[The Lucas]

Originally Posted by Libby K

THIS THIS THIS THIS THIS.

Thank you.

Stop screaming at field resetters/queuers when your robot doesn't move. Whether it's a comm issue, a robot issue, a field issue- whatever. Only the FTA (navy blue polo with 'FTA' on the back) and FTAAs (yellow vest with 'FTAA 'on the back) can help with those sort of issues. Approach them calmly with your notes on the observed issue and they should do their best to help you out.

Also, the Control System Advisors who often wear an orange hat and help setup WPA keys can be an invaluable source of in pit help (since the FTA cannot leave the field for long). CSA can talk with the FTA to get techincal details about your issue and actually check/fix your control system. They can relay info back and forth to the FTA so they can help diagnose in future matches.

The field and robots at week1 Hatboro had a lot of issues on Fri and Sat. The 2 FTAs and FTAAs worked tirelessly to resolve these issues. They proactively sent 2 CSA and me (when I had time away from RI duties) to pits of teams identified as having issues. Issues where diagnosed and Sun seemed to pretty smoothly.

I have been officially or unofficially helping resolve these issues for years now. In my observation, it is almost always a robot issue. I think the Pareto principle applies, where 20% of the robots cause 80% of the issues. Of course the field, comms, fireware and libraries all can have thier issues, most notably the mandatory C++ update released during week 1. If something like that is possibly the issue, the FTA are eager find it and share it with FRC engineering if necessary.

Although your robot may have issues, it only becomes your fault if you don't work responsibly to resolve the issues. Get as much info as you can from the FTA. If you don't have the experience to troubleshoot it yourself, please seek help from someone. Even if you are experienced, a second set of eyes can find those things you glance over without really thinking.

[marccenter]

Dear CD,
I believe the communication problems we were having on the field at the Michigan Waterford District were solved by eliminating unnecessary Dashboard variables and upgrading the 09 classmate to a newer Laptop for the FRC Dashboard component used with the Joysticks.
Here's what happened and my solution. Our student driver complained of lagging on the field during the last minute of operation. Upon finishing the match I took the robot (with same battery) to the practice field, tethered the system, and had the other student driver drive until the battery was dead and the system worked fine. I began to think perhaps a radio issue, however, the radio is mounted in a good position so I didn't think that was a good path to go down.
I went home for the evening and remembered that when I was experimenting with potentiometers at home with the Dashboard and the 09 classmate that the POT readings tracked closely upon start up but after a minute or so of operation would track poorly. I checked the 09 classmate CPU and it was like 75% utilization. I just made a mental note at this time (at home). But I decided this was the path to follow.
Saturday morning I talked to the Field FTA and he confirmed that he has saw long "trip times" on our robot on the field the previous day (why didn't he tell me?). I think he quoted 25 msec. I told him I would be changing Laptops for the dashboard and would like to be updated on my communication packet trip times.

Now, because I wanted my best operation on the field. I also changed the Dashboard program to eliminate four dashboard variables I did not need. So I changed both the FRC dashboard laptop and my program. I had the same driver drive in the morning and she reported smooth operation. The Field FTA later quoted that my packet trip times were down to 4 or 5 msec which was "good".

I am not using a camera (disabled) and the software I am using is "solid".

So my recommendation to teams is to certainly disable the camera if you are not using it, eliminate any Smart Dashboard parameters that you aren't using for the field, and update your FRC Dashboard laptop computer (faster CPU/more RAM?) for the Field.

I hope this helps you.. Please comment if you think this solution may have helped you so we can continue to make a better season for everyone.

PS: Talk to the CSA & FTA field folks if you are having what you think is a field communication issue. I am also adding that the possibility of a programming issue is always present in this competition environment.

[Mark McLeod]

As an addon to Marc's comments, all team programmers should get in the habit of checking their Driver Station log for each match when the robot comes back from the playing field.
It contains a wealth of information recorded 50 times a second for the entire match:

• Communication packet trip time
• Lost packet count
• Robot battery voltage
• cRIO CPU utilization %
• Markers for lost comms (and most importantly, the duration of lost comms: ~25 sec = radio reboot, ~40 sec=cRIO reboot)
• When Auto/Teleop occurred

Start -> All Programs -> FRC Driver station Log Viewer

Also, get in the habit of checking the built-in Window Task Manager or Resource Manager tools (Cntl-Alt-Del) for:
- PC CPU %
- Memory %
- Networking bandwidth

[tzjin]

At Lone Star, there were a few matches with lag and comm issues. The FTA asked us, 1477 and possibly others to turn down our camera resolution/framerate, citing an overloaded bandwidth as the cause of our problems. As a result, we turned down our Axis camera feed from approximately 5mb/s to 3mb/s. Our programming lead was a bit confused, as we though the bandwidth was capped at 7mb/s. Anyways, we replayed two of our matches, still losing one and winning the other, when the other team complained of lag or lack of comm.

Interestingly, we experienced jerkiness during these matches, but were never as severely affected as other teams, some of whom did not move. Also, in eliminations, our alliance captain lost communication multiple times. Luckily, we were able to narrowly upset the second alliance, but were defeated in semi-finals by a single autonomous frisbee in both matches.

[codes02]

At ORPO (Autodesk Oregon Regional in Portland), we replayed one match (match 39) because of communication issues (all robots were showing disconnects approximately half way through telop). During the replay the same issue was seen. Teams were then asked to decrease camera resolution or turn off cameras. After a third try, the match completed without issue.

[Nuttyman54]

Originally Posted by tzjin

At Lone Star, there were a few matches with lag and comm issues. The FTA asked us, 1477 and possibly others to turn down our camera resolution/framerate, citing an overloaded bandwidth as the cause of our problems. As a result, we turned down our Axis camera feed from approximately 5mb/s to 3mb/s. Our programming lead was a bit confused, as we though the bandwidth was capped at 7mb/s. Anyways, we replayed two of our matches, still losing one and winning the other, when the other team complained of lag or lack of comm.

Interestingly, we experienced jerkiness during these matches, but were never as severely affected as other teams, some of whom did not move. Also, in eliminations, our alliance captain lost communication multiple times. Luckily, we were able to narrowly upset the second alliance, but were defeated in semi-finals by a single autonomous frisbee in both matches.

The bandwidth limit is published as 7mb/s, but that's for all communications for each robot. It's possible that with the camera at 5mb/s, the rest of the robot communications was putting it close to 7mb/s.

When you say that other teams were more severely affected, are you implying that other teams in the same match as you (including ones without cameras) were having more severe lag/comms problems, and that lowering your bandwidth improved their trip times?

[Alan Anderson]

One thing I've heard from the people looking into this sort of problem is that sometimes multiple robot bridges might connect to the field access point with just the right timing to cause the wireless network to drop down to a lower communication rate. If that's the case, then the available bandwidth will be significantly lower than it ought to be.

When the status displays available to the FTA show symptoms of this, shutting down all the robots and powering them back up one at a time brings things back to normal. It seems to be happening often enough that it can eventually be repeated under controlled conditions, and I expect that the problem will be understood and resolved.

[MamaSpoldi]

At the BAE regional, we were very lucky to be contacted by the FTA after a few practice matches on Thursday because they had noted that our packet latency was very high... on the order of 10x the rest of the robots on the field. We turned off a few items on our dashboard (although there weren't a huge amount on there to begin with) but were asked to come to the field during the link testing after all practice matches were completed. (The link test is for teams which have not made it to any practice matches to verify that they can connect to FMS successfully before the qualification rounds begin). During that testing time it was determined that the initial problem with our communications was in fact a bad D-Link. When it was replaced by one from spare parts, we were good to go.

However, the FTAs continued to monitor our comms through our beginning matches on Friday... and at one point sent over a volunteer to tell us that we might be experiencing some additional communications issues and we should verify that our camera frame rate was no more than 15 fps. When we checked this we noticed that we had never touched that parameter, we had only configured the resolution to be smaller. Upon turning down the frame rate we ceased to have any comm issues observed by us or the FTAs.

[ykarkason]

The Israel regional replayed somthing like 5 qualifications matches becuase of network problems.

Our team personally was asked to lower the camera's resolution and frame rate and after that all of our matches worked great, but I'm not sure if it was this that solved the problems for us or if it was a field thing...

[MikeE]

Originally Posted by Mark McLeod

As an addon to Marc's comments, all team programmers should get in the habit of checking their Driver Station log for each match when the robot comes back from the playing field.
It contains a wealth of information recorded 50 times a second for the entire match:

• Communication packet trip time
• Lost packet count
• Robot battery voltage
• cRIO CPU utilization %
• Markers for lost comms (and most importantly, the duration of lost comms: ~25 sec = radio reboot, ~40 sec=cRIO reboot)
• When Auto/Teleop occurred

We competed in the Week 1 Palmetto Regional and by Friday afternoon the field staff were very busy diagnosing the networking problems that eventually resulted in the C++ update.
I rechecked our driver station logs yesterday and most matches were as expected, except for Q44 which showed quite spectacular activity with lost comms to the robot every couple of seconds. (We were not using the latest DS version so the packet loss/latency graph is missing.)

But this post isn't to complain about that match since both alliances were operating under the same conditions, and the probable cause was fixed during Week 1.

I just wanted to illustrate that we now have some excellent tools for teams to identify problems themselves.
It is easy to blame robot problems on "the field" but in my experience it is almost always a problem with the robot, and teams are now in a far better position to diagnose and solve those problems, perhaps with the help of a friendly CSA.

So a huge thank you for the DS logging tool.