2012 Field Comm. Issue Logs

[Tristan Lall]

Originally Posted by Greg McKaskle

Does the robot connect but not stay connected, or does never connect to or through the field? Are the cabled tests in the pits practice matches? what rev radio and what FW version?

Cabled tests were in the pits. It never worked with field FMS.

Radio was initially hardware A1 with 1.4; swapped with a loaner with hardware A1 and 1.21. (When I said "reflashed", I should have specified that the WPA key was updated using the kiosk; neither radio had new firmware installed.)

We verified the WPA key in the FMS and on the robot radio, and manually re-entered it. It was correct initially, and re-saving it had no new effect.

As for the build/deploy, we independently confirmed what was mentioned above: these symptoms don't match that fault. (And we made sure the code was deployed when reloading it.)

Further attempts were made: a third laptop (known good) as the DS had no effect, except under one very specific condition. The initial DS laptop (some sort of ordinary Windows PC) was connected to the field at the driver station via Ethernet, and the DS VI was started. (It indicated FMS was found.) The known good laptop was connected directly to the robot radio via Ethernet, and various software was loaded (potentially including another instance of the DS VI). At this point, the communications and robot code lights on the driver station DS VI flashed on and off intermittently, for a couple seconds at a time. Nobody had any firm idea what was going on there.

[dkearle]

Originally Posted by Greg McKaskle

I'm glad you were able to capture this. I wasn't able to post last night, but the logs you posted seemed to indicate a crash. If the battery trace doesn't go away, the robot it connected to the field. The CPU goes low because the thread the code was in was terminated. Hopefully you can use practice matches and some code walkthroughs or some clever debugging to track it down.

Greg McKaskle

After removing the SmartDashboard we (Team 1280) performed beautifully for our next seven matches at Silicon Valley. We did not have any further crashes, either on the field or tethered in the pits.

GRT (Team 192) did get their problem resolved at Silicon Valley as well. Their symptoms were different from ours. The report I heard was they had an IP address conflict - they had an on-board laptop with the same IP address as their Axis camera. This is 2nd-hand information but I heard it from a very reliable source.

One note for those looking at their radio firmware as a potential root cause. We ran 1.21 in Silicon Valley and continued to have our issues. As Greg said, we continued to log our battery voltage and CPU usage from the robot on the driver station so communication with our robot through the field was not our issue. It wasn't until we removed the SmartDashboard that we ran consistently.

Silicon Valley is our last competition for the season so we won't have any additional information to provide this year on our performance. I hope everyone else experiencing issues is able to get theirs identified and resolved! Best of luck to the teams still competing and those headed to St. Louis!

[farmersvilleRob]

We were having lag issues at the Dallas East Regional. Basically, since our shooter was so dependent on precise timing, any delay would cause the shot to be off. We determined there was probably a delay due to encryption and packet sniffing (to find battery voltage). We also assume there was interference because there typically is with so much wireless accessing going on.

So whenever we were tuning in the practice field hard tethered, we shot upwards of 60 feet accurately, but when connecting to the field and running a match, we never got an accurate shot. We eventually did by guessing, and that was 50 degrees off of our practice shots.

[tzjin]

As noted above, we had some comm issues due to the onboard computer having the IP address of an Axiss camera. However, we had no issues Thursday, and the problem only surfaced on Friday. In addition, after 971 and 100 helped us locate the problem, we were never able to turn on tracking (we're pretty sure the computer as well) again.

Unfortunately, there is no way to test our connection status until Championships, as the robot operates perfectly tethered and from a wireless router.

[Tom Line]

At the Troy regional, we experienced our first bout with the 'lag monster'. For two painful matches, we were seeing the robot lag significantly - the driver and gunner both said they would try to make a motion and the robot would wait a few moments, then (sometimes) make the motion, then it would sometimes continue that motion well after they had stopped. As we neared the end of one match, we completely lost control of the robot.

The first thing we checked was CPU usage of the Crio. Rock solid at 70%. During tethered operation, we didn't see the same problems. Only on FMS.

Then, while tethered in the pits troubleshooting the issue, I clicked over to the charts tab our our driver station (09 Classmate model). We were seeing significant packet loss and round-trip-times off the chart.

This is with the standard driver station and dashboard. No modifications.

Initially I thought it was a bad tether cable, so we swapped it with no effect. Then, I hooked up our backup driver station (our top end programming laptop) and the lag and round trip issues completely went away.

Plugged back into to our classmate, and the problems came back instantly. Keep in mind however, while tethered, the problems were still not exhibiting themselves as lag. I suspect the added delay of traveling through the FMS system is what pushed it over the edge into creating a problem.

The classmate is 100% stock, completely clean and reimaged with nothing extra on it.

We won't be using the classmate anymore. The lag was gone using our higher end laptop in the next match.

[DonRotolo]

Originally Posted by DominickC

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor?

As an update to my last post in this thread, we duplicated the issue in the lab and found the driver station CPU maxing out. We optimized that down to maybe 50%, and at Mt Olive this past weekend we experienced no further issues.

So for us, problem solved.

[mdrouillard]

Which event? Greater Toronto WEST
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? Yes
Did you have the radio mounted near motors/large metal structure? no
Using classmate (or similarly slow computer) as driver station? no

On practice day the field was refusing connects from our team and many veteran teams. The DS laptop would connect, then immediately loose connection to the field. After much trying (4 hours) and approx 12 teams of the 50 there could not connect, it was discovered that radios at version 1.4 were the common thread. We had our radio which worked flawlessly at 1.4 in Knoxville, had it reimaged to 1.2. After which we could connect but images from the camera and control response was definitely laggy'er than at the previous competition. FIRST field personel did everything to have buiding wifi networks shutdown, (on thursday there were over a dozen non-First networks in range) and after warning the crowd to turn off their mobile wifi things got a bit better. But our DS logs do show incredible packet loss on practice thursday. The field in GTR West was the same field used earlier at GTR EAST (which we did not participate in) and First I suspect are still scratching their heads on this one. I wonder if there is any software/hardware version differences of importance between the field used at Knoxville TN and the GTR West? If other fields are having difficulty with 1.4 firmware and how this compares to the configuration of TN?

[basicxman]

Originally Posted by mdrouillard

FIRST field personel did everything to have buiding wifi networks shutdown, (on thursday there were over a dozen non-First networks in range) and after warning the crowd to turn off their mobile wifi things got a bit better.

Members of 2200 helped find and turn off these APs as well, it seemed to help a little but certainly didn't fix the problem. Went from two or three robots running simultaneously to four. Many robots were pulled off the field so practice matches could keep occurring. The downgrade to 1.21 seemed to do the trick for quals+elims.