2012 Field Comm. Issue Logs

[DominickC]

I won't be able to get any FW info to you until Wednesday at the earliest, but I can tell the drive team to pop a comment in here with a full account.

[Navid Shafa]

Same situation just happened in Seattle Olympic, stay tuned for official post on the issues that happened during the finals.

Problems included: Response time lag from controls to robot through field management and lost camera relay for two robots on one of the alliances.

* I am not there, just talking to people who just had these issues, I'll let them clarify shortly.

[dsalvucci]

As Dom has requested, this is Team 23's recap of what happened to our driving situation.

Autonomous started and everything went exactly as planned. No problems whatsoever, everything checked out functional.

Teleop started and the team reached for the controls. The driver of the robot then went to turn the robot around and the robot began spinning uncontrollably. The drive team did initially have control during teleop for a couple seconds at most before the robot began to spin out of control (it appeared to be repeating the command to turn). The moment we lost control, the camera feed on the driver station had frozen and the joysticks were not being responsive.

A field tech came over and advised us to reboot the cRIO. I am not sure if the robot was spinning straight through to the point of rebooting or not. I think it was, but am not entirely sure. Upon rebooting, the joysticks became responsive again, however by the time robot code had been sent and communication was recovered, the match had only 1 second left.

That was the more physical memory of the situation we went through during our last elimination match. If you're looking for any messages or code problems, refer back to DominickC instead of myself as i was merely driving.

[Timz3082]

Which event? 10,000 Lakes
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? No
Using classmate (or similarly slow computer) as driver station? Yes

During our last Qualification match, we were all running in the beginning, then two robots on our alliance dropped at the same time. 30 seconds later we dropped connection causing our alliance to loose the match! We talked with the staff and they blamed it on each teams code, which makes no sense since they all were working in every other match without any code change. The languages were different to, one was labview, the other, C++. Is this just a fluke with our robots (only match we dropped to) or would it have been some sort of field problem?

[thephpdev]

We're not sure what the problem was exactly for our robot during competition (Team 2502) and we didn't get to find the exact reason it didn't work, but we did fix it.

First, the radio was set to auto rather than bridge. For competition it needs to be on bridge, but it seems that for other testing (such as in the pits) the radio should be on auto.

Second, the radio was right next to our drive train motor and clumped around a lot of wires. So we moved that up farther so that it could be seen from over the ramp.

Last change, we made a slight hack to allow faster operator control loops. Which meant only putting drive train in the operator control loop, but creating a new thread to handle everything that would have been in operator control.

Note on the last change, our graph before this change was showing very long loop time (green line) and after the change it decreased drastically (of course). I'm not sure which of these things could be the issue, but I would double check these.

[Tristan Lall]

Originally Posted by Greg McKaskle

Does the robot connect but not stay connected, or does never connect to or through the field? Are the cabled tests in the pits practice matches? what rev radio and what FW version?

Cabled tests were in the pits. It never worked with field FMS.

Radio was initially hardware A1 with 1.4; swapped with a loaner with hardware A1 and 1.21. (When I said "reflashed", I should have specified that the WPA key was updated using the kiosk; neither radio had new firmware installed.)

We verified the WPA key in the FMS and on the robot radio, and manually re-entered it. It was correct initially, and re-saving it had no new effect.

As for the build/deploy, we independently confirmed what was mentioned above: these symptoms don't match that fault. (And we made sure the code was deployed when reloading it.)

Further attempts were made: a third laptop (known good) as the DS had no effect, except under one very specific condition. The initial DS laptop (some sort of ordinary Windows PC) was connected to the field at the driver station via Ethernet, and the DS VI was started. (It indicated FMS was found.) The known good laptop was connected directly to the robot radio via Ethernet, and various software was loaded (potentially including another instance of the DS VI). At this point, the communications and robot code lights on the driver station DS VI flashed on and off intermittently, for a couple seconds at a time. Nobody had any firm idea what was going on there.

[slijin]

I can say with certainty that we ran into lag issues with computer CPU consumption by the SmartDashboard; keeping it closed solved apparent lag problems, although the NYC FTA (Mark Mcleod) noted that we retained high trip times (~175 ms, as opposed to a standard 5-6 ms) throughout the entire event.

• Which event? NYC
• Wireless bridge radio HW revision? (A1, A2, or B1) A1
• Radio firmware rev? 1.21
• Programming language? Java
• Using a dashboard app? n/a
• Using vision with Axis Camera and cRIO processing? Ran a live stream with no processing at default fps
• Using vision with driver station processing? DS display; nothing else.
• Did you have the radio mounted near motors/large metal structure? Near two Black Jaguars, above the 12V/5V converter*.
• Using classmate as driver station? 2go E11 Classmate, not the KoP one
• 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4-slot
• CAN Jaguars? What FW version? Yes, rev92.

*This issue is something I realized post-NY. As Alan has pointed out, this could possibly produce issues; it will be fixed when we go to CT.

[DonRotolo]

Some data points:

At Rutgers to a small extent and at NYC to a much greater extent, we were seeing the robot stop responding for a second to several seconds, somewhat randomly. This would occur both tethered and via radio (!)

Reviewing the logs from NYC we found our latency was around 170 mSec ('good' would be <10 mSec).

Our 8-slot cRio showed a CPU usage of 50-60%, which is reasonable.

Carefully working to recreate the issue on our prototype robot, we ultimately found that out driver station CPU was maxing out.

We have not yet deemed this issue resolved, but we believe we're almost there. We'll find out when we unbag this week in anticipation of Mount Olive.

(I don't have the details requested in the OP, but can get them if that will help. All our cRios are 8-slot, we use LabView, and our radios are all Rev A).

[Greg McKaskle]

Reviewing logs sent to me from Bayou, I'm seeing a good number of glitches caused by the DS laptop. They don't necessarily cause lag, but I see periodic spikes in trip times and sometimes in lost packets. I've accidentally reproduced this myself by having IE open on the laptop. I also saw teams, with no dropouts or issues, that had 100% CPU usage continuously -- auto, tele, and disabled. I'd like to encourage teams to use the charts tabs and practice match built into the DS to look at their memory usage, CPU usage, and any repeatable spikes in trip time. I was able to open the task manager and see the odd Windows program jumping up in CPU from time to time.

While it is not a requirement to make a kiosk account to run the DS, if you see spikes, see if stopping other apps or killing other processes will get rid of them. And I believe directions for making a kiosk are available from FIRST.

Greg McKaskle

[Joe Ross]

Originally Posted by slijin

[*]CAN Jaguars? What FW version? Yes, rev92.

While probably not the cause of your issues, [R61] requires version 99 or higher. The current version is 101.

[LukeS]

We experienced what were usually short blips where we could not control the robot, and it simply sat there. The FTAs would comment that they usually saw the comm blip, but we never saw comm drop on our driver station. One of the FTAs (Jerry) inspected the logs from our driver station, During the time that we lost control, the battery reported constant voltage, confirming that nothing was running on our robot. Nothing in the log looked "wrong." When we regained control, the ping time would spike.

We suspected that it was a bad USB cable connecting our OI, and that we really did stop giving the robot commands. However, we replaced the USB cable, and the issue persisted.

- Team 1024
- Which event? Boilermaker
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? Java
- Using a dashboard app? SmartDashboard
- Using vision with Axis Camera and cRIO processing? No
- Using vision with driver station processing? No We would, but left the camera power unplugged for fear of it causing more comm issues.
- Did you have the radio mounted near motors/large metal structure? Mounted on a metal panel near metal tubes and near 2 Jaguars
- Using classmate as driver station? No, an HP 6530b
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? All version 101

[plnyyanks]

At CT today, things seemed pretty good. I don't think I saw too many connection problems (but then again, I wasn't hanging around the field much). For us today, we were only dropped by the field in one early match this morning. I had a long talk with the FTA who was pretty helpful, and we decided it was an isolated indecent - we made it the rest of the day with no issues at all, so I think we'll be okay. But I'll fill this out and attach a screenshot of the log viewer just for the sake of another data point.

- Which event? CT
- Wireless bridge radio HW revision? (A1, A2, or B1) A1
- Radio firmware rev? (1.21, 1.4, or maybe something else) 1.21
- Programming language? LabVIEW
- Using a dashboard app? Labview, SmartDashboard, or custom? custom LV dash
- Using vision with Axis Camera and cRIO processing? yes
- Using vision with driver station processing? no
- Did you have the radio mounted near motors/large metal structure? no
- Using classmate (or similarly slow computer) as driver station? yes
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? no

[mdrouillard]

Which event? Greater Toronto WEST
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? Yes
Did you have the radio mounted near motors/large metal structure? no
Using classmate (or similarly slow computer) as driver station? no

On practice day the field was refusing connects from our team and many veteran teams. The DS laptop would connect, then immediately loose connection to the field. After much trying (4 hours) and approx 12 teams of the 50 there could not connect, it was discovered that radios at version 1.4 were the common thread. We had our radio which worked flawlessly at 1.4 in Knoxville, had it reimaged to 1.2. After which we could connect but images from the camera and control response was definitely laggy'er than at the previous competition. FIRST field personel did everything to have buiding wifi networks shutdown, (on thursday there were over a dozen non-First networks in range) and after warning the crowd to turn off their mobile wifi things got a bit better. But our DS logs do show incredible packet loss on practice thursday. The field in GTR West was the same field used earlier at GTR EAST (which we did not participate in) and First I suspect are still scratching their heads on this one. I wonder if there is any software/hardware version differences of importance between the field used at Knoxville TN and the GTR West? If other fields are having difficulty with 1.4 firmware and how this compares to the configuration of TN?

[basicxman]

Originally Posted by mdrouillard

FIRST field personel did everything to have buiding wifi networks shutdown, (on thursday there were over a dozen non-First networks in range) and after warning the crowd to turn off their mobile wifi things got a bit better.

Members of 2200 helped find and turn off these APs as well, it seemed to help a little but certainly didn't fix the problem. Went from two or three robots running simultaneously to four. Many robots were pulled off the field so practice matches could keep occurring. The downgrade to 1.21 seemed to do the trick for quals+elims.