2012 Field Comm. Issue Logs

[Greg McKaskle]

Reviewing logs sent to me from Bayou, I'm seeing a good number of glitches caused by the DS laptop. They don't necessarily cause lag, but I see periodic spikes in trip times and sometimes in lost packets. I've accidentally reproduced this myself by having IE open on the laptop. I also saw teams, with no dropouts or issues, that had 100% CPU usage continuously -- auto, tele, and disabled. I'd like to encourage teams to use the charts tabs and practice match built into the DS to look at their memory usage, CPU usage, and any repeatable spikes in trip time. I was able to open the task manager and see the odd Windows program jumping up in CPU from time to time.

While it is not a requirement to make a kiosk account to run the DS, if you see spikes, see if stopping other apps or killing other processes will get rid of them. And I believe directions for making a kiosk are available from FIRST.

Greg McKaskle

[Madison]

I don't have a lot of the information you'd like to see here since I am not a control systems person by any means. I am, however, the drive coach and so I see the problems we have first hand.

We ran throughout the Alamo regional without any trouble. We ran throughout the qualifying matches of the Seattle Olympic without any trouble.

During the elimination rounds, we experienced severe lag in several matches and lost the video from our camera in at least one match. Several others experienced similar issues including loss of the camera feed, both on our alliance and on others.

The common sense that FIRST is so fond of suggests that the chance of the same issue plaguing multiple teams at the same time is quite small, but does not, apparently, apply to the field or control system.

I'll try to get someone with a better understanding of how our control system is configured to chime in.

[rwood359]

Based on what we saw in Hawaii, I'm going to ask the Houston Inspection crew to check the firmware version. It takes about 4 minutes to download the firmware. If Inspection thinks it would take to long to download any 1.4 bridged there, I'll go to the pits and do it. It may not help, but I don't think that it will hurt.

[mandrews281]

We had comm issues at Palmetto and here's what happened.

1. Started out OK.
2. We turned on our camera software (image processing on the netbook DS). Ping times reported from the FMS tech were spiking at >1sec!!!!! (no data on CPU usage on netbook). The field techs down graded our radio firmware.
3. Tried again with the redone radio. Ping times still >1sec.
4. Turned off camera processing on netbook DS. Everything OK.
5. Setup a laptop to replace the netbook as the DS.
6. We turned on the camera processing with a notebook. Everything OK from both our and the field techs point of view. Notebook CPU < 35%.

Our experience is that many communication issues are probably tied to CPU usage on the netbook or cRIO. This was mentioned in one of the team updates.

On a related note, during eliminations, we started setting up the notebook at the driver station and connected to the FMS. Then we spotted that we only had 10min of battery life left and decided to do a quick swap to the netbook to avoid problems in the match. Sop we unplugged the joysticks, enhancedIO and network from notebook and plugged it into the netbook. DON'T EVER DO THIS !!! Once you connect to the FMS you should consider yourself committed to that DS computer. Our netbook didn't recognize the joysticks soon enough and we couldn't drive the robot during the match. Functions on the EnhancedIO worked (shooter). Nearly cost our alliance the quarter finals.

[DominickC]

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor? We were using my personal computer as our drivers station due to our dedicated computer biting the dust...

2011 MacBook Pro 13''
2.4GHz Intel i5
8gb DDR3
Parallels x64 Windows 7 running on Mac OS X Snow Leopard.

[LukeS]

We experienced what were usually short blips where we could not control the robot, and it simply sat there. The FTAs would comment that they usually saw the comm blip, but we never saw comm drop on our driver station. One of the FTAs (Jerry) inspected the logs from our driver station, During the time that we lost control, the battery reported constant voltage, confirming that nothing was running on our robot. Nothing in the log looked "wrong." When we regained control, the ping time would spike.

We suspected that it was a bad USB cable connecting our OI, and that we really did stop giving the robot commands. However, we replaced the USB cable, and the issue persisted.

- Team 1024
- Which event? Boilermaker
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? Java
- Using a dashboard app? SmartDashboard
- Using vision with Axis Camera and cRIO processing? No
- Using vision with driver station processing? No We would, but left the camera power unplugged for fear of it causing more comm issues.
- Did you have the radio mounted near motors/large metal structure? Mounted on a metal panel near metal tubes and near 2 Jaguars
- Using classmate as driver station? No, an HP 6530b
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? All version 101

[Joe Ross]

Originally Posted by slijin

[*]CAN Jaguars? What FW version? Yes, rev92.

While probably not the cause of your issues, [R61] requires version 99 or higher. The current version is 101.

[rwood359]

Originally Posted by DominickC

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor?

I guess that I would say contributing factor. We had problems when running 1.4. The processor on our DS is an i7. We do vision processing in the DS, but the usage doesn't go much about 25% - as I remember.

[tuyauxtu]

Team 781 competed at Waterloo this past weekend. We have two Axis 1101 cameras connected to a classmate e09 ds (no image processing). At two matches I watched the field display screen and saw round trip times of 100-plus and accumulated lost packets in the hundreds of thousands (versus below 10/100 for the other teams). The field tech indicated that the two cameras were the cause. We will be switching to a faster laptop for the driver station, and would like to know if there is any way we can test this (e.g. Display trip times and lost packets) on our practice bot/field before we head to Queen City. I'll continue to research this, but I figured you folks would be able to provide specific guidance.

- Team 781
- Which event? Waterloo
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? C++
- Using a dashboard app? Regular Dashboard customized to display both camera images (could we have done this incorrectly such that we have initiated the communications issues ?)
- Using vision with Axis Camera and cRIO processing? 2 x m1101, no vision processing
- Using vision with driver station processing? No
- Did you have the radio mounted near motors/large metal structure? Mounted on a lexan panel near the battery
- Using classmate as driver station? Yes but hopefully not for much longer.
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? No CAN

Thanks
Pat

[Mastonevich]

Originally Posted by rwood359

I guess that I would say contributing factor. We had problems when running 1.4. The processor on our DS is an i7. We do vision processing in the DS, but the usage doesn't go much about 25% - as I remember.

Could you be maxing out one of the i7 cores, assuming it is a 4 core processor? I know we have a beefy driver station without image processing and without the smart dashboard and I did not hear our kids complaining about any lag issues. We were at St. Louis where other issues were reported above.

[RufflesRidge]

Originally Posted by tuyauxtu

We will be switching to a faster laptop for the driver station, and would like to know if there is any way we can test this (e.g. Display trip times and lost packets) on our practice bot/field before we head to Queen City. I'll continue to research this, but I figured you folks would be able to provide specific guidance.

Trip times and lost packets are graphed on the Charts tab of the Driver Station and in the logs viewable with the Driver Station Log Viewer in C:\Program Files\FRC Driver Station.

[rwood359]

Originally Posted by Mastonevich

Could you be maxing out one of the i7 cores, assuming it is a 4 core processor? I know we have a beefy driver station without image processing and without the smart dashboard and I did not hear our kids complaining about any lag issues. We were at St. Louis where other issues were reported above.

It is a quad core. I don't think that we are maxing it out. I'll take a closer look in Houston next week.

[n8many]

Team 1280 here and we had some major problems at the Davis competition. On Thursday we ran fine all day. Once we got into matches though, we had problems.

• Match 8: Ran successfully, had spike of 33 dropped packets early in autonomous mode. No latency issues. Max CPU at 90%
• Match 14: Robot froze 3 seconds into autonomous. No spike in latency or dropped packets. Max CPU at 85%
• Match 18: Ran fine at operator end however there was high latency and lots of dropped packets throughout the match. Peak latency at 50ms, packets at 15, Max CPU at 96% mostly under 90%.
• Match 28: Failed early into teleop. Spike in dropped packets at 58 packets at the start of autonomous, we ran through autonomous fine, but a few seconds into teleop we froze and had spike of 18 dropped packets and our CPU drop from around 72 to around 17 and then go all the way down to idle. Latency hovered around 5ms for the entire match.
• Match 37: Ran fine. Spike of 33 dropped packets at start but CPU averaged around 68 during teleop.
• Match 45: Ran fine. We unplugged the camera and our CPU spiked at over 100% and was oscillating between 45% and 100% at regular intervals from when we powered on the robot. Latency had one spike at 22ms, but other than that it stayed around 5ms. We had regularly had dropped packets but they never went above 10
• Match 51: Did not run autonomous. Had an oscillating CPU usage between idle and 100% at the same interval as Match 45. We think the camera was still unplugged. We only lost 5 packets at the start but had latency spike of 25ms.
  
  Saturday things went worse.
• Match 65: We did not run. 41 lost packets at the start of autonomous coupled with a latency spike of 17ms. Our CPU usage did not go above 10 before it stopped.
• Match 70: We did not run again. No dropped packets, we had CPU drop to idle after autonomous, latency was pretty consistent at 5ms.
• Match 77: We ran fine. There was a 38 dropped packet spike at the start of autonomous but latency stayed around 5ms. CPU usage stayed around 80%
• Match 84: We did not run. We had 98 dropped packets just after the start of autonomous and after the last time our robot said it was in autonomous. Latency stayed at 5 while we lost the packets. CPU never went above 15% before we lost communication.

We made minor code changes throughout the weekend. As you can see, our problems got worse as the competition went on and there was no consistent pattern to it. The only consistent thing about the loss in communication was that our robot mode line disappears from the log charts when we lose communication, while the driver station mode line stays. We are not sure if this is a symptom or a cause. We are getting complete loss in communication, rather than temporary loss of communication. We are hopefully going to find out more on Thursday in San Jose, but for now any help would be great.


- Team 1280
- Which event? Sacramento/Davis
- Wireless bridge radio HW revision? (A1, A2, or B1) A1
- Radio firmware rev? 1.40
- Programming language? C++
- Using a dashboard app? Currently using SmartDashboard for display statements using log function. FRC Dashboard is on the side for the camera feed.
- Using vision with Axis Camera and cRIO processing? 1 Camera with vision processing on 8 Slot cRIO. Resolution at 320x240 @ 7fps
- Using vision with driver station processing? No, but we do have a camera feed (FRC Dashboard)
- Did you have the radio mounted near motors/large metal structure? Mounted at top of robot near shooting motor.
- Using classmate as driver station? No, using 6730b HP laptop (Win7)
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 8 slot
- CAN Jaguars? What FW version? No Jaguars

[Greg McKaskle]

A couple things that may help with reading the logs.

1. A spike at the beginning of auto is an glitch in the measurement. The DS was modified to zero the lost packets when the match begins. As a result, the delta loss reported into the logs will sometimes show a high spike at the very beginning of auto. You cannot trust this point.

2. If the communication with the robot fails, the voltage will disappear, and other lines such as CPU should be flat and cached. Lost packets and trip time will often look high for a time when a drop occurs.

3. The lost packets numbers is typically "out of 25". So 10 lost packets over half second means 10 out of 25 were lost. If communications is lost entirely, the timeout is 1 second, so the number may go above 25 and be "out of 50".

4. When packets are lost, it is common for CPU to drop. The code for tele and disabled are typically waiting for new DS packet, so less to do when less packets comes in.

5. If the DS line suddenly shows a disabled line, but the robot line seems to ignore it, this indicates a watchdog due to communications drop. This means the robot outputs were disabled due to more than 5 consecutive packets not arriving. Note that it is possible for occasional packet arrival to keep the robot enabled, and this is typical.

6. If you see periodic spikes in the trip time and/or the lost packets, this may very well be due to the DS. If you observe these when cabled, that is supporting evidence that something on the DS is a likely cause. Opening the Task Manager may show a process bumping to the top in time with the blip.

7. Trip time doesn't necessarily mean latency. The trip time includes the trip to the robot and back again. If delayed to the robot, it impacts driving. If delayed on the robot or on the return trip, or often within the DS laptop, the trip delay does't imply that the robot is hard to drive due to lag.

8. Similarly, lost packets may make it to the robot and keep the watchdog alive and be lost on the return trip.

Greg McKaskle

[tcjinaz]

So have we arrived at the point where we need an RTOS capable system in the Driver's Station?
The piles of threads available in a Core i7 are not exactly conducive to deterministic response times.

Shoot, let's push it out a little. Is 802.11 good enough?