2012 Field Comm. Issue Logs

[DominickC]

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor? We were using my personal computer as our drivers station due to our dedicated computer biting the dust...

2011 MacBook Pro 13''
2.4GHz Intel i5
8gb DDR3
Parallels x64 Windows 7 running on Mac OS X Snow Leopard.

[rwood359]

Originally Posted by DominickC

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor?

I guess that I would say contributing factor. We had problems when running 1.4. The processor on our DS is an i7. We do vision processing in the DS, but the usage doesn't go much about 25% - as I remember.

[tuyauxtu]

Team 781 competed at Waterloo this past weekend. We have two Axis 1101 cameras connected to a classmate e09 ds (no image processing). At two matches I watched the field display screen and saw round trip times of 100-plus and accumulated lost packets in the hundreds of thousands (versus below 10/100 for the other teams). The field tech indicated that the two cameras were the cause. We will be switching to a faster laptop for the driver station, and would like to know if there is any way we can test this (e.g. Display trip times and lost packets) on our practice bot/field before we head to Queen City. I'll continue to research this, but I figured you folks would be able to provide specific guidance.

- Team 781
- Which event? Waterloo
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? C++
- Using a dashboard app? Regular Dashboard customized to display both camera images (could we have done this incorrectly such that we have initiated the communications issues ?)
- Using vision with Axis Camera and cRIO processing? 2 x m1101, no vision processing
- Using vision with driver station processing? No
- Did you have the radio mounted near motors/large metal structure? Mounted on a lexan panel near the battery
- Using classmate as driver station? Yes but hopefully not for much longer.
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? No CAN

Thanks
Pat

[RufflesRidge]

Originally Posted by tuyauxtu

We will be switching to a faster laptop for the driver station, and would like to know if there is any way we can test this (e.g. Display trip times and lost packets) on our practice bot/field before we head to Queen City. I'll continue to research this, but I figured you folks would be able to provide specific guidance.

Trip times and lost packets are graphed on the Charts tab of the Driver Station and in the logs viewable with the Driver Station Log Viewer in C:\Program Files\FRC Driver Station.

[Mastonevich]

Originally Posted by rwood359

I guess that I would say contributing factor. We had problems when running 1.4. The processor on our DS is an i7. We do vision processing in the DS, but the usage doesn't go much about 25% - as I remember.

Could you be maxing out one of the i7 cores, assuming it is a 4 core processor? I know we have a beefy driver station without image processing and without the smart dashboard and I did not hear our kids complaining about any lag issues. We were at St. Louis where other issues were reported above.

[rwood359]

Originally Posted by Mastonevich

Could you be maxing out one of the i7 cores, assuming it is a 4 core processor? I know we have a beefy driver station without image processing and without the smart dashboard and I did not hear our kids complaining about any lag issues. We were at St. Louis where other issues were reported above.

It is a quad core. I don't think that we are maxing it out. I'll take a closer look in Houston next week.

[n8many]

Team 1280 here and we had some major problems at the Davis competition. On Thursday we ran fine all day. Once we got into matches though, we had problems.

• Match 8: Ran successfully, had spike of 33 dropped packets early in autonomous mode. No latency issues. Max CPU at 90%
• Match 14: Robot froze 3 seconds into autonomous. No spike in latency or dropped packets. Max CPU at 85%
• Match 18: Ran fine at operator end however there was high latency and lots of dropped packets throughout the match. Peak latency at 50ms, packets at 15, Max CPU at 96% mostly under 90%.
• Match 28: Failed early into teleop. Spike in dropped packets at 58 packets at the start of autonomous, we ran through autonomous fine, but a few seconds into teleop we froze and had spike of 18 dropped packets and our CPU drop from around 72 to around 17 and then go all the way down to idle. Latency hovered around 5ms for the entire match.
• Match 37: Ran fine. Spike of 33 dropped packets at start but CPU averaged around 68 during teleop.
• Match 45: Ran fine. We unplugged the camera and our CPU spiked at over 100% and was oscillating between 45% and 100% at regular intervals from when we powered on the robot. Latency had one spike at 22ms, but other than that it stayed around 5ms. We had regularly had dropped packets but they never went above 10
• Match 51: Did not run autonomous. Had an oscillating CPU usage between idle and 100% at the same interval as Match 45. We think the camera was still unplugged. We only lost 5 packets at the start but had latency spike of 25ms.
  
  Saturday things went worse.
• Match 65: We did not run. 41 lost packets at the start of autonomous coupled with a latency spike of 17ms. Our CPU usage did not go above 10 before it stopped.
• Match 70: We did not run again. No dropped packets, we had CPU drop to idle after autonomous, latency was pretty consistent at 5ms.
• Match 77: We ran fine. There was a 38 dropped packet spike at the start of autonomous but latency stayed around 5ms. CPU usage stayed around 80%
• Match 84: We did not run. We had 98 dropped packets just after the start of autonomous and after the last time our robot said it was in autonomous. Latency stayed at 5 while we lost the packets. CPU never went above 15% before we lost communication.

We made minor code changes throughout the weekend. As you can see, our problems got worse as the competition went on and there was no consistent pattern to it. The only consistent thing about the loss in communication was that our robot mode line disappears from the log charts when we lose communication, while the driver station mode line stays. We are not sure if this is a symptom or a cause. We are getting complete loss in communication, rather than temporary loss of communication. We are hopefully going to find out more on Thursday in San Jose, but for now any help would be great.


- Team 1280
- Which event? Sacramento/Davis
- Wireless bridge radio HW revision? (A1, A2, or B1) A1
- Radio firmware rev? 1.40
- Programming language? C++
- Using a dashboard app? Currently using SmartDashboard for display statements using log function. FRC Dashboard is on the side for the camera feed.
- Using vision with Axis Camera and cRIO processing? 1 Camera with vision processing on 8 Slot cRIO. Resolution at 320x240 @ 7fps
- Using vision with driver station processing? No, but we do have a camera feed (FRC Dashboard)
- Did you have the radio mounted near motors/large metal structure? Mounted at top of robot near shooting motor.
- Using classmate as driver station? No, using 6730b HP laptop (Win7)
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 8 slot
- CAN Jaguars? What FW version? No Jaguars

[Greg McKaskle]

A couple things that may help with reading the logs.

1. A spike at the beginning of auto is an glitch in the measurement. The DS was modified to zero the lost packets when the match begins. As a result, the delta loss reported into the logs will sometimes show a high spike at the very beginning of auto. You cannot trust this point.

2. If the communication with the robot fails, the voltage will disappear, and other lines such as CPU should be flat and cached. Lost packets and trip time will often look high for a time when a drop occurs.

3. The lost packets numbers is typically "out of 25". So 10 lost packets over half second means 10 out of 25 were lost. If communications is lost entirely, the timeout is 1 second, so the number may go above 25 and be "out of 50".

4. When packets are lost, it is common for CPU to drop. The code for tele and disabled are typically waiting for new DS packet, so less to do when less packets comes in.

5. If the DS line suddenly shows a disabled line, but the robot line seems to ignore it, this indicates a watchdog due to communications drop. This means the robot outputs were disabled due to more than 5 consecutive packets not arriving. Note that it is possible for occasional packet arrival to keep the robot enabled, and this is typical.

6. If you see periodic spikes in the trip time and/or the lost packets, this may very well be due to the DS. If you observe these when cabled, that is supporting evidence that something on the DS is a likely cause. Opening the Task Manager may show a process bumping to the top in time with the blip.

7. Trip time doesn't necessarily mean latency. The trip time includes the trip to the robot and back again. If delayed to the robot, it impacts driving. If delayed on the robot or on the return trip, or often within the DS laptop, the trip delay does't imply that the robot is hard to drive due to lag.

8. Similarly, lost packets may make it to the robot and keep the watchdog alive and be lost on the return trip.

Greg McKaskle

[tcjinaz]

So have we arrived at the point where we need an RTOS capable system in the Driver's Station?
The piles of threads available in a Core i7 are not exactly conducive to deterministic response times.

Shoot, let's push it out a little. Is 802.11 good enough?

[Greg McKaskle]

If it can play video games, it is probably realtime enough to be a DS. The rate of both is defined by human perception and reaction time.

As image processing and other code written by teams migrates to the DS laptop, it will impact the comms. Throw in a Kinect or two, and ...

But at this point, I think some teams have overloaded a component of their robot -- the DS laptop -- not unlike how they are capable of overloading a given circuit or mechanical component. The Task manager alone will give good enough feedback to learn about and avoid the issue.

Greg McKaskle

[slijin]

Originally Posted by slijin

I can say with certainty that we ran into lag issues with computer CPU consumption by the SmartDashboard; keeping it closed solved apparent lag problems, although the NYC FTA (Mark Mcleod) noted that we retained high trip times (~175 ms, as opposed to a standard 5-6 ms) throughout the entire event.

• Which event? NYC
• Wireless bridge radio HW revision? (A1, A2, or B1) A1
• Radio firmware rev? 1.21
• Programming language? Java
• Using a dashboard app? n/a
• Using vision with Axis Camera and cRIO processing? Ran a live stream with no processing at default fps
• Using vision with driver station processing? DS display; nothing else.
• Did you have the radio mounted near motors/large metal structure? Near two Black Jaguars, above the 12V/5V converter*.
• Using classmate as driver station? 2go E11 Classmate, not the KoP one
• 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4-slot
• CAN Jaguars? What FW version? Yes, rev99.

*This issue is something I realized post-NY. As Alan has pointed out, this could possibly produce issues; it will be fixed when we go to CT.

Originally Posted by Joe Ross

While probably not the cause of your issues, [R61] requires version 99 or higher. The current version is 101.

Oops, I made a mistake there. That should be rev99.

Thanks for pointing that out.

[dkearle]

Originally Posted by Greg McKaskle

A couple things that may help with reading the logs.

1. A spike at the beginning of auto is an glitch in the measurement. The DS was modified to zero the lost packets when the match begins. As a result, the delta loss reported into the logs will sometimes show a high spike at the very beginning of auto. You cannot trust this point.

2. If the communication with the robot fails, the voltage will disappear, and other lines such as CPU should be flat and cached. Lost packets and trip time will often look high for a time when a drop occurs.

3. The lost packets numbers is typically "out of 25". So 10 lost packets over half second means 10 out of 25 were lost. If communications is lost entirely, the timeout is 1 second, so the number may go above 25 and be "out of 50".

4. When packets are lost, it is common for CPU to drop. The code for tele and disabled are typically waiting for new DS packet, so less to do when less packets comes in.

5. If the DS line suddenly shows a disabled line, but the robot line seems to ignore it, this indicates a watchdog due to communications drop. This means the robot outputs were disabled due to more than 5 consecutive packets not arriving. Note that it is possible for occasional packet arrival to keep the robot enabled, and this is typical.

6. If you see periodic spikes in the trip time and/or the lost packets, this may very well be due to the DS. If you observe these when cabled, that is supporting evidence that something on the DS is a likely cause. Opening the Task Manager may show a process bumping to the top in time with the blip.

7. Trip time doesn't necessarily mean latency. The trip time includes the trip to the robot and back again. If delayed to the robot, it impacts driving. If delayed on the robot or on the return trip, or often within the DS laptop, the trip delay does't imply that the robot is hard to drive due to lag.

8. Similarly, lost packets may make it to the robot and keep the watchdog alive and be lost on the return trip.

Greg McKaskle

Greg,

I am one of the mentors for Team 1280. Thank you for the guidance on reading the log files you provided. It is helpful to know what is normal.

I've looked carefully at our log files for both our successful matches and our matches where we stopped running. Unfortunately there is no discernible pattern in CPU, dropped packets, or trip time that correlates to when we lose the ability to control our robot on the field.

We do have voltage and CPU readings in all our log files for the duration of the run. It makes sense that if we are getting these readings, we do still have communication with our robot even if we cannot control the robot from the driver station.

For the line on the log files that shows the Robot mode, what does this really tell us? When we run successful matches, the lines that make up the Robot mode clearly show the transition from disabled to autonomous to a short disabled to teleoperated then back to disabled. Occasionally we see a little blip of disabled in the middle of teleoperated but not often. While we see the expected CPU drop when this blip occurs, the robot picks up where it left off once robot mode returns to teleoperated mode. The driver station mode line is always complete.

For our unsuccessful matches, the robot mode line ends abruptly in whatever mode it was in at the time of the failure. We had robot failures in the middle of autonomous, at the beginning of autonomous, in the middle of teleoperated and at the beginning of teleoperated. I'd love to know if you think this is just a symptom or reflects a problem with the processing.

We saw very similar behavior in our robot during practice matches when we were communicating between the driver station wirelessly and when we had the radio configured to automatically select the channel on which it would communicate. The main cause seemed to be interference in the wireless network. When we analyzed the channels being used by the school network's access points, and then switched to a free channel that had no access points, we had no further problems in our practice matches. We also saw similar behavior when we were trying to write too many Log() commands to the SmartDashboard. Again when we reduced the number of SmartDashboard Log() commands, the problem cleared up.

For competition, we are displaying very limited data on the SmartDashboard. For San Jose we are going to have a version of our code where we take out all SmartDashboard logic.

For competition we also reset the radio to automatically select the channel again, since this is how the instructions indicate the radio should be configured.

It feels to us that the more crowded the venue became with spectators, the more likely we were to have a problem with our robot operating properly. Thursday we ran in many practice matches with no failures. Friday we failed 3 of 7 matches. Saturday morning we failed 3 of 4 matches. We declined to participate in the elimination rounds because we were not performing consistently.

I'm attaching a document that has screen print of the log viewer files for our failed matches, as well as the log files. For the screen prints, I expanded the display so you can more clearly see where the robot mode line ends in relation to any dropped packets and drop in CPU.

We are planning to work on the field with Jakub Fiedorowicz tomorrow morning in San Jose before the practice matches begin to see what more we can discern about our failures.