2012 Field Comm. Issue Logs
 

The amount of complaints about field connection issues at events this year has been unusually high. There have been cases of teams getting very slow response times from their robots (like 150ms slow). We should document these along with some information about robot setup in this thread. Maybe with enough eyes on the issue the bug will appear.

If you had communication issues at events (high ping times, non responsive robot, laggy controls), will you take some time to post some information here?

Please also post a short description of your issue as well as an upload of the log file that shows this. Located at : Users\Public\Documents\FRC\Log Files

Also, answers to the following questions should be helpful:

- Which event?
- Wireless bridge radio HW revision? (A1, A2, or B1)
- Radio firmware rev? (1.21, 1.4, or maybe something else)
- Programming language?
- Using a dashboard app? Labview, SmartDashboard, or custom?
- Using vision with Axis Camera and cRIO processing?
- Using vision with driver station processing?
- Did you have the radio mounted near motors/large metal structure?
- Using classmate (or similarly slow computer) as driver station?
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version)
- CAN Jaguars? What FW version?

I know some people are trying to gather data on their own about events they volunteered at. If you are one of these people, can you post your findings here?

EDIT:
Added some other things from Joe, Kevin, and linuxboy. Thanks!

EDIT 2:
Added 4/8 slot cRIO.

EDIT 3: Can jaguar

Re: 2012 Field Comm. Issue Logs
 

Which event? Kansas City
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? Yes
Using classmate (or similarly slow computer) as driver station? Yes

Re: 2012 Field Comm. Issue Logs
 

Which event? Los Angeles
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? Modified Labview Dashboard
Using vision with Axis Camera and cRIO processing? Included image processing, but removed at competition due to lag issues. After removal, only passed image to DS
Using vision with driver station processing? No, Classmate computer was slow as well.
Did you have the radio mounted near motors/large metal structure? ~1 foot clearance
Using classmate (or similarly slow computer) as driver station? Yes

The issue wasn't always related to our bridge/cRio/DS -- in some matches the bridge was unable to connect to the FMS. This was a known problem in LA and resulted in MANY 2v3 matches.

Re: 2012 Field Comm. Issue Logs
 

Erm. In addition to the answers to Tom's above questions, you should probably also post the actual issue/symptoms you were experiencing, since that varies from long pings and delays to intermittent or no Comms.

Re: 2012 Field Comm. Issue Logs
 

For rev A radios, it would be helpful to have the firmware version (1.21, 1.4, or maybe something else).

It would also be helpful for a description on the problem, and some representative DS log files (in Users\Public\Documents\FRC\Log Files)

Re: 2012 Field Comm. Issue Logs
 

Team 846:

Programming language? C++
Using a dashboard app? Is it SmartDashboard? Started with SmartDashboard, but removed all logging during competition (did not solve problem)
Using vision with Axis Camera and cRIO processing? No
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? No
Using classmate (or similarly slow computer) as driver station? Yes

We're using a Rev A radio, and tried firmwares 1.21 and 1.4, with the same results in both.

Problem: Intermittent connection on the field, losing comms generally around the end of autonomous, and randomly thereafter. Total time actually connected varies between 10s to 20s; packet losses of 25 / 50 are noted immediately before the communications goes down.

Re: 2012 Field Comm. Issue Logs
 

Thanks, can't believe I forgot those things. Edited the orignal post.

Re: 2012 Field Comm. Issue Logs
 

I've also edited the original post to include a question about 4 vs. 8 slot cRIO.

Re: 2012 Field Comm. Issue Logs
 

Tom I would suggest to also check if CAN-bus communication was used, and if so, what version firmware was loaded on the jaguars.

Re: 2012 Field Comm. Issue Logs
 

We have correlated (note, not causation, just a hypothesis) multiple field failures (full connection loss) to the Blue alliance station number 1. At FLR, our robot failed multiple times in that station only, as did our alliance partners. Further supporting this theory is the fact that during the qualification matches, the red alliance won 61% of matches. Considering the randomized aspect of schedules, this seems somewhat outside the 50% norm it should be. The field is going to Buckeye this week, and DC next week. We'll be at DC, watching it, but I would suggest anyone at Buckeye to keep an eye on that station. It might have just been assembled at Fingerlakes with a wire loose, or it could be a problem in its hardware. Or this whole thing could just be coincidence. Just something to keep in mind.

Re: 2012 Field Comm. Issue Logs
 

I don't understand. It sounds like you're saying that more than one robot was using the Blue 1 alliance station at the same time. While that's likely not what you meant, it would certainly mess with their ability to connect!

Re: 2012 Field Comm. Issue Logs
 

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

Re: 2012 Field Comm. Issue Logs
 

At our third Elims match at WPI, our bot executed autonomous fine. However, upon entering Teleop, spun in a tight circle, accepting no input from drivers and did not respond to an E-Stop command. What gets me is we were running fine throughout every other match that day. No new code that wasn't tested was loaded onto the robot, and *nothing* changed between our second Elim match to our third. (This includes electrical).

I'll post more details and the wacky logs later.

EDIT: We used the 4-slot cRIO, LabVIEW, and did not use CAN.

Dashboard: Edited version of the FRC Dashboard, did not cause any problems in the past.

Radio: Not mounted near motors. 2012 KOP radio.

Computer: MacBook Pro, 8gb ram, Intel core i3 running Parallels. (Had no problems with the FMS throughout the weekend once we set up the network settings correctly.)

Vision processing: No.

Ave. cRIO CPU usage: 40-60%

Re: 2012 Field Comm. Issue Logs
 

Day one is over at St.Louis Regional and as Jspatz posted above we are experiencing the same issues like we did in KC heres Settings again:

Which event? Kansas City
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? Yes
Using classmate (or similarly slow computer) as driver station? Yes

On Thursday there was a ton of robots not being able to connect(Including our team) to the field mostly on blue side this also is the field that was used at bayou, In which one team couldn't connect the entire day on Thursday. We all had different firmware versions like 1.40,1.20 and 1.30 So they all downgraded the routers at the regional to 1.2 and every robot was able to connect to the field but we still get this "intermittent" com drop when we go to get a ball at a side in the arena.

There are alot of teams here that are or seem to be having the same issue of really high trip times from the FMS even after code is modular and changed.(Shout outs to helping all the teams FTA'S)

What gets me that It only happens on one part of the field closest to the Antenna to the FMS?

Anyone else having this issue? Also if you have trouble at your regional with networking check the version It also makes a really big difference if you are version 2.00 Hardware Revision B which also doesn't work on the field. One robot with a different one can cause a major drop out for potentially the entire field. (Which happened This goes for RevA also) Really good job FTA's on figuring this out

Re: 2012 Field Comm. Issue Logs
 

I'll let our CSA from our team chime in later with details.
Apparently, we are the only team with a 1.4 firmware at our regional, including our backup one.

In our only loss in a match today we lost comm (lag issue) for the last 30 seconds in our match.:(
It affected our ability to do the coop bridge also. Not knowing how it could've changed the outcome was frustrating.

Re: 2012 Field Comm. Issue Logs
 

If that is the field from Bayou, I would be interested in knowing what FW version the Loan Router has. We used it basically all Friday and Saturday.

Can you please get someone to check on this for us?

Re: 2012 Field Comm. Issue Logs
 

When you guys say you lost comms with the FMS, do you mean a spike in lost packets, an exceedingly long trip time, or "No communication" displayed in the DS? I'm interested in the fact that so many people - including Team 23 - are having issues with the FMS.

Re: 2012 Field Comm. Issue Logs
 

I've now heard of another team at WPI where the robot eas only able to spin in circles. Can someone from the drive team give details as to when it started during the match, what you were able to do, what eventually happened, etc?

I looked for footage but couldn't locate either. I was able to investigate one of these right after it happened, and expected it to be a chain, jammed, a speed controller, etc. I found a missing limit jumper on a jag that would have disabled one motor in one direction, but it wasn't clear that this was the issue, contributed to it, or was there all season and they were simply driving with a severely imbalanced drivetrain. They had four motors.

Before I explain what I suspect, I'd like to hear about recent issues involving spinning robots that weren't due to mechanical or speed controller issues.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

I've just sent our drive team a message asking for further details. As soon as they get back to me, I'll let you know. Do you need/want any information on our router version and firmware?

If I remember correctly, the limit switch jumper just reverses the motor direction on the Jag?

EDIT: Our operator is telling me that they were able to control it for literally 2 seconds, enough time to step up to the controls and try to drive it. He went on to say that our two camera feeds froze the second we started spinning. He doesn't remember what the RSL was doing, but since the logs say we were in teleop, I would assume it was flashing the teleop sequence.

Re: 2012 Field Comm. Issue Logs
 

I can't seem to find the Luminary manual that would be definitive, but I think the limit switch is used to enable or disable movement in a particular direction. They are also used for ramping if using the latest FW. If missing, I believe this jumper will stop and either coast or brake the motor and ignore inputs in that particular direction.

As for firmware version info, sure. Give it to me. But honestly, what I want most is a eye-witness account of the behavior or a video of the incident. I didn't see it, and even the one I saw happened unexpectedly, and I didn't see it begin. I walked to the side of the field to see if I could see a jammed ball, a chain obviously jumped off the sprocket, etc. After the match people were excited and talking about field faults and whatnot, and I couldn't get a good explanation of what took place. I don't want to bias someone into telling me symptoms that support what I think may have taken place, so I would like to have a description first.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

I won't be able to get any FW info to you until Wednesday at the earliest, but I can tell the drive team to pop a comment in here with a full account.

Re: 2012 Field Comm. Issue Logs
 

Same situation just happened in Seattle Olympic, stay tuned for official post on the issues that happened during the finals.

Problems included: Response time lag from controls to robot through field management and lost camera relay for two robots on one of the alliances.

* I am not there, just talking to people who just had these issues, I'll let them clarify shortly.

Re: 2012 Field Comm. Issue Logs
 

As Dom has requested, this is Team 23's recap of what happened to our driving situation.

Autonomous started and everything went exactly as planned. No problems whatsoever, everything checked out functional.

Teleop started and the team reached for the controls. The driver of the robot then went to turn the robot around and the robot began spinning uncontrollably. The drive team did initially have control during teleop for a couple seconds at most before the robot began to spin out of control (it appeared to be repeating the command to turn). The moment we lost control, the camera feed on the driver station had frozen and the joysticks were not being responsive.

A field tech came over and advised us to reboot the cRIO. I am not sure if the robot was spinning straight through to the point of rebooting or not. I think it was, but am not entirely sure. Upon rebooting, the joysticks became responsive again, however by the time robot code had been sent and communication was recovered, the match had only 1 second left.

That was the more physical memory of the situation we went through during our last elimination match. If you're looking for any messages or code problems, refer back to DominickC instead of myself as i was merely driving.

Re: 2012 Field Comm. Issue Logs
 

This is correct.

We've been watching the field at buckeye, and some of the robots that we noted were failing at FLR, and have come to the conclusion that it was indeed coincidence.

Re: 2012 Field Comm. Issue Logs
 

In Hawaii today, I can note at least 20 robots that had the lag issues today.
It cost us the opportunity for coopertition in our remaining 3 matches today.
Luckily for us, it always happened in the last 30 seconds where we already had the lead, since all 6 robots would experience lag and couldnt do anything at that point.
For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

I'm no expert in control systems, but smart enough to understand that something is definitely fishy here.
Mark Koors, our FTA is well aware of that situation now.

Re: 2012 Field Comm. Issue Logs
 

The quarters in Hawaii were definitely infected with spooks! All six robots lost control in our first match for a few seconds near the middle and then lagged for about 15 before ending fine. The second match was fine for about 15 seconds and then we had no control of any buttons for the rest of the entire match. We could drive, barely, but all six robots experienced the same thing. I think the outcome was inevitable but I know I'd be upset if I was on the losing alliance with so many field issues. They seemed to fix it during a long break between quarters and semi's.

Re: 2012 Field Comm. Issue Logs
 

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

Re: 2012 Field Comm. Issue Logs
 

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.
The only permanent positive note is that it didnt happen to our team again.

I can only conclude that something much bigger is happening, and not just at our regional as noted in this thread.

Re: 2012 Field Comm. Issue Logs
 

I can say with certainty that we ran into lag issues with computer CPU consumption by the SmartDashboard; keeping it closed solved apparent lag problems, although the NYC FTA (Mark Mcleod) noted that we retained high trip times (~175 ms, as opposed to a standard 5-6 ms) throughout the entire event.

• Which event? NYC
• Wireless bridge radio HW revision? (A1, A2, or B1) A1
• Radio firmware rev? 1.21
• Programming language? Java
• Using a dashboard app? n/a
• Using vision with Axis Camera and cRIO processing? Ran a live stream with no processing at default fps
• Using vision with driver station processing? DS display; nothing else.
• Did you have the radio mounted near motors/large metal structure? Near two Black Jaguars, above the 12V/5V converter*.
• Using classmate as driver station? 2go E11 Classmate, not the KoP one
• 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4-slot
• CAN Jaguars? What FW version? Yes, rev92.

*This issue is something I realized post-NY. As Alan has pointed out, this could possibly produce issues; it will be fixed when we go to CT.

Re: 2012 Field Comm. Issue Logs
 

Some data points:

At Rutgers to a small extent and at NYC to a much greater extent, we were seeing the robot stop responding for a second to several seconds, somewhat randomly. This would occur both tethered and via radio (!)

Reviewing the logs from NYC we found our latency was around 170 mSec ('good' would be <10 mSec).

Our 8-slot cRio showed a CPU usage of 50-60%, which is reasonable.

Carefully working to recreate the issue on our prototype robot, we ultimately found that out driver station CPU was maxing out.

We have not yet deemed this issue resolved, but we believe we're almost there. We'll find out when we unbag this week in anticipation of Mount Olive.

(I don't have the details requested in the OP, but can get them if that will help. All our cRios are 8-slot, we use LabView, and our radios are all Rev A).

Re: 2012 Field Comm. Issue Logs
 

Reviewing logs sent to me from Bayou, I'm seeing a good number of glitches caused by the DS laptop. They don't necessarily cause lag, but I see periodic spikes in trip times and sometimes in lost packets. I've accidentally reproduced this myself by having IE open on the laptop. I also saw teams, with no dropouts or issues, that had 100% CPU usage continuously -- auto, tele, and disabled. I'd like to encourage teams to use the charts tabs and practice match built into the DS to look at their memory usage, CPU usage, and any repeatable spikes in trip time. I was able to open the task manager and see the odd Windows program jumping up in CPU from time to time.

While it is not a requirement to make a kiosk account to run the DS, if you see spikes, see if stopping other apps or killing other processes will get rid of them. And I believe directions for making a kiosk are available from FIRST.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

I don't have a lot of the information you'd like to see here since I am not a control systems person by any means. I am, however, the drive coach and so I see the problems we have first hand.

We ran throughout the Alamo regional without any trouble. We ran throughout the qualifying matches of the Seattle Olympic without any trouble.

During the elimination rounds, we experienced severe lag in several matches and lost the video from our camera in at least one match. Several others experienced similar issues including loss of the camera feed, both on our alliance and on others.

The common sense that FIRST is so fond of suggests that the chance of the same issue plaguing multiple teams at the same time is quite small, but does not, apparently, apply to the field or control system.

I'll try to get someone with a better understanding of how our control system is configured to chime in.

Re: 2012 Field Comm. Issue Logs
 

Based on what we saw in Hawaii, I'm going to ask the Houston Inspection crew to check the firmware version. It takes about 4 minutes to download the firmware. If Inspection thinks it would take to long to download any 1.4 bridged there, I'll go to the pits and do it. It may not help, but I don't think that it will hurt.

Re: 2012 Field Comm. Issue Logs
 

We had comm issues at Palmetto and here's what happened.

1. Started out OK.
2. We turned on our camera software (image processing on the netbook DS). Ping times reported from the FMS tech were spiking at >1sec!!!!! (no data on CPU usage on netbook). The field techs down graded our radio firmware.
3. Tried again with the redone radio. Ping times still >1sec.
4. Turned off camera processing on netbook DS. Everything OK.
5. Setup a laptop to replace the netbook as the DS.
6. We turned on the camera processing with a notebook. Everything OK from both our and the field techs point of view. Notebook CPU < 35%.

Our experience is that many communication issues are probably tied to CPU usage on the netbook or cRIO. This was mentioned in one of the team updates.

On a related note, during eliminations, we started setting up the notebook at the driver station and connected to the FMS. Then we spotted that we only had 10min of battery life left and decided to do a quick swap to the netbook to avoid problems in the match. Sop we unplugged the joysticks, enhancedIO and network from notebook and plugged it into the netbook. DON'T EVER DO THIS !!! Once you connect to the FMS you should consider yourself committed to that DS computer. Our netbook didn't recognize the joysticks soon enough and we couldn't drive the robot during the match. Functions on the EnhancedIO worked (shooter). Nearly cost our alliance the quarter finals.

Re: 2012 Field Comm. Issue Logs
 

So, have we isolated issues to the driver station computer's CPU? Or have we deemed it to be a contributing factor? We were using my personal computer as our drivers station due to our dedicated computer biting the dust...

2011 MacBook Pro 13''
2.4GHz Intel i5
8gb DDR3
Parallels x64 Windows 7 running on Mac OS X Snow Leopard.

Re: 2012 Field Comm. Issue Logs
 

We experienced what were usually short blips where we could not control the robot, and it simply sat there. The FTAs would comment that they usually saw the comm blip, but we never saw comm drop on our driver station. One of the FTAs (Jerry) inspected the logs from our driver station, During the time that we lost control, the battery reported constant voltage, confirming that nothing was running on our robot. Nothing in the log looked "wrong." When we regained control, the ping time would spike.

We suspected that it was a bad USB cable connecting our OI, and that we really did stop giving the robot commands. However, we replaced the USB cable, and the issue persisted.

- Team 1024
- Which event? Boilermaker
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? Java
- Using a dashboard app? SmartDashboard
- Using vision with Axis Camera and cRIO processing? No
- Using vision with driver station processing? No We would, but left the camera power unplugged for fear of it causing more comm issues.
- Did you have the radio mounted near motors/large metal structure? Mounted on a metal panel near metal tubes and near 2 Jaguars
- Using classmate as driver station? No, an HP 6530b
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? All version 101

Re: 2012 Field Comm. Issue Logs
 

While probably not the cause of your issues, [R61] requires version 99 or higher. The current version is 101.

Re: 2012 Field Comm. Issue Logs
 

I guess that I would say contributing factor. We had problems when running 1.4. The processor on our DS is an i7. We do vision processing in the DS, but the usage doesn't go much about 25% - as I remember.

Re: 2012 Field Comm. Issue Logs
 

Team 781 competed at Waterloo this past weekend. We have two Axis 1101 cameras connected to a classmate e09 ds (no image processing). At two matches I watched the field display screen and saw round trip times of 100-plus and accumulated lost packets in the hundreds of thousands (versus below 10/100 for the other teams). The field tech indicated that the two cameras were the cause. We will be switching to a faster laptop for the driver station, and would like to know if there is any way we can test this (e.g. Display trip times and lost packets) on our practice bot/field before we head to Queen City. I'll continue to research this, but I figured you folks would be able to provide specific guidance.

- Team 781
- Which event? Waterloo
- Wireless bridge radio HW revision? (A1, A2, or B1) A, can't verify A1 or A2 until I can get into it at Cincinnati.
- Radio firmware rev? I'll check at Cincinnati
- Programming language? C++
- Using a dashboard app? Regular Dashboard customized to display both camera images (could we have done this incorrectly such that we have initiated the communications issues ?)
- Using vision with Axis Camera and cRIO processing? 2 x m1101, no vision processing
- Using vision with driver station processing? No
- Did you have the radio mounted near motors/large metal structure? Mounted on a lexan panel near the battery
- Using classmate as driver station? Yes but hopefully not for much longer.
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? No CAN

Thanks
Pat

Re: 2012 Field Comm. Issue Logs
 

Could you be maxing out one of the i7 cores, assuming it is a 4 core processor? I know we have a beefy driver station without image processing and without the smart dashboard and I did not hear our kids complaining about any lag issues. We were at St. Louis where other issues were reported above.

Re: 2012 Field Comm. Issue Logs
 

Trip times and lost packets are graphed on the Charts tab of the Driver Station and in the logs viewable with the Driver Station Log Viewer in C:\Program Files\FRC Driver Station.

Re: 2012 Field Comm. Issue Logs
 

It is a quad core. I don't think that we are maxing it out. I'll take a closer look in Houston next week.

Re: 2012 Field Comm. Issue Logs
 

Team 1280 here and we had some major problems at the Davis competition. On Thursday we ran fine all day. Once we got into matches though, we had problems.

• Match 8: Ran successfully, had spike of 33 dropped packets early in autonomous mode. No latency issues. Max CPU at 90%
• Match 14: Robot froze 3 seconds into autonomous. No spike in latency or dropped packets. Max CPU at 85%
• Match 18: Ran fine at operator end however there was high latency and lots of dropped packets throughout the match. Peak latency at 50ms, packets at 15, Max CPU at 96% mostly under 90%.
• Match 28: Failed early into teleop. Spike in dropped packets at 58 packets at the start of autonomous, we ran through autonomous fine, but a few seconds into teleop we froze and had spike of 18 dropped packets and our CPU drop from around 72 to around 17 and then go all the way down to idle. Latency hovered around 5ms for the entire match.
• Match 37: Ran fine. Spike of 33 dropped packets at start but CPU averaged around 68 during teleop.
• Match 45: Ran fine. We unplugged the camera and our CPU spiked at over 100% and was oscillating between 45% and 100% at regular intervals from when we powered on the robot. Latency had one spike at 22ms, but other than that it stayed around 5ms. We had regularly had dropped packets but they never went above 10
• Match 51: Did not run autonomous. Had an oscillating CPU usage between idle and 100% at the same interval as Match 45. We think the camera was still unplugged. We only lost 5 packets at the start but had latency spike of 25ms.
  
  Saturday things went worse.
• Match 65: We did not run. 41 lost packets at the start of autonomous coupled with a latency spike of 17ms. Our CPU usage did not go above 10 before it stopped.
• Match 70: We did not run again. No dropped packets, we had CPU drop to idle after autonomous, latency was pretty consistent at 5ms.
• Match 77: We ran fine. There was a 38 dropped packet spike at the start of autonomous but latency stayed around 5ms. CPU usage stayed around 80%
• Match 84: We did not run. We had 98 dropped packets just after the start of autonomous and after the last time our robot said it was in autonomous. Latency stayed at 5 while we lost the packets. CPU never went above 15% before we lost communication.

We made minor code changes throughout the weekend. As you can see, our problems got worse as the competition went on and there was no consistent pattern to it. The only consistent thing about the loss in communication was that our robot mode line disappears from the log charts when we lose communication, while the driver station mode line stays. We are not sure if this is a symptom or a cause. We are getting complete loss in communication, rather than temporary loss of communication. We are hopefully going to find out more on Thursday in San Jose, but for now any help would be great.


- Team 1280
- Which event? Sacramento/Davis
- Wireless bridge radio HW revision? (A1, A2, or B1) A1
- Radio firmware rev? 1.40
- Programming language? C++
- Using a dashboard app? Currently using SmartDashboard for display statements using log function. FRC Dashboard is on the side for the camera feed.
- Using vision with Axis Camera and cRIO processing? 1 Camera with vision processing on 8 Slot cRIO. Resolution at 320x240 @ 7fps
- Using vision with driver station processing? No, but we do have a camera feed (FRC Dashboard)
- Did you have the radio mounted near motors/large metal structure? Mounted at top of robot near shooting motor.
- Using classmate as driver station? No, using 6730b HP laptop (Win7)
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 8 slot
- CAN Jaguars? What FW version? No Jaguars

Re: 2012 Field Comm. Issue Logs
 

A couple things that may help with reading the logs.

1. A spike at the beginning of auto is an glitch in the measurement. The DS was modified to zero the lost packets when the match begins. As a result, the delta loss reported into the logs will sometimes show a high spike at the very beginning of auto. You cannot trust this point.

2. If the communication with the robot fails, the voltage will disappear, and other lines such as CPU should be flat and cached. Lost packets and trip time will often look high for a time when a drop occurs.

3. The lost packets numbers is typically "out of 25". So 10 lost packets over half second means 10 out of 25 were lost. If communications is lost entirely, the timeout is 1 second, so the number may go above 25 and be "out of 50".

4. When packets are lost, it is common for CPU to drop. The code for tele and disabled are typically waiting for new DS packet, so less to do when less packets comes in.

5. If the DS line suddenly shows a disabled line, but the robot line seems to ignore it, this indicates a watchdog due to communications drop. This means the robot outputs were disabled due to more than 5 consecutive packets not arriving. Note that it is possible for occasional packet arrival to keep the robot enabled, and this is typical.

6. If you see periodic spikes in the trip time and/or the lost packets, this may very well be due to the DS. If you observe these when cabled, that is supporting evidence that something on the DS is a likely cause. Opening the Task Manager may show a process bumping to the top in time with the blip.

7. Trip time doesn't necessarily mean latency. The trip time includes the trip to the robot and back again. If delayed to the robot, it impacts driving. If delayed on the robot or on the return trip, or often within the DS laptop, the trip delay does't imply that the robot is hard to drive due to lag.

8. Similarly, lost packets may make it to the robot and keep the watchdog alive and be lost on the return trip.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

So have we arrived at the point where we need an RTOS capable system in the Driver's Station?
The piles of threads available in a Core i7 are not exactly conducive to deterministic response times.

Shoot, let's push it out a little. Is 802.11 good enough?

Re: 2012 Field Comm. Issue Logs
 

If it can play video games, it is probably realtime enough to be a DS. The rate of both is defined by human perception and reaction time.

As image processing and other code written by teams migrates to the DS laptop, it will impact the comms. Throw in a Kinect or two, and ...

But at this point, I think some teams have overloaded a component of their robot -- the DS laptop -- not unlike how they are capable of overloading a given circuit or mechanical component. The Task manager alone will give good enough feedback to learn about and avoid the issue.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

Greg,

I am one of the mentors for Team 1280. Thank you for the guidance on reading the log files you provided. It is helpful to know what is normal.

I've looked carefully at our log files for both our successful matches and our matches where we stopped running. Unfortunately there is no discernible pattern in CPU, dropped packets, or trip time that correlates to when we lose the ability to control our robot on the field.

We do have voltage and CPU readings in all our log files for the duration of the run. It makes sense that if we are getting these readings, we do still have communication with our robot even if we cannot control the robot from the driver station.

For the line on the log files that shows the Robot mode, what does this really tell us? When we run successful matches, the lines that make up the Robot mode clearly show the transition from disabled to autonomous to a short disabled to teleoperated then back to disabled. Occasionally we see a little blip of disabled in the middle of teleoperated but not often. While we see the expected CPU drop when this blip occurs, the robot picks up where it left off once robot mode returns to teleoperated mode. The driver station mode line is always complete.

For our unsuccessful matches, the robot mode line ends abruptly in whatever mode it was in at the time of the failure. We had robot failures in the middle of autonomous, at the beginning of autonomous, in the middle of teleoperated and at the beginning of teleoperated. I'd love to know if you think this is just a symptom or reflects a problem with the processing.

We saw very similar behavior in our robot during practice matches when we were communicating between the driver station wirelessly and when we had the radio configured to automatically select the channel on which it would communicate. The main cause seemed to be interference in the wireless network. When we analyzed the channels being used by the school network's access points, and then switched to a free channel that had no access points, we had no further problems in our practice matches. We also saw similar behavior when we were trying to write too many Log() commands to the SmartDashboard. Again when we reduced the number of SmartDashboard Log() commands, the problem cleared up.

For competition, we are displaying very limited data on the SmartDashboard. For San Jose we are going to have a version of our code where we take out all SmartDashboard logic.

For competition we also reset the radio to automatically select the channel again, since this is how the instructions indicate the radio should be configured.

It feels to us that the more crowded the venue became with spectators, the more likely we were to have a problem with our robot operating properly. Thursday we ran in many practice matches with no failures. Friday we failed 3 of 7 matches. Saturday morning we failed 3 of 4 matches. We declined to participate in the elimination rounds because we were not performing consistently.

I'm attaching a document that has screen print of the log viewer files for our failed matches, as well as the log files. For the screen prints, I expanded the display so you can more clearly see where the robot mode line ends in relation to any dropped packets and drop in CPU.

We are planning to work on the field with Jakub Fiedorowicz tomorrow morning in San Jose before the practice matches begin to see what more we can discern about our failures.

Re: 2012 Field Comm. Issue Logs
 

Oops, I made a mistake there. That should be rev99.

Thanks for pointing that out.

Re: 2012 Field Comm. Issue Logs
 

At CT today, things seemed pretty good. I don't think I saw too many connection problems (but then again, I wasn't hanging around the field much). For us today, we were only dropped by the field in one early match this morning. I had a long talk with the FTA who was pretty helpful, and we decided it was an isolated indecent - we made it the rest of the day with no issues at all, so I think we'll be okay. But I'll fill this out and attach a screenshot of the log viewer just for the sake of another data point.

- Which event? CT
- Wireless bridge radio HW revision? (A1, A2, or B1) A1
- Radio firmware rev? (1.21, 1.4, or maybe something else) 1.21
- Programming language? LabVIEW
- Using a dashboard app? Labview, SmartDashboard, or custom? custom LV dash
- Using vision with Axis Camera and cRIO processing? yes
- Using vision with driver station processing? no
- Did you have the radio mounted near motors/large metal structure? no
- Using classmate (or similarly slow computer) as driver station? yes
- 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4 slot
- CAN Jaguars? What FW version? no

Re: 2012 Field Comm. Issue Logs
 

One other team at the Hawaii Regional was identified as having 1.4, but not until teams were preparing to pack up at the end of the event. I wasn't aware at the time that the WPA programming kiosk kept records of D-Link firmware revisions. Looking at its logs would have been a much quicker way to find the potentially problematic 'bots.

The larger issue is still mysterious, but it does seem to involve the order connections are established.

Re: 2012 Field Comm. Issue Logs
 

Durring elemination matches we ad 1985 connected last and we didn't have problems. the order was really prevalent or when they did a reset there were no problems. How do teams correct the problem if its all about revisions though? We could go out to the field and it could all be messed up at champs? hope we can get to the bottom of this.

Re: 2012 Field Comm. Issue Logs
 

During the St. Louis Regional, 1985 experienced what seemed to be connection issues throughout the entire first part of the competition. Those issues turned out to be a hardware problem. We went through the process of replacing our classmate first, then our entire power distribution block. When these did not fix it, we had to replace the CRio with a replacement, although the one we had been using was a brand new one. This completely fixed our problems and we returned the CRio to NI and they replaced it still under warranty. It made for an incredibly frustrating day, but literally our connection issues turned out to be from a bad part. All of this did happen after the firmware issues with the radio were corrected.

Re: 2012 Field Comm. Issue Logs
 

We (Team 1280) had another similar crash in one of our matches at Silicon Valley today. Back in the pit we crashed again, this time with NetConsole running and were able to capture the following exception and error messages on NetConsole:

We use SmartDashboard to display some data from our robot using the Log() method. We do not access the NetworkTables class independent from SmartDashboard..

Since we are not able to run NetConsole during matches, we have no idea if we get this same exception every time we crash, but we'll keep looking for it when we are tethered in the pit. Because of this error, we have removed our use of SmartDashboard from our robot code.

Re: 2012 Field Comm. Issue Logs
 

I don't have many details right now, but we had to sit out 5 matches at SVR today. I'll try to post back with some details tomorrow.

Re: 2012 Field Comm. Issue Logs
 

I'm glad you were able to capture this. I wasn't able to post last night, but the logs you posted seemed to indicate a crash. If the battery trace doesn't go away, the robot it connected to the field. The CPU goes low because the thread the code was in was terminated. Hopefully you can use practice matches and some code walkthroughs or some clever debugging to track it down.

Greg McKaskle

Re: 2012 Field Comm. Issue Logs
 

Although probably a different issue than is being experienced by the others, 253 at SVR has been consistently unable to connect to the field. The robot works correctly on the tether, but will not communicate with the FMS.

So far, the cRIO-II has been imaged (v. 43, 2012-01-20 imaging tool) and replaced, code has been recompiled and reloaded, the radio has been reflashed and replaced, and all of the cRIO modules have been either checked or replaced (a sidecar and bad 37-pin cable were replaced). The robot is running C++ code and Jaguars over PWM.

The system has been tested with two driver station computers: a regular laptop and a loaner classmate. Neither works on the field.

I have a couple more wild theories: can anyone sanity-check them?

Firstly, what would happen if the WPA key given in the kiosk and the one from the FMS were inconsistent? Would we see red indications for code and communications on the dashboard?

Secondly, wasn't there something in the LV code (and maybe C++) where you could run code from the laptop, but it wouldn't be installed on the cRIO? I haven't seen that in a while, but maybe the team has an incorrect build option set? (I don't think this is it, though, because the robot flash codes seem to indicate it's running code.)

Re: 2012 Field Comm. Issue Logs
 

With an incorrect WPA key, the bridge could never link to the field and you would see red for code and communications and red for Bridge on the diagnostics tab

You can do that for both LabVIEW and C++ but in both cases you would see green for the Communication light and red for Robot Code.

Re: 2012 Field Comm. Issue Logs
 

Does the robot connect but not stay connected, or does never connect to or through the field? Are the cabled tests in the pits practice matches? what rev radio and what FW version?

Greg Mckaskle

Re: 2012 Field Comm. Issue Logs
 

Which event? 10,000 Lakes
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? No
Using classmate (or similarly slow computer) as driver station? Yes

During our last Qualification match, we were all running in the beginning, then two robots on our alliance dropped at the same time. 30 seconds later we dropped connection causing our alliance to loose the match! We talked with the staff and they blamed it on each teams code, which makes no sense since they all were working in every other match without any code change. The languages were different to, one was labview, the other, C++. Is this just a fluke with our robots (only match we dropped to) or would it have been some sort of field problem?

Re: 2012 Field Comm. Issue Logs
 

We're not sure what the problem was exactly for our robot during competition (Team 2502) and we didn't get to find the exact reason it didn't work, but we did fix it.

First, the radio was set to auto rather than bridge. For competition it needs to be on bridge, but it seems that for other testing (such as in the pits) the radio should be on auto.

Second, the radio was right next to our drive train motor and clumped around a lot of wires. So we moved that up farther so that it could be seen from over the ramp.

Last change, we made a slight hack to allow faster operator control loops. Which meant only putting drive train in the operator control loop, but creating a new thread to handle everything that would have been in operator control.

Note on the last change, our graph before this change was showing very long loop time (green line) and after the change it decreased drastically (of course). I'm not sure which of these things could be the issue, but I would double check these.

Re: 2012 Field Comm. Issue Logs
 

Cabled tests were in the pits. It never worked with field FMS.

Radio was initially hardware A1 with 1.4; swapped with a loaner with hardware A1 and 1.21. (When I said "reflashed", I should have specified that the WPA key was updated using the kiosk; neither radio had new firmware installed.)

We verified the WPA key in the FMS and on the robot radio, and manually re-entered it. It was correct initially, and re-saving it had no new effect.

As for the build/deploy, we independently confirmed what was mentioned above: these symptoms don't match that fault. (And we made sure the code was deployed when reloading it.)

Further attempts were made: a third laptop (known good) as the DS had no effect, except under one very specific condition. The initial DS laptop (some sort of ordinary Windows PC) was connected to the field at the driver station via Ethernet, and the DS VI was started. (It indicated FMS was found.) The known good laptop was connected directly to the robot radio via Ethernet, and various software was loaded (potentially including another instance of the DS VI). At this point, the communications and robot code lights on the driver station DS VI flashed on and off intermittently, for a couple seconds at a time. Nobody had any firm idea what was going on there.

Re: 2012 Field Comm. Issue Logs
 

After removing the SmartDashboard we (Team 1280) performed beautifully for our next seven matches at Silicon Valley. We did not have any further crashes, either on the field or tethered in the pits.

GRT (Team 192) did get their problem resolved at Silicon Valley as well. Their symptoms were different from ours. The report I heard was they had an IP address conflict - they had an on-board laptop with the same IP address as their Axis camera. This is 2nd-hand information but I heard it from a very reliable source.

One note for those looking at their radio firmware as a potential root cause. We ran 1.21 in Silicon Valley and continued to have our issues. As Greg said, we continued to log our battery voltage and CPU usage from the robot on the driver station so communication with our robot through the field was not our issue. It wasn't until we removed the SmartDashboard that we ran consistently.

Silicon Valley is our last competition for the season so we won't have any additional information to provide this year on our performance. I hope everyone else experiencing issues is able to get theirs identified and resolved! Best of luck to the teams still competing and those headed to St. Louis!

Re: 2012 Field Comm. Issue Logs
 

We were having lag issues at the Dallas East Regional. Basically, since our shooter was so dependent on precise timing, any delay would cause the shot to be off. We determined there was probably a delay due to encryption and packet sniffing (to find battery voltage). We also assume there was interference because there typically is with so much wireless accessing going on.

So whenever we were tuning in the practice field hard tethered, we shot upwards of 60 feet accurately, but when connecting to the field and running a match, we never got an accurate shot. We eventually did by guessing, and that was 50 degrees off of our practice shots.

Re: 2012 Field Comm. Issue Logs
 

As noted above, we had some comm issues due to the onboard computer having the IP address of an Axiss camera. However, we had no issues Thursday, and the problem only surfaced on Friday. In addition, after 971 and 100 helped us locate the problem, we were never able to turn on tracking (we're pretty sure the computer as well) again.

Unfortunately, there is no way to test our connection status until Championships, as the robot operates perfectly tethered and from a wireless router.

Re: 2012 Field Comm. Issue Logs
 

At the Troy regional, we experienced our first bout with the 'lag monster'. For two painful matches, we were seeing the robot lag significantly - the driver and gunner both said they would try to make a motion and the robot would wait a few moments, then (sometimes) make the motion, then it would sometimes continue that motion well after they had stopped. As we neared the end of one match, we completely lost control of the robot.

The first thing we checked was CPU usage of the Crio. Rock solid at 70%. During tethered operation, we didn't see the same problems. Only on FMS.

Then, while tethered in the pits troubleshooting the issue, I clicked over to the charts tab our our driver station (09 Classmate model). We were seeing significant packet loss and round-trip-times off the chart.

This is with the standard driver station and dashboard. No modifications.

Initially I thought it was a bad tether cable, so we swapped it with no effect. Then, I hooked up our backup driver station (our top end programming laptop) and the lag and round trip issues completely went away.

Plugged back into to our classmate, and the problems came back instantly. Keep in mind however, while tethered, the problems were still not exhibiting themselves as lag. I suspect the added delay of traveling through the FMS system is what pushed it over the edge into creating a problem.

The classmate is 100% stock, completely clean and reimaged with nothing extra on it.

We won't be using the classmate anymore. The lag was gone using our higher end laptop in the next match.

Re: 2012 Field Comm. Issue Logs
 

As an update to my last post in this thread, we duplicated the issue in the lab and found the driver station CPU maxing out. We optimized that down to maybe 50%, and at Mt Olive this past weekend we experienced no further issues.

So for us, problem solved.

Re: 2012 Field Comm. Issue Logs
 

Which event? Greater Toronto WEST
Wireless bridge radio HW revision? (A or B) A
Programming language? Labview
Using a dashboard app? Is it SmartDashboard? No
Using vision with Axis Camera and cRIO processing? Passing image, no processing
Using vision with driver station processing? Yes
Did you have the radio mounted near motors/large metal structure? no
Using classmate (or similarly slow computer) as driver station? no

On practice day the field was refusing connects from our team and many veteran teams. The DS laptop would connect, then immediately loose connection to the field. After much trying (4 hours) and approx 12 teams of the 50 there could not connect, it was discovered that radios at version 1.4 were the common thread. We had our radio which worked flawlessly at 1.4 in Knoxville, had it reimaged to 1.2. After which we could connect but images from the camera and control response was definitely laggy'er than at the previous competition. FIRST field personel did everything to have buiding wifi networks shutdown, (on thursday there were over a dozen non-First networks in range) and after warning the crowd to turn off their mobile wifi things got a bit better. But our DS logs do show incredible packet loss on practice thursday. The field in GTR West was the same field used earlier at GTR EAST (which we did not participate in) and First I suspect are still scratching their heads on this one. I wonder if there is any software/hardware version differences of importance between the field used at Knoxville TN and the GTR West? If other fields are having difficulty with 1.4 firmware and how this compares to the configuration of TN?

Re: 2012 Field Comm. Issue Logs
 

Members of 2200 helped find and turn off these APs as well, it seemed to help a little but certainly didn't fix the problem. Went from two or three robots running simultaneously to four. Many robots were pulled off the field so practice matches could keep occurring. The downgrade to 1.21 seemed to do the trick for quals+elims.