2012 Field Comm. Issue Logs

[Kevin Sevcik]

Erm. In addition to the answers to Tom's above questions, you should probably also post the actual issue/symptoms you were experiencing, since that varies from long pings and delays to intermittent or no Comms.

[Joe Ross]

For rev A radios, it would be helpful to have the firmware version (1.21, 1.4, or maybe something else).

It would also be helpful for a description on the problem, and some representative DS log files (in Users\Public\Documents\FRC\Log Files)

[Tom Bottiglieri]

Originally Posted by Joe Ross

For rev A radios, it would be helpful to have the firmware version (1.21, 1.4, or maybe something else).

It would also be helpful for a description on the problem, and some representative DS log files (in Users\Public\Documents\FRC\Log Files)

Originally Posted by Kevin Sevcik

Erm. In addition to the answers to Tom's above questions, you should probably also post the actual issue/symptoms you were experiencing, since that varies from long pings and delays to intermittent or no Comms.

Thanks, can't believe I forgot those things. Edited the orignal post.

[Tom Bottiglieri]

I've also edited the original post to include a question about 4 vs. 8 slot cRIO.

[dez250]

Tom I would suggest to also check if CAN-bus communication was used, and if so, what version firmware was loaded on the jaguars.

[Grim Tuesday]

We have correlated (note, not causation, just a hypothesis) multiple field failures (full connection loss) to the Blue alliance station number 1. At FLR, our robot failed multiple times in that station only, as did our alliance partners. Further supporting this theory is the fact that during the qualification matches, the red alliance won 61% of matches. Considering the randomized aspect of schedules, this seems somewhat outside the 50% norm it should be. The field is going to Buckeye this week, and DC next week. We'll be at DC, watching it, but I would suggest anyone at Buckeye to keep an eye on that station. It might have just been assembled at Fingerlakes with a wire loose, or it could be a problem in its hardware. Or this whole thing could just be coincidence. Just something to keep in mind.

[Alan Anderson]

Originally Posted by Grim Tuesday

We have correlated (note, not causation, just a hypothesis) multiple field failures (full connection loss) to the Blue alliance station number 1. At FLR, our robot failed multiple times in that station only, as did our alliance partners.

I don't understand. It sounds like you're saying that more than one robot was using the Blue 1 alliance station at the same time. While that's likely not what you meant, it would certainly mess with their ability to connect!

[Gray Adams]

Originally Posted by Alan Anderson

I don't understand. It sounds like you're saying that more than one robot was using the Blue 1 alliance station at the same time. While that's likely not what you meant, it would certainly mess with their ability to connect!

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

[Grim Tuesday]

Originally Posted by Gray Adams

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

This is correct.

We've been watching the field at buckeye, and some of the robots that we noted were failing at FLR, and have come to the conclusion that it was indeed coincidence.

[waialua359]

In Hawaii today, I can note at least 20 robots that had the lag issues today.
It cost us the opportunity for coopertition in our remaining 3 matches today.
Luckily for us, it always happened in the last 30 seconds where we already had the lead, since all 6 robots would experience lag and couldnt do anything at that point.
For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

I'm no expert in control systems, but smart enough to understand that something is definitely fishy here.
Mark Koors, our FTA is well aware of that situation now.

[Lavapicker]

Originally Posted by DominickC

When you guys say you lost comms with the FMS, do you mean a spike in lost packets, an exceedingly long trip time, or "No communication" displayed in the DS? I'm interested in the fact that so many people - including Team 23 - are having issues with the FMS.

The quarters in Hawaii were definitely infected with spooks! All six robots lost control in our first match for a few seconds near the middle and then lagged for about 15 before ending fine. The second match was fine for about 15 seconds and then we had no control of any buttons for the rest of the entire match. We could drive, barely, but all six robots experienced the same thing. I think the outcome was inevitable but I know I'd be upset if I was on the losing alliance with so many field issues. They seemed to fix it during a long break between quarters and semi's.

[Alan Anderson]

Originally Posted by waialua359

In Hawaii today, I can note at least 20 robots that had the lag issues today...For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

[waialua359]

Originally Posted by Alan Anderson

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.
The only permanent positive note is that it didnt happen to our team again.

I can only conclude that something much bigger is happening, and not just at our regional as noted in this thread.

[Alan Anderson]

Originally Posted by waialua359

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.

One other team at the Hawaii Regional was identified as having 1.4, but not until teams were preparing to pack up at the end of the event. I wasn't aware at the time that the WPA programming kiosk kept records of D-Link firmware revisions. Looking at its logs would have been a much quicker way to find the potentially problematic 'bots.

The larger issue is still mysterious, but it does seem to involve the order connections are established.

[rbtying]

Team 846:

Programming language? C++
Using a dashboard app? Is it SmartDashboard? Started with SmartDashboard, but removed all logging during competition (did not solve problem)
Using vision with Axis Camera and cRIO processing? No
Using vision with driver station processing? No
Did you have the radio mounted near motors/large metal structure? No
Using classmate (or similarly slow computer) as driver station? Yes

We're using a Rev A radio, and tried firmwares 1.21 and 1.4, with the same results in both.

Problem: Intermittent connection on the field, losing comms generally around the end of autonomous, and randomly thereafter. Total time actually connected varies between 10s to 20s; packet losses of 25 / 50 are noted immediately before the communications goes down.