2012 Field Comm. Issue Logs

[Grim Tuesday]

We have correlated (note, not causation, just a hypothesis) multiple field failures (full connection loss) to the Blue alliance station number 1. At FLR, our robot failed multiple times in that station only, as did our alliance partners. Further supporting this theory is the fact that during the qualification matches, the red alliance won 61% of matches. Considering the randomized aspect of schedules, this seems somewhat outside the 50% norm it should be. The field is going to Buckeye this week, and DC next week. We'll be at DC, watching it, but I would suggest anyone at Buckeye to keep an eye on that station. It might have just been assembled at Fingerlakes with a wire loose, or it could be a problem in its hardware. Or this whole thing could just be coincidence. Just something to keep in mind.

[Alan Anderson]

Originally Posted by Grim Tuesday

We have correlated (note, not causation, just a hypothesis) multiple field failures (full connection loss) to the Blue alliance station number 1. At FLR, our robot failed multiple times in that station only, as did our alliance partners.

I don't understand. It sounds like you're saying that more than one robot was using the Blue 1 alliance station at the same time. While that's likely not what you meant, it would certainly mess with their ability to connect!

[Gray Adams]

Originally Posted by Alan Anderson

I don't understand. It sounds like you're saying that more than one robot was using the Blue 1 alliance station at the same time. While that's likely not what you meant, it would certainly mess with their ability to connect!

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

[Grim Tuesday]

Originally Posted by Gray Adams

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

This is correct.

We've been watching the field at buckeye, and some of the robots that we noted were failing at FLR, and have come to the conclusion that it was indeed coincidence.

[waialua359]

In Hawaii today, I can note at least 20 robots that had the lag issues today.
It cost us the opportunity for coopertition in our remaining 3 matches today.
Luckily for us, it always happened in the last 30 seconds where we already had the lead, since all 6 robots would experience lag and couldnt do anything at that point.
For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

I'm no expert in control systems, but smart enough to understand that something is definitely fishy here.
Mark Koors, our FTA is well aware of that situation now.

[Lavapicker]

Originally Posted by DominickC

When you guys say you lost comms with the FMS, do you mean a spike in lost packets, an exceedingly long trip time, or "No communication" displayed in the DS? I'm interested in the fact that so many people - including Team 23 - are having issues with the FMS.

The quarters in Hawaii were definitely infected with spooks! All six robots lost control in our first match for a few seconds near the middle and then lagged for about 15 before ending fine. The second match was fine for about 15 seconds and then we had no control of any buttons for the rest of the entire match. We could drive, barely, but all six robots experienced the same thing. I think the outcome was inevitable but I know I'd be upset if I was on the losing alliance with so many field issues. They seemed to fix it during a long break between quarters and semi's.

[Alan Anderson]

Originally Posted by waialua359

In Hawaii today, I can note at least 20 robots that had the lag issues today...For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

[waialua359]

Originally Posted by Alan Anderson

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.
The only permanent positive note is that it didnt happen to our team again.

I can only conclude that something much bigger is happening, and not just at our regional as noted in this thread.

[Alan Anderson]

Originally Posted by waialua359

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.

One other team at the Hawaii Regional was identified as having 1.4, but not until teams were preparing to pack up at the end of the event. I wasn't aware at the time that the WPA programming kiosk kept records of D-Link firmware revisions. Looking at its logs would have been a much quicker way to find the potentially problematic 'bots.

The larger issue is still mysterious, but it does seem to involve the order connections are established.

[Blackphantom91]

Durring elemination matches we ad 1985 connected last and we didn't have problems. the order was really prevalent or when they did a reset there were no problems. How do teams correct the problem if its all about revisions though? We could go out to the field and it could all be messed up at champs? hope we can get to the bottom of this.

[catsylve]

During the St. Louis Regional, 1985 experienced what seemed to be connection issues throughout the entire first part of the competition. Those issues turned out to be a hardware problem. We went through the process of replacing our classmate first, then our entire power distribution block. When these did not fix it, we had to replace the CRio with a replacement, although the one we had been using was a brand new one. This completely fixed our problems and we returned the CRio to NI and they replaced it still under warranty. It made for an incredibly frustrating day, but literally our connection issues turned out to be from a bad part. All of this did happen after the firmware issues with the radio were corrected.

[dkearle]

We (Team 1280) had another similar crash in one of our matches at Silicon Valley today. Back in the pit we crashed again, this time with NetConsole running and were able to capture the following exception and error messages on NetConsole:

Exception current instruction address: 0x02beb868
Machine Status Register: 0x0008b012
Condition Register: 0x20000084
Task: 0x2b30980 "FRC_NetworkTablesWriteTask"
0x2b30980 (FRC_NetworkTablesWriteTask): task 0x2b30980 has had a failure and has been stopped.
0x2b30980 (FRC_NetworkTablesWriteTask): fatal kernel task-level exception!
task 0x2b247e0 (FRC_NetworkTablesWatchdogTask) deleted: errno=0 (0) status=0 (0)

>>>>ERROR: A timeout has been exceeded: NetworkTables watchdog expired... disconnecting ...in WatchdogTaskRun() in C:/WindRiver/workspace/WPILib/NetworkTables/Connection.cpp at line 565

>>>>ERROR: A timeout has been exceeded: NetworkTables watchdog expired... disconnecting ...in WatchdogTaskRun() in C:/WindRiver/workspace/WPILib/NetworkTables/Connection.cpp at line 565

>>>>ERROR: A timeout has been exceeded: NetworkTables watchdog expired... disconnecting ...in WatchdogTaskRun() in C:/WindRiver/workspace/WPILib/NetworkTables/Connection.cpp at line 565
task 0x259d568 (FTP Server Connection Thread) deleted: errno=0 (0) status=0 (0)
?

We use SmartDashboard to display some data from our robot using the Log() method. We do not access the NetworkTables class independent from SmartDashboard..

Since we are not able to run NetConsole during matches, we have no idea if we get this same exception every time we crash, but we'll keep looking for it when we are tethered in the pit. Because of this error, we have removed our use of SmartDashboard from our robot code.

[Gray Adams]

I don't have many details right now, but we had to sit out 5 matches at SVR today. I'll try to post back with some details tomorrow.