2012 Field Comm. Issue Logs

[RyanN]

Originally Posted by waialua359

I'll let our CSA from our team chime in later with details.
Apparently, we are the only team with a 1.4 firmware at our regional, including our backup one.

In our only loss in a match today we lost comm (lag issue) for the last 30 seconds in our match.
It affected our ability to do the coop bridge also. Not knowing how it could've changed the outcome was frustrating.

If that is the field from Bayou, I would be interested in knowing what FW version the Loan Router has. We used it basically all Friday and Saturday.

Can you please get someone to check on this for us?

[DominickC]

When you guys say you lost comms with the FMS, do you mean a spike in lost packets, an exceedingly long trip time, or "No communication" displayed in the DS? I'm interested in the fact that so many people - including Team 23 - are having issues with the FMS.

[Greg McKaskle]

I've now heard of another team at WPI where the robot eas only able to spin in circles. Can someone from the drive team give details as to when it started during the match, what you were able to do, what eventually happened, etc?

I looked for footage but couldn't locate either. I was able to investigate one of these right after it happened, and expected it to be a chain, jammed, a speed controller, etc. I found a missing limit jumper on a jag that would have disabled one motor in one direction, but it wasn't clear that this was the issue, contributed to it, or was there all season and they were simply driving with a severely imbalanced drivetrain. They had four motors.

Before I explain what I suspect, I'd like to hear about recent issues involving spinning robots that weren't due to mechanical or speed controller issues.

Greg McKaskle

[DominickC]

I've just sent our drive team a message asking for further details. As soon as they get back to me, I'll let you know. Do you need/want any information on our router version and firmware?

If I remember correctly, the limit switch jumper just reverses the motor direction on the Jag?

EDIT: Our operator is telling me that they were able to control it for literally 2 seconds, enough time to step up to the controls and try to drive it. He went on to say that our two camera feeds froze the second we started spinning. He doesn't remember what the RSL was doing, but since the logs say we were in teleop, I would assume it was flashing the teleop sequence.

[Greg McKaskle]

I can't seem to find the Luminary manual that would be definitive, but I think the limit switch is used to enable or disable movement in a particular direction. They are also used for ramping if using the latest FW. If missing, I believe this jumper will stop and either coast or brake the motor and ignore inputs in that particular direction.

As for firmware version info, sure. Give it to me. But honestly, what I want most is a eye-witness account of the behavior or a video of the incident. I didn't see it, and even the one I saw happened unexpectedly, and I didn't see it begin. I walked to the side of the field to see if I could see a jammed ball, a chain obviously jumped off the sprocket, etc. After the match people were excited and talking about field faults and whatnot, and I couldn't get a good explanation of what took place. I don't want to bias someone into telling me symptoms that support what I think may have taken place, so I would like to have a description first.

Greg McKaskle

[DominickC]

I won't be able to get any FW info to you until Wednesday at the earliest, but I can tell the drive team to pop a comment in here with a full account.

[Navid Shafa]

Same situation just happened in Seattle Olympic, stay tuned for official post on the issues that happened during the finals.

Problems included: Response time lag from controls to robot through field management and lost camera relay for two robots on one of the alliances.

* I am not there, just talking to people who just had these issues, I'll let them clarify shortly.

[dsalvucci]

As Dom has requested, this is Team 23's recap of what happened to our driving situation.

Autonomous started and everything went exactly as planned. No problems whatsoever, everything checked out functional.

Teleop started and the team reached for the controls. The driver of the robot then went to turn the robot around and the robot began spinning uncontrollably. The drive team did initially have control during teleop for a couple seconds at most before the robot began to spin out of control (it appeared to be repeating the command to turn). The moment we lost control, the camera feed on the driver station had frozen and the joysticks were not being responsive.

A field tech came over and advised us to reboot the cRIO. I am not sure if the robot was spinning straight through to the point of rebooting or not. I think it was, but am not entirely sure. Upon rebooting, the joysticks became responsive again, however by the time robot code had been sent and communication was recovered, the match had only 1 second left.

That was the more physical memory of the situation we went through during our last elimination match. If you're looking for any messages or code problems, refer back to DominickC instead of myself as i was merely driving.

[Grim Tuesday]

Originally Posted by Gray Adams

Correct me if I'm wrong, but he's saying that anyone who used Blue 1 had issues.

This is correct.

We've been watching the field at buckeye, and some of the robots that we noted were failing at FLR, and have come to the conclusion that it was indeed coincidence.

[waialua359]

In Hawaii today, I can note at least 20 robots that had the lag issues today.
It cost us the opportunity for coopertition in our remaining 3 matches today.
Luckily for us, it always happened in the last 30 seconds where we already had the lead, since all 6 robots would experience lag and couldnt do anything at that point.
For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

I'm no expert in control systems, but smart enough to understand that something is definitely fishy here.
Mark Koors, our FTA is well aware of that situation now.

[Lavapicker]

Originally Posted by DominickC

When you guys say you lost comms with the FMS, do you mean a spike in lost packets, an exceedingly long trip time, or "No communication" displayed in the DS? I'm interested in the fact that so many people - including Team 23 - are having issues with the FMS.

The quarters in Hawaii were definitely infected with spooks! All six robots lost control in our first match for a few seconds near the middle and then lagged for about 15 before ending fine. The second match was fine for about 15 seconds and then we had no control of any buttons for the rest of the entire match. We could drive, barely, but all six robots experienced the same thing. I think the outcome was inevitable but I know I'd be upset if I was on the losing alliance with so many field issues. They seemed to fix it during a long break between quarters and semi's.

[Alan Anderson]

Originally Posted by waialua359

In Hawaii today, I can note at least 20 robots that had the lag issues today...For us personally, we swapped out our 1.4 firmware router with our backup that was downgraded to a lower firmware.
We had absolutely no issues after that.

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

[waialua359]

Originally Posted by Alan Anderson

There is now a lot of evidence, but still no good theory about what's going on. The virtual networks on the field should be isolating robots from each other, but something obviously isn't right. I don't think it can be called a "field fault" for which replaying a match would be appropriate, as it's being caused by a specific robot and would very likely happen again anyway (I heard that it indeed occurred for multiple replays at another event this weekend).

From what we saw at the Hawaii Regional, it's apparent that D-Link firmware 1.4 is a factor. Every match that had obvious multiple-robot control lag included a robot bearing that revision. It's not completely clear, since not every match with a 1.4-bearing robot had a problem. But we have something to work with, and we can try to prevent it from happening again by downgrading to revision 1.21 when necessary.

Alan,
the only puzzling part here, is that even after switching to the 1.21 as you noted BEFORE eliminations, the problem still existed for teams AND in matches that we werent even in.
The only permanent positive note is that it didnt happen to our team again.

I can only conclude that something much bigger is happening, and not just at our regional as noted in this thread.

[slijin]

I can say with certainty that we ran into lag issues with computer CPU consumption by the SmartDashboard; keeping it closed solved apparent lag problems, although the NYC FTA (Mark Mcleod) noted that we retained high trip times (~175 ms, as opposed to a standard 5-6 ms) throughout the entire event.

• Which event? NYC
• Wireless bridge radio HW revision? (A1, A2, or B1) A1
• Radio firmware rev? 1.21
• Programming language? Java
• Using a dashboard app? n/a
• Using vision with Axis Camera and cRIO processing? Ran a live stream with no processing at default fps
• Using vision with driver station processing? DS display; nothing else.
• Did you have the radio mounted near motors/large metal structure? Near two Black Jaguars, above the 12V/5V converter*.
• Using classmate as driver station? 2go E11 Classmate, not the KoP one
• 4 or 8 slot cRIO? (FRC-CRIO2 or Old version) 4-slot
• CAN Jaguars? What FW version? Yes, rev92.

*This issue is something I realized post-NY. As Alan has pointed out, this could possibly produce issues; it will be fixed when we go to CT.

[DonRotolo]

Some data points:

At Rutgers to a small extent and at NYC to a much greater extent, we were seeing the robot stop responding for a second to several seconds, somewhat randomly. This would occur both tethered and via radio (!)

Reviewing the logs from NYC we found our latency was around 170 mSec ('good' would be <10 mSec).

Our 8-slot cRio showed a CPU usage of 50-60%, which is reasonable.

Carefully working to recreate the issue on our prototype robot, we ultimately found that out driver station CPU was maxing out.

We have not yet deemed this issue resolved, but we believe we're almost there. We'll find out when we unbag this week in anticipation of Mount Olive.

(I don't have the details requested in the OP, but can get them if that will help. All our cRios are 8-slot, we use LabView, and our radios are all Rev A).