bridge/cRio communications sporadic

[Tom Line]

Have you tried and seen this issue with the default robotic example?

We saw a very similar issue when we were overtaxing the C-rio with vision code and the processor couldn't keep up. Shortly after that (when we added more code) it started watchdogging.

Double check that the default code exhibits this same behavior. If it doesn't, I'd suggest reviewing your code and trying to cut out un-needed bits.

[Ziaholic]

Agreed, errant code is something that can adversely affect comm's ... but I think that the nature of this thread is more based on coming up from a cold/hard/OFF situation.

... not speaking for the other posters here, but my team's issues are only when establishing initial comm's ... once established, and everything stays powered-on, we've been A-OK.

[Gary.C]

Originally Posted by Ziaholic

... not speaking for the other posters here, but my team's issues are only when establishing initial comm's ... once established, and everything stays powered-on, we've been A-OK.

Same here...

[jhersh]

Originally Posted by Ziaholic

I'm glad to see that NI was able to reproduce it ... but sporadic errors are difficult to troubleshoot.

We've identified what's going on with the wireless bridge (gaming adapter). It seems that it maintains an arp table to direct traffic to its wired interface. It makes sense that it would want to filter what traffic it puts onto its wired interface to limit the wasted bandwidth from traffic destined for devices that are not on that segment of the network. However it only snoops arp traffic, so if the classmate has already cached the MAC address of the cRIO (i.e. you haven't stopped the driver station app on the classmate) then there is no arp request for the wireless bridge to snoop.

The reason it works sometimes is that there is a race condition between the cRIO network interface coming up and the bridge connecting. When the cRIO boots and brings up the network interface, it sends a gratuitous arp reply to announce itself. If the bridge is already booted, this gratuitous arp reply will cause the bridge adds the cRIO to its list-of-devices-connected-to-its-wired-interface and begin to put traffic destined for the cRIO on its wired interface (where the cRIO is) and therefore will work. If the bridge is not booted when this gratuitous arp reply is sent, then when it is booted, there is no arp traffic for the bridge to snoop and therefore won't forward traffic destined for the cRIO to its wired interface, making the cRIO unreachable. This very fact is why a warm boot of the cRIO fixes the problem (since there will be a new gratuitous arp reply from the cRIO).

The other way to address the issue is to force the classmate to resend its arp request for the cRIO so that the bridge can snoop it. After this happens, the bridge knows about the cRIO on its wired interface and starts forwarding traffic and everything works. This can be done by running "arp -d" on the classmate to clear the arp cache.

There will be an update to the driver station application that will clear the arp cache if it has trouble connecting to the cRIO.

Cheers!
-Joe

[Ziaholic]

Hooray! Kudos for such a thorough and detailed summary of the issue.

Those are some great details, and they seem to agree exactly with the symptoms that we've seen.

Thanks for the support! It's good to know that we've got a team of SuperNerds at NI that are there to help us out.

[PaulRevere]

Team 3183 is (was?) having a very similar issue. Our robot network was fine until yesterday, when it randomly started to have stuttering video feed from our camera, even when our robot was disabled. We then enabled the robot, and our DS threw countless “Watchdog Not Fed!” errors. Along with that, our compressor’s Spike was cycling on/off like mad because the code kept being aborted and resumed at a rapid pace (every time the cRIO lost communication with our DS, it would stop the program, and resume it as soon as it saw our DS again [less than a sec. later]) In addition, when this was occurring, our RSL seemed to be indicating the correct status (disabled or enabled) but due to the cycling being so rapid, it was VERY hard to tell. We tried to drive the robot with this, and (to no surprise) we had very little to no control over it. (Which was to be expected when the code kept stopping.) After about a dozen of this cycling, the DS would just say “no robot communication”, indicating that it no longer could see the robot. After maybe 8 seconds or so, the communications link would be back up, and we could do the same thing. We disconnected our Bridge from the cRIO’s Ethernet port (#1), and tethered the robot to our router, and it was fine. But, if we switched back to wireless, it resulted in the same thing. (Code aborts, stuttering video, etc.) This morning, one of our programmers tried the wireless again, still to no avail. He then turned off the robot, went over to get a drink, came back, turned on the robot, and low and behold, the wireless link was fine! Camera worked, no watchdog errors, and he could drive it with no issue. We’re at a loss for the reason behind this. Our biggest concern is that if this happens during a match, it could be the difference between losing a regional Qualifier, or going to Atlanta. (Not to mention, we didn’t try to E-Stop it during this problem, so even if our RSL was in fact saying the right thing, we don’t know if our Robot will even care about being E-Stop’d. (Especially if the code keeps restarting itself, as it wouldn’t know the difference between an E-Stop, or a “go” from FMS (not sure how this would play out with FMS thrown in, but using our DS, we don’t know. (not tested.))

[Alan Anderson]

Originally Posted by PaulRevere

We disconnected our Bridge from the cRIO’s Ethernet port (#1), and tethered the robot to our router, and it was fine. But, if we switched back to wireless, it resulted in the same thing. (Code aborts, stuttering video, etc.) This morning, one of our programmers tried the wireless again, still to no avail. He then turned off the robot, went over to get a drink, came back, turned on the robot, and low and behold, the wireless link was fine!

Please describe for us how you have power connected to the Bridge on your robot. Might the plug be slightly loose? Might the wires have an intermittent open where they are inserted into the white Wago connector, or perhaps even an intermittent short across them?

[PaulRevere]

Thanks for the fast response Alen! We have our Bridge connected directly to our Power Distribution board, using the Wago connector. We’ve removed and seated the Wago connector into the Power Distribution Board, and reinserted the wires into the Wago connector. I doubt it’s a power issue, as the bridge never reboots. (Like the Power, Wireless, and Link LEDs never go out.) Though the link/act light DOES stop flashing when these communication losses occurs. (This would indicate that there was no network traffic being passed over the Bridge.)

[jhersh]

Originally Posted by PaulRevere

Thanks for the fast response Alen! We have our Bridge connected directly to our Power Distribution board, using the Wago connector. We’ve removed and seated the Wago connector into the Power Distribution Board, and reinserted the wires into the Wago connector. I doubt it’s a power issue, as the bridge never reboots. (Like the Power, Wireless, and Link LEDs never go out.) Though the link/act light DOES stop flashing when these communication losses occurs. (This would indicate that there was no network traffic being passed over the Bridge.)

Make sure you don't have any other part of the robot close to the front of the gaming adapter. If the "security" button gets pressed it will disconnect from the access point. We saw this with a number of teams last year.

-Joe