bridge/cRio communications sporadic
 

I thought I'd seek input here on this rather intriguing problem:

When programming a prototypes system today, I discovered an issue with the wireless connection. What would happen was that when the electronics board was powered down and restarted, the wireless connection would only come back up about 50% of the time. On further investigation, this proved to be due to a malfunction in the link between the bridge and the cRio--ICMP pings sent to the cRio when the communication was down would just never get there, even though we could successfully ping the bridge.Changing out the cable did not affect the problem, and the robot runs fine tethered, which suggests to me that the problem lies with the bridge itself. After killing and reinstating power several times, the problem is resolved until the board is powered down again, after which it may or may not work.

Has anyone else had this issue or know of any solutions/workarounds? Our router and bridge settings are the same as we've been using to control our 2009 robot, and all our software is up-to-date.

Re: bridge/cRio communications sporadic
 

This sums up my day today too and sometimes it wouldn't work. Any solutions would be helpful.

Re: bridge/cRio communications sporadic
 

I think I was having this problem today as well. The robot signal light was blinking like it had no connection(when in fact it did), but when enabled would pause for about 50ms every 30seconds. It was fine when programming, but I should start looking into fixing it sometime soon.

Re: bridge/cRio communications sporadic
 

I haven't hooked up the signal light yet, but as I recall it has a certain flash pattern this year that actually indicates that it is in teleop mode.

Re: bridge/cRio communications sporadic
 

We have been having communications glitches that temporarily shut down the motors (as if disabled,) but then they resume after a fraction of a second.

Occaisonally, we have had it shut down entirely, but this may be unrelated. Out light wouldn't blink, and the driver station would report that there was no communication. Once, I also noticed the lights for the power regulators were out (on the power distribution board,) so I think this may be an entirely unrelated problem.

I hope this gets solved.

Re: bridge/cRio communications sporadic
 

My team has seen something like this also.

I'm wondering if it is a network configuration issue.

When we are developing, and the 'bot is close to the driverstation, we can sometimes switch from tethered to wireless, and in the process, lose communications for a while. Then, without changing anything, it starts working again.


My theory is that it is network related ... but I need to know something about the cRIO's gateway settings before I can conclude anything.

Here's my theory:
- When wireless, the cRIO (with IP addy: 10.xx.yy.2) uses the bridge (with IP addy: 10.xx.yy.1) as its gateway
- When tethered, the cRIO goes straight to the Linksys wireless router via one of the 4 hard-wired ethernet jacks
- If you leave the bridge powered on, even when you are tethered and not using it, then it could be confusing the network connectivity between the devices. (that is, when the bridge is on and unused, it could be confusing the route between the driverstation and the cRIO)


I need to know if the cRIO gateway setting is hard-set to 10.xx.yy.1. If so, then going from tethered to un-tethered could cause some headaches, especially when the bridge is left on.


I plan on doing some more investigation this afternoon at our team meeting. Stay tuned.

Re: bridge/cRio communications sporadic
 

That problem might be due to a persistent cache of the routing table for active IP addresses on the router.

Re: bridge/cRio communications sporadic
 

I have also been having the troubles of the motors turning off for like a second and robot code and communication go red but then go green after a little bit. Also the communication only works like 50% of the time and sometimes it takes a little more than 2 minutes to load everything! I will stay tuned for updates or solutions.

Re: bridge/cRio communications sporadic
 

I'm still not positive about the cRIO gateway setting, or if it is related to the problem, however, some of the guys on my team are theorizing that it could be a startup issue.

Here's a timeline, a few educated guesses, and maybe a fix: (when operating in a wireless mode)
- apply power to the bot
- cRIO comes up first, wireless bridge still booting
- cRIO network comes up, doesn't see an "active" bridge, and times out
- wireless bridge comes up, but too late for cRIO
- no ethernet connection
- reset the cRIO (using a paper clip on the reset button)
- wireless ethernet connects almost immediately

I didn't get much time with the 'bot yesterday, so it was hard to troubleshoot, but when we followed the timeline above, we had no disconnection issues at all.

I'd like to see if the cRIO will re-attempt after 30 or 60 seconds ... stay tuned.

If this theory is correct, and we want to avoid delays at competitions, then we might expect a cRIO firmware patch to slow down its bootup process and wait a little while for the bridge to come up.

Like I said above, this is all just a theory. I hope that it either gets patched, or I'm wrong, because I don't want to have to paper-clip-reset the cRIO every time we turn it on ...


edit: I just posted a question to the NI forums. Perhaps their engineers could shed some light on this for us.
edit2: here's a link to the NI forum question (no activity yet) http://decibel.ni.com/content/thread/5712?tstart=0

Re: bridge/cRio communications sporadic
 

I think we had a bad game bridge last year or something because we were having similar issues. We also lost coms multiple times and changing the game bridge fixed our issues. We lost a couple matches because we just died in the middle...:ahh:

So needless to say, we haven't been very impressed with the game bridges so far...

Re: bridge/cRio communications sporadic
 

We have had similar experiences. The process we now have in place to always ensure that everything talks is to turn off the robot and unplug our router. Turn on the robot and wait about 10 seconds and then plug the router back in. Then wait for the connection to be made and everything works from that point.

We're using the same router that the rookies were given this year the 160. We bought a whole new control system since we wanted to keep last year's awesome robot demo and fundraiser worthy. We are using a factory reconditioned/recertified game bridge because we couldn't find a new one.

We were going nuts trying to figure out why sometimes it would connect to the wireless bridge and sometimes why it wouldn't. That procedure above is what has worked consistently for us. I just hope we don't have similar issues at the regionals.

Re: bridge/cRio communications sporadic
 

I am pretty sure this is right and I have had to press the reset button on our cRIO too because it won't catch the bridge. I hope there is a fix. Thanks for pointing this out.

Re: bridge/cRio communications sporadic
 

We're looking into this issue. We've been able to reproduce it, but have not yet identified the cause.

-Joe

Re: bridge/cRio communications sporadic
 

Thursday we had no issues. It came right up and connected pretty quickly. We'll keep our eyes on it and report any more details when/if we see it happen again.

I'm glad to see that NI was able to reproduce it ... but sporadic errors are difficult to troubleshoot.

Re: bridge/cRio communications sporadic
 

We've been having this issue, too. We're using last year's cRIO, Bridge and Router. It's sporadic and hard to reproduce.

Re: bridge/cRio communications sporadic
 

Have you tried and seen this issue with the default robotic example?

We saw a very similar issue when we were overtaxing the C-rio with vision code and the processor couldn't keep up. Shortly after that (when we added more code) it started watchdogging.

Double check that the default code exhibits this same behavior. If it doesn't, I'd suggest reviewing your code and trying to cut out un-needed bits.

Re: bridge/cRio communications sporadic
 

Agreed, errant code is something that can adversely affect comm's ... but I think that the nature of this thread is more based on coming up from a cold/hard/OFF situation.

... not speaking for the other posters here, but my team's issues are only when establishing initial comm's ... once established, and everything stays powered-on, we've been A-OK.

Re: bridge/cRio communications sporadic
 

Same here...

Re: bridge/cRio communications sporadic
 

We've identified what's going on with the wireless bridge (gaming adapter). It seems that it maintains an arp table to direct traffic to its wired interface. It makes sense that it would want to filter what traffic it puts onto its wired interface to limit the wasted bandwidth from traffic destined for devices that are not on that segment of the network. However it only snoops arp traffic, so if the classmate has already cached the MAC address of the cRIO (i.e. you haven't stopped the driver station app on the classmate) then there is no arp request for the wireless bridge to snoop.

The reason it works sometimes is that there is a race condition between the cRIO network interface coming up and the bridge connecting. When the cRIO boots and brings up the network interface, it sends a gratuitous arp reply to announce itself. If the bridge is already booted, this gratuitous arp reply will cause the bridge adds the cRIO to its list-of-devices-connected-to-its-wired-interface and begin to put traffic destined for the cRIO on its wired interface (where the cRIO is) and therefore will work. If the bridge is not booted when this gratuitous arp reply is sent, then when it is booted, there is no arp traffic for the bridge to snoop and therefore won't forward traffic destined for the cRIO to its wired interface, making the cRIO unreachable. This very fact is why a warm boot of the cRIO fixes the problem (since there will be a new gratuitous arp reply from the cRIO).

The other way to address the issue is to force the classmate to resend its arp request for the cRIO so that the bridge can snoop it. After this happens, the bridge knows about the cRIO on its wired interface and starts forwarding traffic and everything works. This can be done by running "arp -d" on the classmate to clear the arp cache.

There will be an update to the driver station application that will clear the arp cache if it has trouble connecting to the cRIO.

Cheers!
-Joe

Re: bridge/cRio communications sporadic
 

Hooray! Kudos for such a thorough and detailed summary of the issue.

Those are some great details, and they seem to agree exactly with the symptoms that we've seen.

Thanks for the support! It's good to know that we've got a team of SuperNerds at NI that are there to help us out.

Re: bridge/cRio communications sporadic
 

Team 3183 is (was?) having a very similar issue. Our robot network was fine until yesterday, when it randomly started to have stuttering video feed from our camera, even when our robot was disabled. We then enabled the robot, and our DS threw countless “Watchdog Not Fed!” errors. Along with that, our compressor’s Spike was cycling on/off like mad because the code kept being aborted and resumed at a rapid pace (every time the cRIO lost communication with our DS, it would stop the program, and resume it as soon as it saw our DS again [less than a sec. later]) In addition, when this was occurring, our RSL seemed to be indicating the correct status (disabled or enabled) but due to the cycling being so rapid, it was VERY hard to tell. We tried to drive the robot with this, and (to no surprise) we had very little to no control over it. (Which was to be expected when the code kept stopping.) After about a dozen of this cycling, the DS would just say “no robot communication”, indicating that it no longer could see the robot. After maybe 8 seconds or so, the communications link would be back up, and we could do the same thing. We disconnected our Bridge from the cRIO’s Ethernet port (#1), and tethered the robot to our router, and it was fine. But, if we switched back to wireless, it resulted in the same thing. (Code aborts, stuttering video, etc.) This morning, one of our programmers tried the wireless again, still to no avail. He then turned off the robot, went over to get a drink, came back, turned on the robot, and low and behold, the wireless link was fine! Camera worked, no watchdog errors, and he could drive it with no issue. We’re at a loss for the reason behind this. Our biggest concern is that if this happens during a match, it could be the difference between losing a regional Qualifier, or going to Atlanta. (Not to mention, we didn’t try to E-Stop it during this problem, so even if our RSL was in fact saying the right thing, we don’t know if our Robot will even care about being E-Stop’d. (Especially if the code keeps restarting itself, as it wouldn’t know the difference between an E-Stop, or a “go” from FMS (not sure how this would play out with FMS thrown in, but using our DS, we don’t know. (not tested.))

Re: bridge/cRio communications sporadic
 

Please describe for us how you have power connected to the Bridge on your robot. Might the plug be slightly loose? Might the wires have an intermittent open where they are inserted into the white Wago connector, or perhaps even an intermittent short across them?

Re: bridge/cRio communications sporadic
 

Thanks for the fast response Alen! We have our Bridge connected directly to our Power Distribution board, using the Wago connector. We’ve removed and seated the Wago connector into the Power Distribution Board, and reinserted the wires into the Wago connector. I doubt it’s a power issue, as the bridge never reboots. (Like the Power, Wireless, and Link LEDs never go out.) Though the link/act light DOES stop flashing when these communication losses occurs. (This would indicate that there was no network traffic being passed over the Bridge.)

Re: bridge/cRio communications sporadic
 

Make sure you don't have any other part of the robot close to the front of the gaming adapter. If the "security" button gets pressed it will disconnect from the access point. We saw this with a number of teams last year.

-Joe