bridge/cRio communications sporadic

[nathanww]

I thought I'd seek input here on this rather intriguing problem:

When programming a prototypes system today, I discovered an issue with the wireless connection. What would happen was that when the electronics board was powered down and restarted, the wireless connection would only come back up about 50% of the time. On further investigation, this proved to be due to a malfunction in the link between the bridge and the cRio--ICMP pings sent to the cRio when the communication was down would just never get there, even though we could successfully ping the bridge.Changing out the cable did not affect the problem, and the robot runs fine tethered, which suggests to me that the problem lies with the bridge itself. After killing and reinstating power several times, the problem is resolved until the board is powered down again, after which it may or may not work.

Has anyone else had this issue or know of any solutions/workarounds? Our router and bridge settings are the same as we've been using to control our 2009 robot, and all our software is up-to-date.

[Gary.C]

Originally Posted by nathanww

I thought I'd seek input here on this rather intriguing problem:

When programming a prototypes system today, I discovered an issue with the wireless connection. What would happen was that when the electronics board was powered down and restarted, the wireless connection would only come back up about 50% of the time. On further investigation, this proved to be due to a malfunction in the link between the bridge and the cRio--ICMP pings sent to the cRio when the communication was down would just never get there, even though we could successfully ping the bridge.Changing out the cable did not affect the problem, and the robot runs fine tethered, which suggests to me that the problem lies with the bridge itself. After killing and reinstating power several times, the problem is resolved until the board is powered down again, after which it may or may not work.

Has anyone else had this issue or know of any solutions/workarounds? Our router and bridge settings are the same as we've been using to control our 2009 robot, and all our software is up-to-date.

This sums up my day today too and sometimes it wouldn't work. Any solutions would be helpful.

[Boydean]

I think I was having this problem today as well. The robot signal light was blinking like it had no connection(when in fact it did), but when enabled would pause for about 50ms every 30seconds. It was fine when programming, but I should start looking into fixing it sometime soon.

[nathanww]

I haven't hooked up the signal light yet, but as I recall it has a certain flash pattern this year that actually indicates that it is in teleop mode.

[flameout]

We have been having communications glitches that temporarily shut down the motors (as if disabled,) but then they resume after a fraction of a second.

Occaisonally, we have had it shut down entirely, but this may be unrelated. Out light wouldn't blink, and the driver station would report that there was no communication. Once, I also noticed the lights for the power regulators were out (on the power distribution board,) so I think this may be an entirely unrelated problem.

I hope this gets solved.

[Ziaholic]

My team has seen something like this also.

I'm wondering if it is a network configuration issue.

When we are developing, and the 'bot is close to the driverstation, we can sometimes switch from tethered to wireless, and in the process, lose communications for a while. Then, without changing anything, it starts working again.


My theory is that it is network related ... but I need to know something about the cRIO's gateway settings before I can conclude anything.

Here's my theory:
- When wireless, the cRIO (with IP addy: 10.xx.yy.2) uses the bridge (with IP addy: 10.xx.yy.1) as its gateway
- When tethered, the cRIO goes straight to the Linksys wireless router via one of the 4 hard-wired ethernet jacks
- If you leave the bridge powered on, even when you are tethered and not using it, then it could be confusing the network connectivity between the devices. (that is, when the bridge is on and unused, it could be confusing the route between the driverstation and the cRIO)


I need to know if the cRIO gateway setting is hard-set to 10.xx.yy.1. If so, then going from tethered to un-tethered could cause some headaches, especially when the bridge is left on.


I plan on doing some more investigation this afternoon at our team meeting. Stay tuned.

[Mark McLeod]

That problem might be due to a persistent cache of the routing table for active IP addresses on the router.

[Gary.C]

I have also been having the troubles of the motors turning off for like a second and robot code and communication go red but then go green after a little bit. Also the communication only works like 50% of the time and sometimes it takes a little more than 2 minutes to load everything! I will stay tuned for updates or solutions.

[Ziaholic]

I'm still not positive about the cRIO gateway setting, or if it is related to the problem, however, some of the guys on my team are theorizing that it could be a startup issue.

Here's a timeline, a few educated guesses, and maybe a fix: (when operating in a wireless mode)
- apply power to the bot
- cRIO comes up first, wireless bridge still booting
- cRIO network comes up, doesn't see an "active" bridge, and times out
- wireless bridge comes up, but too late for cRIO
- no ethernet connection
- reset the cRIO (using a paper clip on the reset button)
- wireless ethernet connects almost immediately

I didn't get much time with the 'bot yesterday, so it was hard to troubleshoot, but when we followed the timeline above, we had no disconnection issues at all.

I'd like to see if the cRIO will re-attempt after 30 or 60 seconds ... stay tuned.

If this theory is correct, and we want to avoid delays at competitions, then we might expect a cRIO firmware patch to slow down its bootup process and wait a little while for the bridge to come up.

Like I said above, this is all just a theory. I hope that it either gets patched, or I'm wrong, because I don't want to have to paper-clip-reset the cRIO every time we turn it on ...


edit: I just posted a question to the NI forums. Perhaps their engineers could shed some light on this for us.
edit2: here's a link to the NI forum question (no activity yet) http://decibel.ni.com/content/thread/5712?tstart=0

[Bryscus]

I think we had a bad game bridge last year or something because we were having similar issues. We also lost coms multiple times and changing the game bridge fixed our issues. We lost a couple matches because we just died in the middle...

So needless to say, we haven't been very impressed with the game bridges so far...

[sircedric4]

We have had similar experiences. The process we now have in place to always ensure that everything talks is to turn off the robot and unplug our router. Turn on the robot and wait about 10 seconds and then plug the router back in. Then wait for the connection to be made and everything works from that point.

We're using the same router that the rookies were given this year the 160. We bought a whole new control system since we wanted to keep last year's awesome robot demo and fundraiser worthy. We are using a factory reconditioned/recertified game bridge because we couldn't find a new one.

We were going nuts trying to figure out why sometimes it would connect to the wireless bridge and sometimes why it wouldn't. That procedure above is what has worked consistently for us. I just hope we don't have similar issues at the regionals.

[Gary.C]

Originally Posted by Ziaholic

I'm still not positive about the cRIO gateway setting, or if it is related to the problem, however, some of the guys on my team are theorizing that it could be a startup issue.

Here's a timeline, a few educated guesses, and maybe a fix: (when operating in a wireless mode)
- apply power to the bot
- cRIO comes up first, wireless bridge still booting
- cRIO network comes up, doesn't see an "active" bridge, and times out
- wireless bridge comes up, but too late for cRIO
- no ethernet connection
- reset the cRIO (using a paper clip on the reset button)
- wireless ethernet connects almost immediately

I didn't get much time with the 'bot yesterday, so it was hard to troubleshoot, but when we followed the timeline above, we had no disconnection issues at all.

I'd like to see if the cRIO will re-attempt after 30 or 60 seconds ... stay tuned.

If this theory is correct, and we want to avoid delays at competitions, then we might expect a cRIO firmware patch to slow down its bootup process and wait a little while for the bridge to come up.

Like I said above, this is all just a theory. I hope that it either gets patched, or I'm wrong, because I don't want to have to paper-clip-reset the cRIO every time we turn it on ...


edit: I just posted a question to the NI forums. Perhaps their engineers could shed some light on this for us.
edit2: here's a link to the NI forum question (no activity yet) http://decibel.ni.com/content/thread/5712?tstart=0

I am pretty sure this is right and I have had to press the reset button on our cRIO too because it won't catch the bridge. I hope there is a fix. Thanks for pointing this out.

[jhersh]

We're looking into this issue. We've been able to reproduce it, but have not yet identified the cause.

-Joe

[PaulRevere]

Team 3183 is (was?) having a very similar issue. Our robot network was fine until yesterday, when it randomly started to have stuttering video feed from our camera, even when our robot was disabled. We then enabled the robot, and our DS threw countless “Watchdog Not Fed!” errors. Along with that, our compressor’s Spike was cycling on/off like mad because the code kept being aborted and resumed at a rapid pace (every time the cRIO lost communication with our DS, it would stop the program, and resume it as soon as it saw our DS again [less than a sec. later]) In addition, when this was occurring, our RSL seemed to be indicating the correct status (disabled or enabled) but due to the cycling being so rapid, it was VERY hard to tell. We tried to drive the robot with this, and (to no surprise) we had very little to no control over it. (Which was to be expected when the code kept stopping.) After about a dozen of this cycling, the DS would just say “no robot communication”, indicating that it no longer could see the robot. After maybe 8 seconds or so, the communications link would be back up, and we could do the same thing. We disconnected our Bridge from the cRIO’s Ethernet port (#1), and tethered the robot to our router, and it was fine. But, if we switched back to wireless, it resulted in the same thing. (Code aborts, stuttering video, etc.) This morning, one of our programmers tried the wireless again, still to no avail. He then turned off the robot, went over to get a drink, came back, turned on the robot, and low and behold, the wireless link was fine! Camera worked, no watchdog errors, and he could drive it with no issue. We’re at a loss for the reason behind this. Our biggest concern is that if this happens during a match, it could be the difference between losing a regional Qualifier, or going to Atlanta. (Not to mention, we didn’t try to E-Stop it during this problem, so even if our RSL was in fact saying the right thing, we don’t know if our Robot will even care about being E-Stop’d. (Especially if the code keeps restarting itself, as it wouldn’t know the difference between an E-Stop, or a “go” from FMS (not sure how this would play out with FMS thrown in, but using our DS, we don’t know. (not tested.))

[Alan Anderson]

Originally Posted by PaulRevere

We disconnected our Bridge from the cRIO’s Ethernet port (#1), and tethered the robot to our router, and it was fine. But, if we switched back to wireless, it resulted in the same thing. (Code aborts, stuttering video, etc.) This morning, one of our programmers tried the wireless again, still to no avail. He then turned off the robot, went over to get a drink, came back, turned on the robot, and low and behold, the wireless link was fine!

Please describe for us how you have power connected to the Bridge on your robot. Might the plug be slightly loose? Might the wires have an intermittent open where they are inserted into the white Wago connector, or perhaps even an intermittent short across them?