bridge/cRio communications sporadic

[nathanww]

I thought I'd seek input here on this rather intriguing problem:

When programming a prototypes system today, I discovered an issue with the wireless connection. What would happen was that when the electronics board was powered down and restarted, the wireless connection would only come back up about 50% of the time. On further investigation, this proved to be due to a malfunction in the link between the bridge and the cRio--ICMP pings sent to the cRio when the communication was down would just never get there, even though we could successfully ping the bridge.Changing out the cable did not affect the problem, and the robot runs fine tethered, which suggests to me that the problem lies with the bridge itself. After killing and reinstating power several times, the problem is resolved until the board is powered down again, after which it may or may not work.

Has anyone else had this issue or know of any solutions/workarounds? Our router and bridge settings are the same as we've been using to control our 2009 robot, and all our software is up-to-date.

[Gary.C]

Originally Posted by nathanww

I thought I'd seek input here on this rather intriguing problem:

When programming a prototypes system today, I discovered an issue with the wireless connection. What would happen was that when the electronics board was powered down and restarted, the wireless connection would only come back up about 50% of the time. On further investigation, this proved to be due to a malfunction in the link between the bridge and the cRio--ICMP pings sent to the cRio when the communication was down would just never get there, even though we could successfully ping the bridge.Changing out the cable did not affect the problem, and the robot runs fine tethered, which suggests to me that the problem lies with the bridge itself. After killing and reinstating power several times, the problem is resolved until the board is powered down again, after which it may or may not work.

Has anyone else had this issue or know of any solutions/workarounds? Our router and bridge settings are the same as we've been using to control our 2009 robot, and all our software is up-to-date.

This sums up my day today too and sometimes it wouldn't work. Any solutions would be helpful.

[Boydean]

I think I was having this problem today as well. The robot signal light was blinking like it had no connection(when in fact it did), but when enabled would pause for about 50ms every 30seconds. It was fine when programming, but I should start looking into fixing it sometime soon.

[nathanww]

I haven't hooked up the signal light yet, but as I recall it has a certain flash pattern this year that actually indicates that it is in teleop mode.

[flameout]

We have been having communications glitches that temporarily shut down the motors (as if disabled,) but then they resume after a fraction of a second.

Occaisonally, we have had it shut down entirely, but this may be unrelated. Out light wouldn't blink, and the driver station would report that there was no communication. Once, I also noticed the lights for the power regulators were out (on the power distribution board,) so I think this may be an entirely unrelated problem.

I hope this gets solved.

[Ziaholic]

My team has seen something like this also.

I'm wondering if it is a network configuration issue.

When we are developing, and the 'bot is close to the driverstation, we can sometimes switch from tethered to wireless, and in the process, lose communications for a while. Then, without changing anything, it starts working again.


My theory is that it is network related ... but I need to know something about the cRIO's gateway settings before I can conclude anything.

Here's my theory:
- When wireless, the cRIO (with IP addy: 10.xx.yy.2) uses the bridge (with IP addy: 10.xx.yy.1) as its gateway
- When tethered, the cRIO goes straight to the Linksys wireless router via one of the 4 hard-wired ethernet jacks
- If you leave the bridge powered on, even when you are tethered and not using it, then it could be confusing the network connectivity between the devices. (that is, when the bridge is on and unused, it could be confusing the route between the driverstation and the cRIO)


I need to know if the cRIO gateway setting is hard-set to 10.xx.yy.1. If so, then going from tethered to un-tethered could cause some headaches, especially when the bridge is left on.


I plan on doing some more investigation this afternoon at our team meeting. Stay tuned.

[Mark McLeod]

That problem might be due to a persistent cache of the routing table for active IP addresses on the router.

[Gary.C]

I have also been having the troubles of the motors turning off for like a second and robot code and communication go red but then go green after a little bit. Also the communication only works like 50% of the time and sometimes it takes a little more than 2 minutes to load everything! I will stay tuned for updates or solutions.

[Ziaholic]

I'm still not positive about the cRIO gateway setting, or if it is related to the problem, however, some of the guys on my team are theorizing that it could be a startup issue.

Here's a timeline, a few educated guesses, and maybe a fix: (when operating in a wireless mode)
- apply power to the bot
- cRIO comes up first, wireless bridge still booting
- cRIO network comes up, doesn't see an "active" bridge, and times out
- wireless bridge comes up, but too late for cRIO
- no ethernet connection
- reset the cRIO (using a paper clip on the reset button)
- wireless ethernet connects almost immediately

I didn't get much time with the 'bot yesterday, so it was hard to troubleshoot, but when we followed the timeline above, we had no disconnection issues at all.

I'd like to see if the cRIO will re-attempt after 30 or 60 seconds ... stay tuned.

If this theory is correct, and we want to avoid delays at competitions, then we might expect a cRIO firmware patch to slow down its bootup process and wait a little while for the bridge to come up.

Like I said above, this is all just a theory. I hope that it either gets patched, or I'm wrong, because I don't want to have to paper-clip-reset the cRIO every time we turn it on ...


edit: I just posted a question to the NI forums. Perhaps their engineers could shed some light on this for us.
edit2: here's a link to the NI forum question (no activity yet) http://decibel.ni.com/content/thread/5712?tstart=0

[Bryscus]

I think we had a bad game bridge last year or something because we were having similar issues. We also lost coms multiple times and changing the game bridge fixed our issues. We lost a couple matches because we just died in the middle...

So needless to say, we haven't been very impressed with the game bridges so far...

[sircedric4]

We have had similar experiences. The process we now have in place to always ensure that everything talks is to turn off the robot and unplug our router. Turn on the robot and wait about 10 seconds and then plug the router back in. Then wait for the connection to be made and everything works from that point.

We're using the same router that the rookies were given this year the 160. We bought a whole new control system since we wanted to keep last year's awesome robot demo and fundraiser worthy. We are using a factory reconditioned/recertified game bridge because we couldn't find a new one.

We were going nuts trying to figure out why sometimes it would connect to the wireless bridge and sometimes why it wouldn't. That procedure above is what has worked consistently for us. I just hope we don't have similar issues at the regionals.

[Gary.C]

Originally Posted by Ziaholic

I'm still not positive about the cRIO gateway setting, or if it is related to the problem, however, some of the guys on my team are theorizing that it could be a startup issue.

Here's a timeline, a few educated guesses, and maybe a fix: (when operating in a wireless mode)
- apply power to the bot
- cRIO comes up first, wireless bridge still booting
- cRIO network comes up, doesn't see an "active" bridge, and times out
- wireless bridge comes up, but too late for cRIO
- no ethernet connection
- reset the cRIO (using a paper clip on the reset button)
- wireless ethernet connects almost immediately

I didn't get much time with the 'bot yesterday, so it was hard to troubleshoot, but when we followed the timeline above, we had no disconnection issues at all.

I'd like to see if the cRIO will re-attempt after 30 or 60 seconds ... stay tuned.

If this theory is correct, and we want to avoid delays at competitions, then we might expect a cRIO firmware patch to slow down its bootup process and wait a little while for the bridge to come up.

Like I said above, this is all just a theory. I hope that it either gets patched, or I'm wrong, because I don't want to have to paper-clip-reset the cRIO every time we turn it on ...


edit: I just posted a question to the NI forums. Perhaps their engineers could shed some light on this for us.
edit2: here's a link to the NI forum question (no activity yet) http://decibel.ni.com/content/thread/5712?tstart=0

I am pretty sure this is right and I have had to press the reset button on our cRIO too because it won't catch the bridge. I hope there is a fix. Thanks for pointing this out.

[jhersh]

We're looking into this issue. We've been able to reproduce it, but have not yet identified the cause.

-Joe

[Ziaholic]

Thursday we had no issues. It came right up and connected pretty quickly. We'll keep our eyes on it and report any more details when/if we see it happen again.

I'm glad to see that NI was able to reproduce it ... but sporadic errors are difficult to troubleshoot.

[richwong]

We've been having this issue, too. We're using last year's cRIO, Bridge and Router. It's sporadic and hard to reproduce.