Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   General Forum (http://www.chiefdelphi.com/forums/forumdisplay.php?f=16)
-   -   "No Comms" (http://www.chiefdelphi.com/forums/showthread.php?t=75319)

sabruce01 01-03-2009 11:10

"No Comms"
 
Has anyone experienced this during competition? I know in OKC there were a few that had problems. Our team lost communications 5 of the 8 qualification rounds. Our bot would simply quit in the playing field. I spoke with nearly everyone there and got nowhere. Everything they told us to check had already been replaced in the pit. Auto would be fine, then about 1/2 way through tele, we would loose communication. During our last elimination round we didn't even have an auto which caused us to be eliminated. There is no way we were loosing power to the receiver. We troubleshot that to death and could not get it to fail. But that always seemed to be the default answer I got when asking around.

Any suggestions??? Anybody experience the same?

big1boom 01-03-2009 11:12

Re: "No Comms"
 
Sounds like the problem a few teams were having with the Drivers Station. The Ethernet ports on the DS seem to break easily, so once one breaks, make sure to tape over it. We had Port 2 break.

#1packerfan92 01-03-2009 11:41

Re: "No Comms"
 
this happened to our team at the NJ regional. We beilieve it was the server on the fields fault because it happened to numorus teams on each alliance over the three days

GBIT 01-03-2009 11:47

Re: "No Comms"
 
At GSR they re-ran about 12 matches and were down with comms problems for most of the morning thursday.... Our robot lost about 3 times cause of comms issues....

HERE IS HOW THEY FIXED IT!!!

1. Plug in your DS and wait for it to fully load....
2 THEN turn on your robot and wait for it to get comms

Also make sure that your gateway isn't tucked inside your robot somewhere that caused half of our problems.

Also at GSR they ended up sprinkling water in the driver's box to help cut down on static...

So be careful of placement of your gateway and such and you should be fine..

Oh and one more thing:
With the new DS update given at the competition it actually says "no comm" not "no comms"... Just saying :rolleyes:

MrForbes 01-03-2009 12:10

Re: "No Comms"
 
Quote:

Originally Posted by GBIT (Post 829371)

Also make sure that your gateway isn't tucked inside your robot somewhere that caused half of our problems.

I was wondering if this would be a problem. Also does the orientation of the gateway seem to make any difference? we had some problems during testing before ship, and ended up putting the thing on a top surface of the robot.

Quote:

With the new DS update given at the competition
Is that "new" update the one that has been on the FIRST site since Feb 10th or so?

http://usfirst.org/community/frc/content.aspx?id=10934

we found it the day before crating up the robot, it kind of messed up our plans for autnomous control.

Dmentor 01-03-2009 12:22

Re: "No Comms"
 
Quote:

Originally Posted by GBIT (Post 829371)
1. Plug in your DS and wait for it to fully load....
2 THEN turn on your robot and wait for it to get comms

In pit testing via tether on Thursday morning, we would often see the described problem if we started the cRIO before the DS. We got in the habit of always starting up the DS first when testing in the pits and didn't have any more problems until we started practicing on the field... In several of our matches the robot worked without problems, in the others however we would encounter odd failures associated with the digital IO controls that are relayed through the DS. We would bring the robot back to the pit and go through our checklist only to discover no problems. After a while of head-scratching, we connected the data points by guessing that there might be a startup order dependency that the driver team wasn't aware of. Once we determined this we were able to recreate the odd behavior (DS digitial IO failures) in the pit via tether. Our driver team implemented the startup procedure above using hand signals to ensure that we would always be able to run reliably. We were pretty scared until we discovered this though since debugging unpredictable behavior is tricky.

On a related note (perhaps obvious to most folks) but since the practice sessions have two portions, we needed to reset the cRIO between the portions in order to ensure proper software initialization of states. Otherwise we ended up with the system carrying over states from the first portion into the second.

FYI. We are using the Labview basic framework so I'm not sure how much different the Labview advanced framework or C++ implementations would be.

GGCO 01-03-2009 12:25

Re: "No Comms"
 
Might be that annoying VxWorks bug showing it's ugly head again...

That happened to a few teams at Traverse City. Make sure you have all of the latest updates for your hardware and software.

billbo911 01-03-2009 12:26

Re: "No Comms"
 
Quote:

Originally Posted by squirrel (Post 829388)
Is that "new" update the one that has been on the FIRST site since Feb 10th or so?

http://usfirst.org/community/frc/content.aspx?id=10934

we found it the day before crating up the robot, it kind of messed up our plans for autnomous control.

Jim,
Can you please expand on, or explain, this comment. I'm not sure I understand how updating the DS impacts Autonomous. Inquiring minds want to know.

Daniel_LaFleur 01-03-2009 12:43

Re: "No Comms"
 
Quote:

Originally Posted by sabruce01 (Post 829344)
Has anyone experienced this during competition? I know in OKC there were a few that had problems. Our team lost communications 5 of the 8 qualification rounds. Our bot would simply quit in the playing field. I spoke with nearly everyone there and got nowhere. Everything they told us to check had already been replaced in the pit. Auto would be fine, then about 1/2 way through tele, we would loose communication. During our last elimination round we didn't even have an auto which caused us to be eliminated. There is no way we were loosing power to the receiver. We troubleshot that to death and could not get it to fail. But that always seemed to be the default answer I got when asking around.

Any suggestions??? Anybody experience the same?

We did not, but a lot of teams at GSR did. Here's what we did, that may have contributed to our seeming immunity to the comm issues:

1> Mounted the Wi-Fi on the robot away from all high EM fields (motors) and Faraday cages (steel). Orientation does not seem to be an issue but location is important.
2> Kept good care of our ethernet connectors (both on the DS and on the robot). We inspected them for bad spring pins, made sure we inserted the connector straight, and made sure no debris got into them.
3> Grounded the DS chassis as requested by FIRST.
4> always had our DS fully booted before booting the robot.
5> all electronics are mounted on non-conductive surfaces.

After some observations, I believe that #5 above is the culprit for most teams. Our robot (with it's propellers and wheels) generates a tremendus amount of static when on the regolith. We saw at least 2 machines have a 'No Comm' failure after an impact with our robot. My recomendation to all teams is to ensure that the robot chassis is isolated from all electronics.

BradMello 01-03-2009 12:45

Re: "No Comms"
 
Quote:

Originally Posted by GBIT (Post 829371)
At GSR they re-ran about 12 matches and were down with comms problems for most of the morning thursday.... Our robot lost about 3 times cause of comms issues....

HERE IS HOW THEY FIXED IT!!!

1. Plug in your DS and wait for it to fully load....
2 THEN turn on your robot and wait for it to get comms

Also make sure that your gateway isn't tucked inside your robot somewhere that caused half of our problems.

Also at GSR they ended up sprinkling water in the driver's box to help cut down on static...

Yeah, at GSR a good amount of qualifying matches ended up being replayed becuase of this problem. They came to the conclusion that it was static electricity and put conductive tape on the metal grate which the moon rocks rolled into in the corners of the field, and it seemed to work. They also sprayed water on the carpet in the corners of the field. After they did that I can't recall one match where anyone lost comms.

Nothing like being behind the glass and seeing Dean Kamen on the field spraying water on the carpet between matches :D

David Brinza 01-03-2009 12:47

Re: "No Comms"
 
"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point. Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Here are some likely causes of the problem: unplugged or intermittent power/ethernet connections, burying the adapter in the robot, problem with the wireless adapter +12V supply from the PD module and mounting the wireless adapter such that the security button gets pushed as the robot is bumped.]

If you momentary lose the wireless adapter, it can take up to 45 sec for the robot to get synch'd with the field - very painful!

pitzoid 01-03-2009 13:35

Re: "No Comms"
 
The ultimate conclusion is still being investigated, but I'll tell you what I know:

Robots spin wheels on FRP, plastic on plastic with a metal frame to attract the charge. robots build charge, like helicopter, get near field border, being its metal, grounded, you get EMP. DS seem susceptable to EMP.

So for all you people who saw a robot hit a field end and DSes reset and you thought it was because the physical impact knocked a connection, doesn't seem to be the case, it was the EMP discharge. This would happen anytime a robot had been running around in the middle of the field for a few minutes and then got near a field border.

All the FMS equipment is sheilded for this, I check my engineering logs on fields like BAE, no issues I could find. I run QOS on all the switches, sat and watched the field end switches for a while at BAE while matches were running, no issues I could see.

A suggestion to help. If you have your DS mounted in a control case. run a ground wire from the DS mode FIRST already mandated to a grounding connection that contacts the drivers table when you set your drivers station on the table, this will help to establish a faraday cage around your DS internal circuitry. If your DS sits directly on the table, it should ground.

We're still looking into whether the EMP being caught by the DS to SCCE ethernet cord and traveling the cord to the DS is resetting it. It looks like it doesn't affect the SCCE switches as they are industrial switches sheilded for this and I can't find issues in their logs.

The link up issues seems to be this (still being investigated):

When the WGA bridge starts up in security mode, if it doesn't see its SSID within a certain amount of time, it times out. So the general rule with this should be, DO NOT turn on your robot till you see your team number on a team station. Reason - this means the scorekeeper already activated the network configuration for your match, i.e. FMS is or has written your SSID to the Access Point, and by the time your WGA bridge boots up, your SSID will be present and it will be able to connect without timing out. This is why it seems once things got rolling, things went more smooth, but when trying to restart in the morning or after lunch where things would get out of "sync" there were issues.

These are all my theories currently
YMMV

Will keep investigating problems.

dragonrulr288 01-03-2009 13:36

Re: "No Comms"
 
AT Traverse City, the RObodawgs and Robodawgs O.T.L. Expierienced lots of this!

GBIT 01-03-2009 13:47

Re: "No Comms"
 
Quote:

Originally Posted by David Brinza (Post 829410)
"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point. Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Here are some likely causes of the problem: unplugged or intermittent power/ethernet connections, burying the adapter in the robot, problem with the wireless adapter +/-12V supply from the PD module and mounting the wireless adapter such that the security button gets pushed as the robot is bumped.]

If you momentary lose the wireless adapter, it can take up to 45 sec for the robot to get synch'd with the field - very painful!


There were instances at GSR i know cause it was our robot where ours and only ours died and they ended up replaying the match due to a field error not our robot....

MrForbes 01-03-2009 13:54

Re: "No Comms"
 
Quote:

Originally Posted by billbo911 (Post 829397)
Jim,
Can you please expand on, or explain, this comment. I'm not sure I understand how updating the DS impacts Autonomous. Inquiring minds want to know.

The older version of the DS software would "see" DS digital inputs in autonomous mode, the newer version does not. We had a set of switches connected to the DS to control autnomous, but now those switches need to be moved to the robot, and the program changed. There was no time to do that before ship.

We also did not withhold our control system to work on programming...based on our experience of making some simple changes in one place, and several other things not working as a result, all during build season. We put a mostly working robot in the crate, hopefully we'll make all our practice matches, and fix the programming problems one at a time as time allows at the competitions.

sabruce01 01-03-2009 13:55

Re: "No Comms"
 
Quote:

Originally Posted by David Brinza (Post 829410)
"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point. Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Here are some likely causes of the problem: unplugged or intermittent power/ethernet connections, burying the adapter in the robot, problem with the wireless adapter +/-12V supply from the PD module and mounting the wireless adapter such that the security button gets pushed as the robot is bumped.]

If you momentary lose the wireless adapter, it can take up to 45 sec for the robot to get synch'd with the field - very painful!

We eliminated all of these trough inspection and troubleshooting at the match. ALL electronics were mounted on lexan...3 different ethernet cables were tried with the same results...2 power cables were tried with the same results, PD block was changed, receiver mounted on top of robot on lexan with the same result, security button protected...I literally beat my head on the robot with this one. It is sad to see what the students built fail because of something they did not build.

On a side note, we were also told about the 45 second "re synch" however, during our final elimination, our bot lost coms at the beginning of the match and NEVER regained comms for the rest of the match...about 1:45...so I don't buy that either.

sabruce01 01-03-2009 14:00

Re: "No Comms"
 
Quote:

Originally Posted by Daniel_LaFleur (Post 829404)
We did not, but a lot of teams at GSR did. Here's what we did, that may have contributed to our seeming immunity to the comm issues:

1> Mounted the Wi-Fi on the robot away from all high EM fields (motors) and Faraday cages (steel). Orientation does not seem to be an issue but location is important.
2> Kept good care of our ethernet connectors (both on the DS and on the robot). We inspected them for bad spring pins, made sure we inserted the connector straight, and made sure no debris got into them.
3> Grounded the DS chassis as requested by FIRST.
4> always had our DS fully booted before booting the robot.
5> all electronics are mounted on non-conductive surfaces.

After some observations, I believe that #5 above is the culprit for most teams. Our robot (with it's propellers and wheels) generates a tremendus amount of static when on the regolith. We saw at least 2 machines have a 'No Comm' failure after an impact with our robot. My recomendation to all teams is to ensure that the robot chassis is isolated from all electronics.

We did each of these as well....and 5 out of 8 matches we lost coms. for the other 3 matches, we ran so well we gained enough points to be placed in the elimination rounds. We ran 2 very well, then died our first 2 games in the semi. This was the kiss of death for us because in the final round, we didn't even get our auto! Again, I think this is a bigger problem than FIRST wants to admit.

Joe Ross 01-03-2009 14:07

Re: "No Comms"
 
Quote:

Originally Posted by squirrel (Post 829480)
The older version of the DS software would "see" DS digital inputs in autonomous mode, the newer version does not. We had a set of switches connected to the DS to control autnomous, but now those switches need to be moved to the robot, and the program changed. There was no time to do that before ship.

The new DS update only affects the autonomous enabled mode. You can still read the switches in autonomous disabled (or any other mode). If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS.

MrForbes 01-03-2009 14:16

Re: "No Comms"
 
Thanks, I'll tell Kevin!

Swttrt224 01-03-2009 14:28

Re: "No Comms"
 
In the Midwest Regional, Team 2022, my team, had this happen to us several times. Many times our bot wouldn't connect up to the field and I would have to run out onto the field and power cycle it. By Saturday, I had to do this before every single match that we played.

I also never thought of the static electricity problem. We definitely had it happen where were running fine and then would run into a barrier on the end of the field, and we would all of a sudden get "no comms". The field said that they didn't disable us or anything, and the wireless connections were all still in, so that had to be the problem.

Another issue that we had at one point was the ethernet port on our DS. EVERYBODY CHECK YOUR ETHERNET CONNECTION ON YOUR DS. We found on ours and several others that one of the ethernet connections was up too high, so that when you plug in the ethernet cable, the clip that keeps it in place doesn't click because it is constantly pressed down. We just taped over that ethernet port and used the other one....but this is definitely something that caused us to lose a quarterfinals match in the elimination rounds, so check yours.

That's just my 2 cents on the matter....but no comms issues made us lose many many times in the qualifications matches, so I think this is definitely a problem that needs to be looked into.

-Caitlin
Team 2022 IRS

rrossbach 01-03-2009 14:30

Re: "No Comms"
 
Quote:

Originally Posted by Joe Ross (Post 829491)
The new DS update only affects the autonomous enabled mode. You can still read the switches in autonomous disabled (or any other mode). If you latch the data from before the transition to autonomous disabled, you can keep the switches on the DS.

Just a small correction - I think Joe meant to say "If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS".

I was helping numerous teams with this and some other control system "gotchas" at the NJ regional this past weekend. As long as you read the DS inputs during startup/initialization (while the match is in auto disabled) and save the info somewhere (like a functional global in LabView or a static in C++), then you can use that info during auto enabled. We use DS switches to select which autonomous mode we want to run - much more convenient than having them on the robot.

Thanks,
Ron
Team 2607 - software mentor

rrossbach 01-03-2009 15:07

Re: "No Comms"
 
Quote:

Originally Posted by David Brinza (Post 829410)
"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point.

I'm not sure that's 100% accurate as written. "No Comm" does indicate a failure of communication between the DS and the cRIO, and that could certainly be due link failure between the robot and field wireless access point. But as I understand the network architecture, it could also be due to a number of other causes:
- dropped UDP packets on the field network, for example due to traffic congestion, radio interference, etc
- a software error in the cRIO firmware causing comms to be momentarily interrupted
- a software error in the DS firmware
- etc, etc

Quote:

Originally Posted by David Brinza (Post 829410)
Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Based on my observations, I feel 99.9% certain that that is NOT correct. It looks like all of the DS positions have at least some components of their comm links which are independent - for example:
- each DS position has it's own ethernet tether back to a patch panel
- it also sounds like there are six different individual wireless AP's on the field, each with it's own SSID and WPA config which is setup through FMS at the start of the match
- etc, etc

Saying that a comms failure automatically indicates a problem with the robot is not only incorrect, it also implies that the problem lies entirely within the teams' control to diagnose and resolve. Based on what I observed at the NJ regional this weekend, with multiple unpredictable comm failures on the field, I don't think the problems are solely within the teams' control.

I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem. There are definitely issues with the DS<->FMS<->cRIO comms during matches, and there are a number of different parameters which should be checked, double-checked, re-checked, and stress-tested :)

If there's anything I can do to help out with this, please feel free to let me know via PM. It was personally disappointing to see how these issues adversely impacted so many teams' experiences during the NJ regional - due to alliance pairings, it impacts not just the team(s) having a problem but also many teams (such as #2607) which did not have any comms problems.

On a brighter note, BRAVO to the folks running the field this weekend - they did an outstanding job trying to find work-arounds and get the matches running as smoothly as possible.

Thanks,
Ron
Team 2607 - software mentor

Alan Anderson 01-03-2009 17:32

Re: "No Comms"
 
Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.

pitzoid 01-03-2009 17:58

Re: "No Comms"
 
Quote:

Originally Posted by rrossbach (Post 829530)
I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem.

We have a running QOS log on every switch in the network that can be accessed through a dedicated VPN network running back to the FMS engineering server, QOS records any error in communication that switches have, I did not find any FMS network errors at the events I was able to check over the weekend including NJ and NH. I have an engineering log running on every robot running during any match which logs every DS message recieved and records it along side the signals we recieve from the PLC IO, I have not found an instance yet where a DS dropped and the PLC IO errored out at the same time (PLC IO communicates every 20ms), still reviewing. The logs from the DSes that dropped out over the weekend during matches indicate that DSes stopped sending FMS status messages, FMS timed out listening for the messages and went into try to reacquire pinging to get the DSes reacquired. What this means is the DSes reset. It seems the DSes are resetting from the EMP that occurs when a robot that has a high voltage charged up from spinning its wheels out in the middle of the field gets in proximity of a field boundary (aluminum structure that is grounded) and does the EMP zap. Robots hitting the end of the field aren't resetting DSes from the physical impact, its the EMP discharge.

Looking into ways to mitigate the susceptability of the DSes to EMP because of the massive static issues that are occuring at some events. You'll notice the issue gets worse towards the end of a tournament because there's usually more static generated then from increased robot action in the elims.

I know its easy to blame the field because this was the first time you all were introduced to it, but I tried to account for these type issues in my designs. My background is semiconductor automation, so I know something about static sensitivities.

New complex system put together with hundreds of other complex systems for the first time, things to figure out, this is engineering...

We need to isolate the issues and then present the solutions. I put this out so you would know what I do at this time. Static is not our friend.

Edit: also, the network is gigabit, we're only utilizing a couple percent of it now, no traffic "congestion" issues.

MrForbes 01-03-2009 18:40

Re: "No Comms"
 
Thanks for the explanation and the status update Bob!

rrossbach 01-03-2009 19:11

Re: "No Comms"
 
Thanks for the update and thorough explanation, Bob - MUCH appreciated!

Sorry if the tone of my post came through as harsh, or as though I was blaming anyone (including the FMS). Was definitely not trying to place any blame - on the contrary was just getting concerned that some folks on this thread (not you) seemed to be directing the issue onto the teams' robots, and didn't want to see any effort get misdirected prematurely. Sounds like my concerns were totally unwarranted.

Hats off to everyone working on these things!

pitzoid 01-03-2009 19:34

Re: "No Comms"
 
Quote:

Originally Posted by rrossbach (Post 829746)
Sorry if the tone of my post came through as harsh

I'm living in the Hatch shadow, I'm used to getting beat on. And the issue we run into these days is all the admin personnel in these events now look to FMS to solve every issue, its a tough challenge :D

Just like you guys I want nothing more than to have perfect running competitions, I think the teams deserve it. But because of the time frames and resources available sometimes this doesn't seem to work out. Remember last years week one events ran pretty well, that was the second year of that FMS system. FMS is now is now about 3X the complexity of the IFI system and I don't think its that far off from being dialed in for great running events. This is a new control system (both FMS and the robot systems) and obviously we have to get it characterized.

The initial link up issues, the DS drop issues, I saw over the weekend where teams had stops in their cRio code that reset the bot on watchdog timer, we see that when the cRio voltage jumps to 37.37 on the status screen. All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out.

We'll get it figured out, we're determined, there are a bunch of really smart people working on this....

petek 01-03-2009 19:47

Re: "No Comms"
 
Quote:

Originally Posted by pitzoid (Post 829772)
All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out.

We'll get it figured out, we're determined, there are a bunch of really smart people working on this....

Hmmm... That sounds suspiciously like real-world problem-solving and engineering to me! Did anybody ever say the game was the only challenge we're supposed to be solving?

Al Skierkiewicz 01-03-2009 21:39

Re: "No Comms"
 
In Chicago the robots that had the issues listed above (working for auto and 30-60 seconds into the match) had the same problem. Teams had wired the wireless power to a breakered output of the PD rather than the radio power output. When the battery falls to less than 12 volts the radio suddenly cuts out That is why the PD has a special regulated 12 volt output. Inspectors will check for this next week when it is added to the inspection checklist. Pleased follow the control system instructions on how to power the radio.

MrForbes 02-03-2009 11:05

Re: "No Comms"
 
Quote:

Originally Posted by Alan Anderson (Post 829655)
Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.

Alan--

Trying to get a better understanding of this, what distances are you talking about? Should the WGA be at least 6" away from the Jaguar(s)? a foot? more?

thanks

pitzoid 02-03-2009 12:08

Re: "No Comms"
 
Quote:

Originally Posted by squirrel (Post 830239)
Should the WGA be at least 6" away from the Jaguar(s)?

At least 6" and as free of metal structure as possible.

:D

MrForbes 02-03-2009 12:10

Re: "No Comms"
 
Thanks, that's what I wanted to hear! We have the WGA on top of the wood electronics box now, not sure if it's 6" away from the nearest Jaguar, but we can make that happen if we still have problems.

GBIT 02-03-2009 12:13

Re: "No Comms"
 
Quote:

Originally Posted by squirrel (Post 830239)
Alan--

Trying to get a better understanding of this, what distances are you talking about? Should the WGA be at least 6" away from the Jaguar(s)? a foot? more?

thanks

Quote:

Originally Posted by pitzoid (Post 830299)
At least 6" and as free of metal structure as possible.

:D

once we moved ours away from our control system and out of the frame of our robot we had no problems with lost comms again....

Alan Anderson 02-03-2009 12:23

Re: "No Comms"
 
Quote:

Originally Posted by Alan Anderson (Post 829655)
Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar...

Quote:

Originally Posted by squirrel (Post 830239)
Trying to get a better understanding of this, what distances are you talking about?

A half inch is definitely too close. :P The team I assisted directly had put a Jaguar on the top side of a piece of lexan, and the WGA was mounted to the bottom side directly underneath it.

I only heard about the other team second-hand, so I don't actually know how near the parts were to each other, but I got the impression it was less than a couple of inches. I'll try to find out details from the Spare Parts volunteer who helped them correct the problem when they requested a replacement WGA.

MrForbes 02-03-2009 12:52

Re: "No Comms"
 
1 Attachment(s)
Ok, thanks! This is how we had our WGA when we had some occasional no comm problems, it's now on top of the box, sitting flat.

Al Skierkiewicz 02-03-2009 14:15

Re: "No Comms"
 
Let us not forget that any time the gaming adapter is trying to connect or is rebooting or has lost a security packet, you will see "No comm".

pitzoid 02-03-2009 18:09

Re: "No Comms"
 
Still investigating the "no comms" dropout during match issue, but it does appear its the network switch chip reset in the DS being susceptable to EMP. I can repeat the drop of multiple DSes connected to different SCCEs while a match is running with an EMP event. Logs show the reset takes ~31 seconds for the DS to reconnect with FMS after such an event. No FMS issues that I can find when the event happens. Why the reset trips is still undetermined and I don't know if we can come up with a quick fix for it. Will relay more as I know.

rrossbach 02-03-2009 18:45

Re: "No Comms"
 
Thanks for the update!

484-ben 03-03-2009 20:42

Re: "No Comms"
 
During our competition on Saturday at NJ our robot failed completely to operate during autonomous. During teleop motors connected to spikes worked well, motors connected to Jaguar speed controllers did not operate. Everything worked perfectly on Friday. Any suggestions?

484-ben 03-03-2009 20:51

spikes work/jaguar not, suggestions?
 
During our competition on Saturday at NJ our robot failed completely to operate during autonomous. During teleop motors connected to spikes worked well, motors connected to Jaguar speed controllers did not operate. Everything worked perfectly on Friday. Any suggestions?

pitzoid 04-03-2009 00:50

Re: "No Comms"
 
Update:

Sent a bunch of test data to FIRST and DEKA yesterday confirming the DS Network chip susceptability to EMP. Today they were able to repeat my results, so it seems confirmed. We can't fix the DS easily, but we're working on a bunch of "workarounds". FIRST is looking into ways to eliminate static generation, which if they can will fix the source of the issue and I've been testing some FMS network mods today that enable the DS to do a minimal association with the the FMS network. I.e. now when the DS network chip resets on an EMP event, I can have it talking with both FMS and the robot in under 2 seconds. If the DS completely reboots its a few more seconds but it still comes back fast. Not the ideal situation, but it should help. Hopefully we won't have any static issues this weekend.

Oh, and the added extra benefit of the network mods? Looks like we dropped about 20 seconds off the initial DS/robot/fms linkup before a match starts ;)

Real engineering: see a problem, analyze the data, if you can't "fix it", find the work around :ahh:

NickE 04-03-2009 01:11

Re: "No Comms"
 
Quote:

Originally Posted by Daniel_LaFleur (Post 829404)
Orientation does not seem to be an issue but location is important

Actually, we found that orientation can be a huge issue. When we had teams in for our pre-ship practice day and scrimmage, one team had mounted their WGA sideways. Midway through a practice match, they lost comms and did not regain comms for the remainder of the match. Upon later troubleshooting, it was found that rotating the WGA 90° while keeping it in the same position allowed the WGA to connect to the field network again.

It seems that in very close proximity to the router, the orientation does not matter much. However, as the robots get farther away (it was still less than 30 feet, unobstructed, from the router), the orientation matters. We found that it is best to place the WGA on a flat plane (bottom of WGA parallel to ground)

petek 04-03-2009 09:30

Re: "No Comms"
 
Quote:

Originally Posted by NickE (Post 831523)
Actually, we found that orientation can be a huge issue. When we had teams in for our pre-ship practice day and scrimmage, one team had mounted their WGA sideways. Midway through a practice match, they lost comms and did not regain comms for the remainder of the match. Upon later troubleshooting, it was found that rotating the WGA 90° while keeping it in the same position allowed the WGA to connect to the field network again.

It seems that in very close proximity to the router, the orientation does not matter much. However, as the robots get farther away (it was still less than 30 feet, unobstructed, from the router), the orientation matters. We found that it is best to place the WGA on a flat plane (bottom of WGA parallel to ground)

I see that the Linksys documentation shows that you can see its signal strength in the setup Status page. This would allow you to see what orientation gives the best performance. I did a quick search online and saw several posts on DirectTV and gaming forums that suggested that orientation affects bit rate, but nothing that said if there was an optimum configuration.

pitzoid 04-03-2009 12:39

Re: "No Comms"
 
For people interested in finding out more about ESD sensitivities and ways to mitigate. Do a google search on "triboelectric series"

Team 135 04-03-2009 15:43

Re: "No Comms"
 
We had our Linksys wireless bridge go bad. We took it to Pit Admin to have it programed and they said that the setting were corrupted. They would correct them, and unplug the box, plug it back in and it would be corrupted again. We borrowed one from replacement parts and out bot worked fine during the rest of the competition. Luckily the problems were only during practice matches.

rrossbach 04-03-2009 22:06

Re: "No Comms"
 
Quote:

Originally Posted by pitzoid (Post 831518)
Update:

Sent a bunch of test data to FIRST and DEKA yesterday confirming the DS Network chip susceptability to EMP. Today they were able to repeat my results, so it seems confirmed. We can't fix the DS easily, but we're working on a bunch of "workarounds". FIRST is looking into ways to eliminate static generation, which if they can will fix the source of the issue and I've been testing some FMS network mods today that enable the DS to do a minimal association with the the FMS network. I.e. now when the DS network chip resets on an EMP event, I can have it talking with both FMS and the robot in under 2 seconds. If the DS completely reboots its a few more seconds but it still comes back fast. Not the ideal situation, but it should help. Hopefully we won't have any static issues this weekend.

Oh, and the added extra benefit of the network mods? Looks like we dropped about 20 seconds off the initial DS/robot/fms linkup before a match starts ;)

Real engineering: see a problem, analyze the data, if you can't "fix it", find the work around :ahh:

Great news! Hopefully even if there are any issues in future competitions it won't impact teams & their alliance partners as much as in the past. Thanks!!!

Ron

DRH2o 05-03-2009 12:38

Re: "No Comms"
 
Would it be helpful to put a small piece of wire rope dragging the floor to discharge the static. They have done this on carts at stores to solve static problems there.

pitzoid 05-03-2009 15:28

Re: "No Comms"
 
Quote:

Originally Posted by DRH2o (Post 832137)
Would it be helpful to put a small piece of wire rope dragging the floor to discharge the static. They have done this on carts at stores to solve static problems there.

Might help teams to have a ground strap on the bots, they were seeing static issues at Boston today and wetted the carpet which seems to help. Boston in on an Ice Rink, I understand NH was too, so it seems Ice Rinks and Lunacy are challendges. Probably due to the ice pulling all the humidity close to ground level out. Humidifiers would help, just need to keep the humidity at floor level up.

ezygmont708 05-03-2009 19:27

Re: "No Comms"
 
Quote:

Originally Posted by squirrel (Post 829388)
I was wondering if this would be a problem. Also does the orientation of the gateway seem to make any difference? we had some problems during testing before ship, and ended up putting the thing on a top surface of the robot.



Is that "new" update the one that has been on the FIRST site since Feb 10th or so?

http://usfirst.org/community/frc/content.aspx?id=10934

we found it the day before crating up the robot, it kind of messed up our plans for autnomous control.

A few things about the wireless communication system. There are a few things that will limit your tx/rx between the driver station and the wireless gaming adapter...

1) Gaming Adapter should, at no time, be blocked by a piece of metal on any side. I have seen teams that have mounted the adapter on a sheet of metal more than 1/8" thick.

2) Try to keep the adapter away from motors. Motors generate a magnetic field, which at times can interfere with 802.11x communications.

3) Ensure that you are using the correct cable type, and that each cable is properly seated and secured. Ethernet cables are definately more durable than a PWM, but they are suseptible to Electro-Magnetic interference. I have actually ordered some shielded Twisted pair for our next regional.

Being a network technician, I did a simple site survey at the trenton regional, while the fields were not in use. I found a plethora of 2.4Ghz communication. Much of it was the field communication system, but at trenton there was a fairly strong wireless signal active (Trenton Devil's Press Box). Another fairly important fact.... While bluetooth signals are not very strong they can cause interference with b/g/n networks.

Just my 2 cents...

Dave Flowerday 05-03-2009 20:13

Re: "No Comms"
 
Quote:

Originally Posted by ezygmont708 (Post 832285)
I have actually ordered some shielded Twisted pair for our next regional.

Using STP is against the rules. See R57:
Quote:

Originally Posted by R57
The signal output from the wireless bridge must be connected directly to the cRIO Mobile Device Controller using one of the Ethernet cables provided in the 2009 Kit Of Parts.

Quote:

Originally Posted by ezygmont708
I found a plethora of 2.4Ghz communication. Much of it was the field communication system, but at trenton there was a fairly strong wireless signal active (Trenton Devil's Press Box). Another fairly important fact.... While bluetooth signals are not very strong they can cause interference with b/g/n networks.

The control system this year uses 5GHz 802.11n exclusively, so you won't find any field communications in the 2.4 band. Additionally, Bluetooth interference should be a non-issue since it also uses the 2.4GHz band.

pitzoid 05-03-2009 21:28

Re: "No Comms"
 
Quote:

Originally Posted by Dave Flowerday (Post 832313)
The control system this year uses 5GHz 802.11n exclusively, so you won't find any field communications in the 2.4 band.

Actually, the field does use 2.4Ghz, just not for the robots.

ezygmont708 06-03-2009 01:26

Re: "No Comms"
 
Quote:

Originally Posted by Dave Flowerday (Post 832313)
Using STP is against the rules. See R57:



The control system this year uses 5GHz 802.11n exclusively, so you won't find any field communications in the 2.4 band. Additionally, Bluetooth interference should be a non-issue since it also uses the 2.4GHz band.

I was not aware of the rule regarding shielded ethernet cable. Thank You for the notice in advance.

As for the wireless communication, I know that I was seeing massive comms across the 2.4 Ghz network, and I guess it was possible that it was the arena's wireless, but coverage across all 11 channels in the 2.4 Ghz range is abnormal, as one would normally try to keep it seperated out to three non-overlapping channels (Normally 1/6/11). As with rule R57 bluetooth is not permitted on the robot, that is very smart on FIRST's Part.

Last year, while reviewing issues on a wireless system, we noticed an issue with bluetooth clients that were crowding the network, and therefore providing interference across the spectrum. Microwaves, and HVAC motors have been shown to pose similar problems.

I still highly recommend moving the adapter way from any large metal plates, and keeping them away from motors (within reason (5-8"))...

pitzoid 06-03-2009 12:41

Re: "No Comms"
 
The FRC arena has an intelligent Access Point (Cisco 1252), when it starts up it scans the entire spectrum and then tries to establish itself on the least interferring channel. You can force it to a channel, but right now on the road they are all in defualt, which means they'll do this. When we get to Atlanta, I'll have to force them to a certain channel for coordination with 5 arenas.

Mentor007 08-03-2009 22:38

Re: "No Comms"
 
We experienced no comms during competition and tracked it down to one Fisher Price motor being driven by a Jaguar. The WGA was mounted high and well away from the motor however the Jaguar was only a few inches off the CRio near the ethernet port. We kept getting comms drop outs even when tethered. When we disabled the motor, problem went away. We then also ran the motor but at lower PWM levels and the problem went away. At high PWM values the dropouts came back. We swapped the Jaguar for another without any impact. We swapped the Jaguar for a Victor without any impact. We are implementing a rerouting of all wiring to keep all power lines away from the CRio. Putting EMI suppression cap on the FP motor to the extent legal. The problem did not crop up until elimination rounds, so perhaps brush wear or contamination had an impact. The above troubleshooting strategy seems to have pretty definitively pointed the FP motor's brush noise out as the source. Better electrical board layout may have helped to prevent the failure. Hope this can help someone else. Thanks to Eric of 1511 for his help.

Al Skierkiewicz 09-03-2009 11:32

Re: "No Comms"
 
Please check the Q&A http://forums.usfirst.org/showthread.php?t=12228
Dragging wires and other attempts at fixes to esd issues are not to be used while First researches the problems. Until further notice, please standby.

Darkcrosbone 09-03-2009 12:30

Re: "No Comms"
 
at our regionals there was a static electricity problem that would make the robots stop talking to the DS except when you had pneumatics

pitzoid 09-03-2009 13:50

Re: "No Comms"
 
Quote:

Originally Posted by Mentor007 (Post 833386)
Putting EMI suppression cap on the FP motor to the extent legal.

I've talked to FIRST for years about this. We went through a painful phase in robot combat where there would be all sorts of radio drop out issues due to noisy motors and it could be fixed with supression caps across the motor leads. I actually think this should be mandated for the KOP motors as most of them are extremely "inexpensive" motors that will create alot of brush noise EMP. Some of the better designed robots I've seen the last few years as the head mechanical Judge at the PHX regional were the ones that would seperate the control electronics (in this case it'd be the cRio, WGA and breakout boards) from the motor controllers and motors by some sort of sheilding screen or bulkhead. But this is a consideration with initial design.

I'm in the camp right now that bots should have a drag chain, I don't know what the hold up is on that, ultimately its FIRST's decision. I know they are contemplating putting drag chains on the trailers for this weekend, I'd prefer that they give teams the option to put them on their bots as we've proved that the KOP wheel from this year is very efficient at isolating static voltage buildup.

On another note: FMS ran well this past weekend, we had a couple equipment issues with Estop that were manufactured incorrectly. But most other issues seen were related to robots and ESD.

Hopefully, tomorrow we'll have some resolution in the team update.


All times are GMT -5. The time now is 14:43.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi