"No Comms"

[sabruce01]

Originally Posted by David Brinza

"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point. Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Here are some likely causes of the problem: unplugged or intermittent power/ethernet connections, burying the adapter in the robot, problem with the wireless adapter +/-12V supply from the PD module and mounting the wireless adapter such that the security button gets pushed as the robot is bumped.]

If you momentary lose the wireless adapter, it can take up to 45 sec for the robot to get synch'd with the field - very painful!

We eliminated all of these trough inspection and troubleshooting at the match. ALL electronics were mounted on lexan...3 different ethernet cables were tried with the same results...2 power cables were tried with the same results, PD block was changed, receiver mounted on top of robot on lexan with the same result, security button protected...I literally beat my head on the robot with this one. It is sad to see what the students built fail because of something they did not build.

On a side note, we were also told about the 45 second "re synch" however, during our final elimination, our bot lost coms at the beginning of the match and NEVER regained comms for the rest of the match...about 1:45...so I don't buy that either.

[sabruce01]

Originally Posted by Daniel_LaFleur

We did not, but a lot of teams at GSR did. Here's what we did, that may have contributed to our seeming immunity to the comm issues:

1> Mounted the Wi-Fi on the robot away from all high EM fields (motors) and Faraday cages (steel). Orientation does not seem to be an issue but location is important.
2> Kept good care of our ethernet connectors (both on the DS and on the robot). We inspected them for bad spring pins, made sure we inserted the connector straight, and made sure no debris got into them.
3> Grounded the DS chassis as requested by FIRST.
4> always had our DS fully booted before booting the robot.
5> all electronics are mounted on non-conductive surfaces.

After some observations, I believe that #5 above is the culprit for most teams. Our robot (with it's propellers and wheels) generates a tremendus amount of static when on the regolith. We saw at least 2 machines have a 'No Comm' failure after an impact with our robot. My recomendation to all teams is to ensure that the robot chassis is isolated from all electronics.

We did each of these as well....and 5 out of 8 matches we lost coms. for the other 3 matches, we ran so well we gained enough points to be placed in the elimination rounds. We ran 2 very well, then died our first 2 games in the semi. This was the kiss of death for us because in the final round, we didn't even get our auto! Again, I think this is a bigger problem than FIRST wants to admit.

[Joe Ross]

Originally Posted by squirrel

The older version of the DS software would "see" DS digital inputs in autonomous mode, the newer version does not. We had a set of switches connected to the DS to control autnomous, but now those switches need to be moved to the robot, and the program changed. There was no time to do that before ship.

The new DS update only affects the autonomous enabled mode. You can still read the switches in autonomous disabled (or any other mode). If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS.

[MrForbes]

Thanks, I'll tell Kevin!

[Swttrt224]

In the Midwest Regional, Team 2022, my team, had this happen to us several times. Many times our bot wouldn't connect up to the field and I would have to run out onto the field and power cycle it. By Saturday, I had to do this before every single match that we played.

I also never thought of the static electricity problem. We definitely had it happen where were running fine and then would run into a barrier on the end of the field, and we would all of a sudden get "no comms". The field said that they didn't disable us or anything, and the wireless connections were all still in, so that had to be the problem.

Another issue that we had at one point was the ethernet port on our DS. EVERYBODY CHECK YOUR ETHERNET CONNECTION ON YOUR DS. We found on ours and several others that one of the ethernet connections was up too high, so that when you plug in the ethernet cable, the clip that keeps it in place doesn't click because it is constantly pressed down. We just taped over that ethernet port and used the other one....but this is definitely something that caused us to lose a quarterfinals match in the elimination rounds, so check yours.

That's just my 2 cents on the matter....but no comms issues made us lose many many times in the qualifications matches, so I think this is definitely a problem that needs to be looked into.

-Caitlin
Team 2022 IRS

[rrossbach]

Originally Posted by Joe Ross

The new DS update only affects the autonomous enabled mode. You can still read the switches in autonomous disabled (or any other mode). If you latch the data from before the transition to autonomous disabled, you can keep the switches on the DS.

Just a small correction - I think Joe meant to say "If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS".

I was helping numerous teams with this and some other control system "gotchas" at the NJ regional this past weekend. As long as you read the DS inputs during startup/initialization (while the match is in auto disabled) and save the info somewhere (like a functional global in LabView or a static in C++), then you can use that info during auto enabled. We use DS switches to select which autonomous mode we want to run - much more convenient than having them on the robot.

Thanks,
Ron
Team 2607 - software mentor

[rrossbach]

Originally Posted by David Brinza

"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point.

I'm not sure that's 100% accurate as written. "No Comm" does indicate a failure of communication between the DS and the cRIO, and that could certainly be due link failure between the robot and field wireless access point. But as I understand the network architecture, it could also be due to a number of other causes:
- dropped UDP packets on the field network, for example due to traffic congestion, radio interference, etc
- a software error in the cRIO firmware causing comms to be momentarily interrupted
- a software error in the DS firmware
- etc, etc

Originally Posted by David Brinza

Unless ALL of the robots lose their links in a match, the problem lies within the robot.

Based on my observations, I feel 99.9% certain that that is NOT correct. It looks like all of the DS positions have at least some components of their comm links which are independent - for example:
- each DS position has it's own ethernet tether back to a patch panel
- it also sounds like there are six different individual wireless AP's on the field, each with it's own SSID and WPA config which is setup through FMS at the start of the match
- etc, etc

Saying that a comms failure automatically indicates a problem with the robot is not only incorrect, it also implies that the problem lies entirely within the teams' control to diagnose and resolve. Based on what I observed at the NJ regional this weekend, with multiple unpredictable comm failures on the field, I don't think the problems are solely within the teams' control.

I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem. There are definitely issues with the DS<->FMS<->cRIO comms during matches, and there are a number of different parameters which should be checked, double-checked, re-checked, and stress-tested

If there's anything I can do to help out with this, please feel free to let me know via PM. It was personally disappointing to see how these issues adversely impacted so many teams' experiences during the NJ regional - due to alliance pairings, it impacts not just the team(s) having a problem but also many teams (such as #2607) which did not have any comms problems.

On a brighter note, BRAVO to the folks running the field this weekend - they did an outstanding job trying to find work-arounds and get the matches running as smoothly as possible.

Thanks,
Ron
Team 2607 - software mentor

[Alan Anderson]

Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.

[pitzoid]

Originally Posted by rrossbach

I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem.

We have a running QOS log on every switch in the network that can be accessed through a dedicated VPN network running back to the FMS engineering server, QOS records any error in communication that switches have, I did not find any FMS network errors at the events I was able to check over the weekend including NJ and NH. I have an engineering log running on every robot running during any match which logs every DS message recieved and records it along side the signals we recieve from the PLC IO, I have not found an instance yet where a DS dropped and the PLC IO errored out at the same time (PLC IO communicates every 20ms), still reviewing. The logs from the DSes that dropped out over the weekend during matches indicate that DSes stopped sending FMS status messages, FMS timed out listening for the messages and went into try to reacquire pinging to get the DSes reacquired. What this means is the DSes reset. It seems the DSes are resetting from the EMP that occurs when a robot that has a high voltage charged up from spinning its wheels out in the middle of the field gets in proximity of a field boundary (aluminum structure that is grounded) and does the EMP zap. Robots hitting the end of the field aren't resetting DSes from the physical impact, its the EMP discharge.

Looking into ways to mitigate the susceptability of the DSes to EMP because of the massive static issues that are occuring at some events. You'll notice the issue gets worse towards the end of a tournament because there's usually more static generated then from increased robot action in the elims.

I know its easy to blame the field because this was the first time you all were introduced to it, but I tried to account for these type issues in my designs. My background is semiconductor automation, so I know something about static sensitivities.

New complex system put together with hundreds of other complex systems for the first time, things to figure out, this is engineering...

We need to isolate the issues and then present the solutions. I put this out so you would know what I do at this time. Static is not our friend.

Edit: also, the network is gigabit, we're only utilizing a couple percent of it now, no traffic "congestion" issues.

[MrForbes]

Thanks for the explanation and the status update Bob!

[rrossbach]

Thanks for the update and thorough explanation, Bob - MUCH appreciated!

Sorry if the tone of my post came through as harsh, or as though I was blaming anyone (including the FMS). Was definitely not trying to place any blame - on the contrary was just getting concerned that some folks on this thread (not you) seemed to be directing the issue onto the teams' robots, and didn't want to see any effort get misdirected prematurely. Sounds like my concerns were totally unwarranted.

Hats off to everyone working on these things!

[pitzoid]

Originally Posted by rrossbach

Sorry if the tone of my post came through as harsh

I'm living in the Hatch shadow, I'm used to getting beat on. And the issue we run into these days is all the admin personnel in these events now look to FMS to solve every issue, its a tough challenge

Just like you guys I want nothing more than to have perfect running competitions, I think the teams deserve it. But because of the time frames and resources available sometimes this doesn't seem to work out. Remember last years week one events ran pretty well, that was the second year of that FMS system. FMS is now is now about 3X the complexity of the IFI system and I don't think its that far off from being dialed in for great running events. This is a new control system (both FMS and the robot systems) and obviously we have to get it characterized.

The initial link up issues, the DS drop issues, I saw over the weekend where teams had stops in their cRio code that reset the bot on watchdog timer, we see that when the cRio voltage jumps to 37.37 on the status screen. All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out.

We'll get it figured out, we're determined, there are a bunch of really smart people working on this....

[petek]

Originally Posted by pitzoid

All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out.

We'll get it figured out, we're determined, there are a bunch of really smart people working on this....

Hmmm... That sounds suspiciously like real-world problem-solving and engineering to me! Did anybody ever say the game was the only challenge we're supposed to be solving?

[Al Skierkiewicz]

In Chicago the robots that had the issues listed above (working for auto and 30-60 seconds into the match) had the same problem. Teams had wired the wireless power to a breakered output of the PD rather than the radio power output. When the battery falls to less than 12 volts the radio suddenly cuts out That is why the PD has a special regulated 12 volt output. Inspectors will check for this next week when it is added to the inspection checklist. Pleased follow the control system instructions on how to power the radio.

[MrForbes]

Originally Posted by Alan Anderson

Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.

Alan--

Trying to get a better understanding of this, what distances are you talking about? Should the WGA be at least 6" away from the Jaguar(s)? a foot? more?

thanks