Re: "No Comms"
 

We eliminated all of these trough inspection and troubleshooting at the match. ALL electronics were mounted on lexan...3 different ethernet cables were tried with the same results...2 power cables were tried with the same results, PD block was changed, receiver mounted on top of robot on lexan with the same result, security button protected...I literally beat my head on the robot with this one. It is sad to see what the students built fail because of something they did not build.

On a side note, we were also told about the 45 second "re synch" however, during our final elimination, our bot lost coms at the beginning of the match and NEVER regained comms for the rest of the match...about 1:45...so I don't buy that either.

Re: "No Comms"
 

We did each of these as well....and 5 out of 8 matches we lost coms. for the other 3 matches, we ran so well we gained enough points to be placed in the elimination rounds. We ran 2 very well, then died our first 2 games in the semi. This was the kiss of death for us because in the final round, we didn't even get our auto! Again, I think this is a bigger problem than FIRST wants to admit.

Re: "No Comms"
 

The new DS update only affects the autonomous enabled mode. You can still read the switches in autonomous disabled (or any other mode). If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS.

Re: "No Comms"
 

Thanks, I'll tell Kevin!

Re: "No Comms"
 

In the Midwest Regional, Team 2022, my team, had this happen to us several times. Many times our bot wouldn't connect up to the field and I would have to run out onto the field and power cycle it. By Saturday, I had to do this before every single match that we played.

I also never thought of the static electricity problem. We definitely had it happen where were running fine and then would run into a barrier on the end of the field, and we would all of a sudden get "no comms". The field said that they didn't disable us or anything, and the wireless connections were all still in, so that had to be the problem.

Another issue that we had at one point was the ethernet port on our DS. EVERYBODY CHECK YOUR ETHERNET CONNECTION ON YOUR DS. We found on ours and several others that one of the ethernet connections was up too high, so that when you plug in the ethernet cable, the clip that keeps it in place doesn't click because it is constantly pressed down. We just taped over that ethernet port and used the other one....but this is definitely something that caused us to lose a quarterfinals match in the elimination rounds, so check yours.

That's just my 2 cents on the matter....but no comms issues made us lose many many times in the qualifications matches, so I think this is definitely a problem that needs to be looked into.

-Caitlin
Team 2022 IRS

Re: "No Comms"
 

Just a small correction - I think Joe meant to say "If you latch the data from before the transition to autonomous enabled, you can keep the switches on the DS".

I was helping numerous teams with this and some other control system "gotchas" at the NJ regional this past weekend. As long as you read the DS inputs during startup/initialization (while the match is in auto disabled) and save the info somewhere (like a functional global in LabView or a static in C++), then you can use that info during auto enabled. We use DS switches to select which autonomous mode we want to run - much more convenient than having them on the robot.

Thanks,
Ron
Team 2607 - software mentor

Re: "No Comms"
 

I'm not sure that's 100% accurate as written. "No Comm" does indicate a failure of communication between the DS and the cRIO, and that could certainly be due link failure between the robot and field wireless access point. But as I understand the network architecture, it could also be due to a number of other causes:
- dropped UDP packets on the field network, for example due to traffic congestion, radio interference, etc
- a software error in the cRIO firmware causing comms to be momentarily interrupted
- a software error in the DS firmware
- etc, etc

Based on my observations, I feel 99.9% certain that that is NOT correct. It looks like all of the DS positions have at least some components of their comm links which are independent - for example:
- each DS position has it's own ethernet tether back to a patch panel
- it also sounds like there are six different individual wireless AP's on the field, each with it's own SSID and WPA config which is setup through FMS at the start of the match
- etc, etc

Saying that a comms failure automatically indicates a problem with the robot is not only incorrect, it also implies that the problem lies entirely within the teams' control to diagnose and resolve. Based on what I observed at the NJ regional this weekend, with multiple unpredictable comm failures on the field, I don't think the problems are solely within the teams' control.

I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem. There are definitely issues with the DS<->FMS<->cRIO comms during matches, and there are a number of different parameters which should be checked, double-checked, re-checked, and stress-tested :)

If there's anything I can do to help out with this, please feel free to let me know via PM. It was personally disappointing to see how these issues adversely impacted so many teams' experiences during the NJ regional - due to alliance pairings, it impacts not just the team(s) having a problem but also many teams (such as #2607) which did not have any comms problems.

On a brighter note, BRAVO to the folks running the field this weekend - they did an outstanding job trying to find work-arounds and get the matches running as smoothly as possible.

Thanks,
Ron
Team 2607 - software mentor

Re: "No Comms"
 

Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.

Re: "No Comms"
 

We have a running QOS log on every switch in the network that can be accessed through a dedicated VPN network running back to the FMS engineering server, QOS records any error in communication that switches have, I did not find any FMS network errors at the events I was able to check over the weekend including NJ and NH. I have an engineering log running on every robot running during any match which logs every DS message recieved and records it along side the signals we recieve from the PLC IO, I have not found an instance yet where a DS dropped and the PLC IO errored out at the same time (PLC IO communicates every 20ms), still reviewing. The logs from the DSes that dropped out over the weekend during matches indicate that DSes stopped sending FMS status messages, FMS timed out listening for the messages and went into try to reacquire pinging to get the DSes reacquired. What this means is the DSes reset. It seems the DSes are resetting from the EMP that occurs when a robot that has a high voltage charged up from spinning its wheels out in the middle of the field gets in proximity of a field boundary (aluminum structure that is grounded) and does the EMP zap. Robots hitting the end of the field aren't resetting DSes from the physical impact, its the EMP discharge.

Looking into ways to mitigate the susceptability of the DSes to EMP because of the massive static issues that are occuring at some events. You'll notice the issue gets worse towards the end of a tournament because there's usually more static generated then from increased robot action in the elims.

I know its easy to blame the field because this was the first time you all were introduced to it, but I tried to account for these type issues in my designs. My background is semiconductor automation, so I know something about static sensitivities.

New complex system put together with hundreds of other complex systems for the first time, things to figure out, this is engineering...

We need to isolate the issues and then present the solutions. I put this out so you would know what I do at this time. Static is not our friend.

Edit: also, the network is gigabit, we're only utilizing a couple percent of it now, no traffic "congestion" issues.

Re: "No Comms"
 

Thanks for the explanation and the status update Bob!

Re: "No Comms"
 

Thanks for the update and thorough explanation, Bob - MUCH appreciated!

Sorry if the tone of my post came through as harsh, or as though I was blaming anyone (including the FMS). Was definitely not trying to place any blame - on the contrary was just getting concerned that some folks on this thread (not you) seemed to be directing the issue onto the teams' robots, and didn't want to see any effort get misdirected prematurely. Sounds like my concerns were totally unwarranted.

Hats off to everyone working on these things!

Re: "No Comms"
 

I'm living in the Hatch shadow, I'm used to getting beat on. And the issue we run into these days is all the admin personnel in these events now look to FMS to solve every issue, its a tough challenge :D

Just like you guys I want nothing more than to have perfect running competitions, I think the teams deserve it. But because of the time frames and resources available sometimes this doesn't seem to work out. Remember last years week one events ran pretty well, that was the second year of that FMS system. FMS is now is now about 3X the complexity of the IFI system and I don't think its that far off from being dialed in for great running events. This is a new control system (both FMS and the robot systems) and obviously we have to get it characterized.

The initial link up issues, the DS drop issues, I saw over the weekend where teams had stops in their cRio code that reset the bot on watchdog timer, we see that when the cRio voltage jumps to 37.37 on the status screen. All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out.

We'll get it figured out, we're determined, there are a bunch of really smart people working on this....

Re: "No Comms"
 

Hmmm... That sounds suspiciously like real-world problem-solving and engineering to me! Did anybody ever say the game was the only challenge we're supposed to be solving?

Re: "No Comms"
 

In Chicago the robots that had the issues listed above (working for auto and 30-60 seconds into the match) had the same problem. Teams had wired the wireless power to a breakered output of the PD rather than the radio power output. When the battery falls to less than 12 volts the radio suddenly cuts out That is why the PD has a special regulated 12 volt output. Inspectors will check for this next week when it is added to the inspection checklist. Pleased follow the control system instructions on how to power the radio.

Re: "No Comms"
 

Alan--

Trying to get a better understanding of this, what distances are you talking about? Should the WGA be at least 6" away from the Jaguar(s)? a foot? more?

thanks