![]() |
"No Comms"
Has anyone experienced this during competition? I know in OKC there were a few that had problems. Our team lost communications 5 of the 8 qualification rounds. Our bot would simply quit in the playing field. I spoke with nearly everyone there and got nowhere. Everything they told us to check had already been replaced in the pit. Auto would be fine, then about 1/2 way through tele, we would loose communication. During our last elimination round we didn't even have an auto which caused us to be eliminated. There is no way we were loosing power to the receiver. We troubleshot that to death and could not get it to fail. But that always seemed to be the default answer I got when asking around.
Any suggestions??? Anybody experience the same? |
Re: "No Comms"
Sounds like the problem a few teams were having with the Drivers Station. The Ethernet ports on the DS seem to break easily, so once one breaks, make sure to tape over it. We had Port 2 break.
|
Re: "No Comms"
this happened to our team at the NJ regional. We beilieve it was the server on the fields fault because it happened to numorus teams on each alliance over the three days
|
Re: "No Comms"
At GSR they re-ran about 12 matches and were down with comms problems for most of the morning thursday.... Our robot lost about 3 times cause of comms issues....
HERE IS HOW THEY FIXED IT!!! 1. Plug in your DS and wait for it to fully load.... 2 THEN turn on your robot and wait for it to get comms Also make sure that your gateway isn't tucked inside your robot somewhere that caused half of our problems. Also at GSR they ended up sprinkling water in the driver's box to help cut down on static... So be careful of placement of your gateway and such and you should be fine.. Oh and one more thing: With the new DS update given at the competition it actually says "no comm" not "no comms"... Just saying :rolleyes: |
Re: "No Comms"
Quote:
Quote:
http://usfirst.org/community/frc/content.aspx?id=10934 we found it the day before crating up the robot, it kind of messed up our plans for autnomous control. |
Re: "No Comms"
Quote:
On a related note (perhaps obvious to most folks) but since the practice sessions have two portions, we needed to reset the cRIO between the portions in order to ensure proper software initialization of states. Otherwise we ended up with the system carrying over states from the first portion into the second. FYI. We are using the Labview basic framework so I'm not sure how much different the Labview advanced framework or C++ implementations would be. |
Re: "No Comms"
Might be that annoying VxWorks bug showing it's ugly head again...
That happened to a few teams at Traverse City. Make sure you have all of the latest updates for your hardware and software. |
Re: "No Comms"
Quote:
Can you please expand on, or explain, this comment. I'm not sure I understand how updating the DS impacts Autonomous. Inquiring minds want to know. |
Re: "No Comms"
Quote:
1> Mounted the Wi-Fi on the robot away from all high EM fields (motors) and Faraday cages (steel). Orientation does not seem to be an issue but location is important. 2> Kept good care of our ethernet connectors (both on the DS and on the robot). We inspected them for bad spring pins, made sure we inserted the connector straight, and made sure no debris got into them. 3> Grounded the DS chassis as requested by FIRST. 4> always had our DS fully booted before booting the robot. 5> all electronics are mounted on non-conductive surfaces. After some observations, I believe that #5 above is the culprit for most teams. Our robot (with it's propellers and wheels) generates a tremendus amount of static when on the regolith. We saw at least 2 machines have a 'No Comm' failure after an impact with our robot. My recomendation to all teams is to ensure that the robot chassis is isolated from all electronics. |
Re: "No Comms"
Quote:
Nothing like being behind the glass and seeing Dean Kamen on the field spraying water on the carpet between matches :D |
Re: "No Comms"
"No Comms" is not a DS problem - it's a failure of the link between the robot and the field wireless access point. Unless ALL of the robots lose their links in a match, the problem lies within the robot.
Here are some likely causes of the problem: unplugged or intermittent power/ethernet connections, burying the adapter in the robot, problem with the wireless adapter +12V supply from the PD module and mounting the wireless adapter such that the security button gets pushed as the robot is bumped.] If you momentary lose the wireless adapter, it can take up to 45 sec for the robot to get synch'd with the field - very painful! |
Re: "No Comms"
The ultimate conclusion is still being investigated, but I'll tell you what I know:
Robots spin wheels on FRP, plastic on plastic with a metal frame to attract the charge. robots build charge, like helicopter, get near field border, being its metal, grounded, you get EMP. DS seem susceptable to EMP. So for all you people who saw a robot hit a field end and DSes reset and you thought it was because the physical impact knocked a connection, doesn't seem to be the case, it was the EMP discharge. This would happen anytime a robot had been running around in the middle of the field for a few minutes and then got near a field border. All the FMS equipment is sheilded for this, I check my engineering logs on fields like BAE, no issues I could find. I run QOS on all the switches, sat and watched the field end switches for a while at BAE while matches were running, no issues I could see. A suggestion to help. If you have your DS mounted in a control case. run a ground wire from the DS mode FIRST already mandated to a grounding connection that contacts the drivers table when you set your drivers station on the table, this will help to establish a faraday cage around your DS internal circuitry. If your DS sits directly on the table, it should ground. We're still looking into whether the EMP being caught by the DS to SCCE ethernet cord and traveling the cord to the DS is resetting it. It looks like it doesn't affect the SCCE switches as they are industrial switches sheilded for this and I can't find issues in their logs. The link up issues seems to be this (still being investigated): When the WGA bridge starts up in security mode, if it doesn't see its SSID within a certain amount of time, it times out. So the general rule with this should be, DO NOT turn on your robot till you see your team number on a team station. Reason - this means the scorekeeper already activated the network configuration for your match, i.e. FMS is or has written your SSID to the Access Point, and by the time your WGA bridge boots up, your SSID will be present and it will be able to connect without timing out. This is why it seems once things got rolling, things went more smooth, but when trying to restart in the morning or after lunch where things would get out of "sync" there were issues. These are all my theories currently YMMV Will keep investigating problems. |
Re: "No Comms"
AT Traverse City, the RObodawgs and Robodawgs O.T.L. Expierienced lots of this!
|
Re: "No Comms"
Quote:
There were instances at GSR i know cause it was our robot where ours and only ours died and they ended up replaying the match due to a field error not our robot.... |
Re: "No Comms"
Quote:
We also did not withhold our control system to work on programming...based on our experience of making some simple changes in one place, and several other things not working as a result, all during build season. We put a mostly working robot in the crate, hopefully we'll make all our practice matches, and fix the programming problems one at a time as time allows at the competitions. |
Re: "No Comms"
Quote:
On a side note, we were also told about the 45 second "re synch" however, during our final elimination, our bot lost coms at the beginning of the match and NEVER regained comms for the rest of the match...about 1:45...so I don't buy that either. |
Re: "No Comms"
Quote:
|
Re: "No Comms"
Quote:
|
Re: "No Comms"
Thanks, I'll tell Kevin!
|
Re: "No Comms"
In the Midwest Regional, Team 2022, my team, had this happen to us several times. Many times our bot wouldn't connect up to the field and I would have to run out onto the field and power cycle it. By Saturday, I had to do this before every single match that we played.
I also never thought of the static electricity problem. We definitely had it happen where were running fine and then would run into a barrier on the end of the field, and we would all of a sudden get "no comms". The field said that they didn't disable us or anything, and the wireless connections were all still in, so that had to be the problem. Another issue that we had at one point was the ethernet port on our DS. EVERYBODY CHECK YOUR ETHERNET CONNECTION ON YOUR DS. We found on ours and several others that one of the ethernet connections was up too high, so that when you plug in the ethernet cable, the clip that keeps it in place doesn't click because it is constantly pressed down. We just taped over that ethernet port and used the other one....but this is definitely something that caused us to lose a quarterfinals match in the elimination rounds, so check yours. That's just my 2 cents on the matter....but no comms issues made us lose many many times in the qualifications matches, so I think this is definitely a problem that needs to be looked into. -Caitlin Team 2022 IRS |
Re: "No Comms"
Quote:
I was helping numerous teams with this and some other control system "gotchas" at the NJ regional this past weekend. As long as you read the DS inputs during startup/initialization (while the match is in auto disabled) and save the info somewhere (like a functional global in LabView or a static in C++), then you can use that info during auto enabled. We use DS switches to select which autonomous mode we want to run - much more convenient than having them on the robot. Thanks, Ron Team 2607 - software mentor |
Re: "No Comms"
Quote:
- dropped UDP packets on the field network, for example due to traffic congestion, radio interference, etc - a software error in the cRIO firmware causing comms to be momentarily interrupted - a software error in the DS firmware - etc, etc Quote:
- each DS position has it's own ethernet tether back to a patch panel - it also sounds like there are six different individual wireless AP's on the field, each with it's own SSID and WPA config which is setup through FMS at the start of the match - etc, etc Saying that a comms failure automatically indicates a problem with the robot is not only incorrect, it also implies that the problem lies entirely within the teams' control to diagnose and resolve. Based on what I observed at the NJ regional this weekend, with multiple unpredictable comm failures on the field, I don't think the problems are solely within the teams' control. I would STRONGLY encourage the folks handling the FMS to apply sound engineering and diagnostic principles to this problem. There are definitely issues with the DS<->FMS<->cRIO comms during matches, and there are a number of different parameters which should be checked, double-checked, re-checked, and stress-tested :) If there's anything I can do to help out with this, please feel free to let me know via PM. It was personally disappointing to see how these issues adversely impacted so many teams' experiences during the NJ regional - due to alliance pairings, it impacts not just the team(s) having a problem but also many teams (such as #2607) which did not have any comms problems. On a brighter note, BRAVO to the folks running the field this weekend - they did an outstanding job trying to find work-arounds and get the matches running as smoothly as possible. Thanks, Ron Team 2607 - software mentor |
Re: "No Comms"
Two teams at the DC Regional had issues where they'd lose communication when driving. Both of them had the WGA mounted adjacent to a Jaguar. After moving the WGA to a less EMI-prone location, both had no more problems.
|
Re: "No Comms"
Quote:
Looking into ways to mitigate the susceptability of the DSes to EMP because of the massive static issues that are occuring at some events. You'll notice the issue gets worse towards the end of a tournament because there's usually more static generated then from increased robot action in the elims. I know its easy to blame the field because this was the first time you all were introduced to it, but I tried to account for these type issues in my designs. My background is semiconductor automation, so I know something about static sensitivities. New complex system put together with hundreds of other complex systems for the first time, things to figure out, this is engineering... We need to isolate the issues and then present the solutions. I put this out so you would know what I do at this time. Static is not our friend. Edit: also, the network is gigabit, we're only utilizing a couple percent of it now, no traffic "congestion" issues. |
Re: "No Comms"
Thanks for the explanation and the status update Bob!
|
Re: "No Comms"
Thanks for the update and thorough explanation, Bob - MUCH appreciated!
Sorry if the tone of my post came through as harsh, or as though I was blaming anyone (including the FMS). Was definitely not trying to place any blame - on the contrary was just getting concerned that some folks on this thread (not you) seemed to be directing the issue onto the teams' robots, and didn't want to see any effort get misdirected prematurely. Sounds like my concerns were totally unwarranted. Hats off to everyone working on these things! |
Re: "No Comms"
Quote:
Just like you guys I want nothing more than to have perfect running competitions, I think the teams deserve it. But because of the time frames and resources available sometimes this doesn't seem to work out. Remember last years week one events ran pretty well, that was the second year of that FMS system. FMS is now is now about 3X the complexity of the IFI system and I don't think its that far off from being dialed in for great running events. This is a new control system (both FMS and the robot systems) and obviously we have to get it characterized. The initial link up issues, the DS drop issues, I saw over the weekend where teams had stops in their cRio code that reset the bot on watchdog timer, we see that when the cRio voltage jumps to 37.37 on the status screen. All sorts of crazy stuff, there's alot to learn, you all are part of the process, we can help each other out. We'll get it figured out, we're determined, there are a bunch of really smart people working on this.... |
Re: "No Comms"
Quote:
|
Re: "No Comms"
In Chicago the robots that had the issues listed above (working for auto and 30-60 seconds into the match) had the same problem. Teams had wired the wireless power to a breakered output of the PD rather than the radio power output. When the battery falls to less than 12 volts the radio suddenly cuts out That is why the PD has a special regulated 12 volt output. Inspectors will check for this next week when it is added to the inspection checklist. Pleased follow the control system instructions on how to power the radio.
|
Re: "No Comms"
Quote:
Trying to get a better understanding of this, what distances are you talking about? Should the WGA be at least 6" away from the Jaguar(s)? a foot? more? thanks |
Re: "No Comms"
Quote:
:D |
Re: "No Comms"
Thanks, that's what I wanted to hear! We have the WGA on top of the wood electronics box now, not sure if it's 6" away from the nearest Jaguar, but we can make that happen if we still have problems.
|
Re: "No Comms"
Quote:
Quote:
|
Re: "No Comms"
Quote:
Quote:
I only heard about the other team second-hand, so I don't actually know how near the parts were to each other, but I got the impression it was less than a couple of inches. I'll try to find out details from the Spare Parts volunteer who helped them correct the problem when they requested a replacement WGA. |
Re: "No Comms"
1 Attachment(s)
Ok, thanks! This is how we had our WGA when we had some occasional no comm problems, it's now on top of the box, sitting flat.
|
Re: "No Comms"
Let us not forget that any time the gaming adapter is trying to connect or is rebooting or has lost a security packet, you will see "No comm".
|
Re: "No Comms"
Still investigating the "no comms" dropout during match issue, but it does appear its the network switch chip reset in the DS being susceptable to EMP. I can repeat the drop of multiple DSes connected to different SCCEs while a match is running with an EMP event. Logs show the reset takes ~31 seconds for the DS to reconnect with FMS after such an event. No FMS issues that I can find when the event happens. Why the reset trips is still undetermined and I don't know if we can come up with a quick fix for it. Will relay more as I know.
|
Re: "No Comms"
Thanks for the update!
|
Re: "No Comms"
During our competition on Saturday at NJ our robot failed completely to operate during autonomous. During teleop motors connected to spikes worked well, motors connected to Jaguar speed controllers did not operate. Everything worked perfectly on Friday. Any suggestions?
|
spikes work/jaguar not, suggestions?
During our competition on Saturday at NJ our robot failed completely to operate during autonomous. During teleop motors connected to spikes worked well, motors connected to Jaguar speed controllers did not operate. Everything worked perfectly on Friday. Any suggestions?
|
Re: "No Comms"
Update:
Sent a bunch of test data to FIRST and DEKA yesterday confirming the DS Network chip susceptability to EMP. Today they were able to repeat my results, so it seems confirmed. We can't fix the DS easily, but we're working on a bunch of "workarounds". FIRST is looking into ways to eliminate static generation, which if they can will fix the source of the issue and I've been testing some FMS network mods today that enable the DS to do a minimal association with the the FMS network. I.e. now when the DS network chip resets on an EMP event, I can have it talking with both FMS and the robot in under 2 seconds. If the DS completely reboots its a few more seconds but it still comes back fast. Not the ideal situation, but it should help. Hopefully we won't have any static issues this weekend. Oh, and the added extra benefit of the network mods? Looks like we dropped about 20 seconds off the initial DS/robot/fms linkup before a match starts ;) Real engineering: see a problem, analyze the data, if you can't "fix it", find the work around :ahh: |
Re: "No Comms"
Quote:
It seems that in very close proximity to the router, the orientation does not matter much. However, as the robots get farther away (it was still less than 30 feet, unobstructed, from the router), the orientation matters. We found that it is best to place the WGA on a flat plane (bottom of WGA parallel to ground) |
Re: "No Comms"
Quote:
|
Re: "No Comms"
For people interested in finding out more about ESD sensitivities and ways to mitigate. Do a google search on "triboelectric series"
|
Re: "No Comms"
We had our Linksys wireless bridge go bad. We took it to Pit Admin to have it programed and they said that the setting were corrupted. They would correct them, and unplug the box, plug it back in and it would be corrupted again. We borrowed one from replacement parts and out bot worked fine during the rest of the competition. Luckily the problems were only during practice matches.
|
Re: "No Comms"
Quote:
Ron |
Re: "No Comms"
Would it be helpful to put a small piece of wire rope dragging the floor to discharge the static. They have done this on carts at stores to solve static problems there.
|
Re: "No Comms"
Quote:
|
Re: "No Comms"
Quote:
1) Gaming Adapter should, at no time, be blocked by a piece of metal on any side. I have seen teams that have mounted the adapter on a sheet of metal more than 1/8" thick. 2) Try to keep the adapter away from motors. Motors generate a magnetic field, which at times can interfere with 802.11x communications. 3) Ensure that you are using the correct cable type, and that each cable is properly seated and secured. Ethernet cables are definately more durable than a PWM, but they are suseptible to Electro-Magnetic interference. I have actually ordered some shielded Twisted pair for our next regional. Being a network technician, I did a simple site survey at the trenton regional, while the fields were not in use. I found a plethora of 2.4Ghz communication. Much of it was the field communication system, but at trenton there was a fairly strong wireless signal active (Trenton Devil's Press Box). Another fairly important fact.... While bluetooth signals are not very strong they can cause interference with b/g/n networks. Just my 2 cents... |
Re: "No Comms"
Quote:
Quote:
Quote:
|
Re: "No Comms"
Quote:
|
Re: "No Comms"
Quote:
As for the wireless communication, I know that I was seeing massive comms across the 2.4 Ghz network, and I guess it was possible that it was the arena's wireless, but coverage across all 11 channels in the 2.4 Ghz range is abnormal, as one would normally try to keep it seperated out to three non-overlapping channels (Normally 1/6/11). As with rule R57 bluetooth is not permitted on the robot, that is very smart on FIRST's Part. Last year, while reviewing issues on a wireless system, we noticed an issue with bluetooth clients that were crowding the network, and therefore providing interference across the spectrum. Microwaves, and HVAC motors have been shown to pose similar problems. I still highly recommend moving the adapter way from any large metal plates, and keeping them away from motors (within reason (5-8"))... |
Re: "No Comms"
The FRC arena has an intelligent Access Point (Cisco 1252), when it starts up it scans the entire spectrum and then tries to establish itself on the least interferring channel. You can force it to a channel, but right now on the road they are all in defualt, which means they'll do this. When we get to Atlanta, I'll have to force them to a certain channel for coordination with 5 arenas.
|
Re: "No Comms"
We experienced no comms during competition and tracked it down to one Fisher Price motor being driven by a Jaguar. The WGA was mounted high and well away from the motor however the Jaguar was only a few inches off the CRio near the ethernet port. We kept getting comms drop outs even when tethered. When we disabled the motor, problem went away. We then also ran the motor but at lower PWM levels and the problem went away. At high PWM values the dropouts came back. We swapped the Jaguar for another without any impact. We swapped the Jaguar for a Victor without any impact. We are implementing a rerouting of all wiring to keep all power lines away from the CRio. Putting EMI suppression cap on the FP motor to the extent legal. The problem did not crop up until elimination rounds, so perhaps brush wear or contamination had an impact. The above troubleshooting strategy seems to have pretty definitively pointed the FP motor's brush noise out as the source. Better electrical board layout may have helped to prevent the failure. Hope this can help someone else. Thanks to Eric of 1511 for his help.
|
Re: "No Comms"
Please check the Q&A http://forums.usfirst.org/showthread.php?t=12228
Dragging wires and other attempts at fixes to esd issues are not to be used while First researches the problems. Until further notice, please standby. |
Re: "No Comms"
at our regionals there was a static electricity problem that would make the robots stop talking to the DS except when you had pneumatics
|
Re: "No Comms"
Quote:
I'm in the camp right now that bots should have a drag chain, I don't know what the hold up is on that, ultimately its FIRST's decision. I know they are contemplating putting drag chains on the trailers for this weekend, I'd prefer that they give teams the option to put them on their bots as we've proved that the KOP wheel from this year is very efficient at isolating static voltage buildup. On another note: FMS ran well this past weekend, we had a couple equipment issues with Estop that were manufactured incorrectly. But most other issues seen were related to robots and ESD. Hopefully, tomorrow we'll have some resolution in the team update. |
| All times are GMT -5. The time now is 14:43. |
Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi