COMM lost way to many times
 

We competed at Buckeye this past weekend and suffered comm issues in most of our later matches on Friday and Saturday.
We had lots of folks trying to help us figure it out from the FTA's and other field personnel to mentors from other teams.
Here is what I know.
From looking at logs and reported observations from the FTA's they see the radio rebooting. Immediately after they see the radio return they see the RIO so the belief is that the RADIO is the only think rebooting as the RIO takes longer than the radio and would not show up immediately after the radio if it was also booting.
Even so. The initial thought, before we found out the above, as that we had loos battery connections. We went over all our batteries and did find some that were marginal so we tightened ALL our battery connections regardless if they were loose or not. This did not solve the problem. The problem could not be duplicated in the pit no matter what we wiggled and banged on. We build a practice bot and had no issues with that all preseason. We had the radio from that along so we swapped it in and no joy with that either.
We pulled and re-seated all the wires in the weidmuller connectors on the PDP and VRM and tug tested. Again, no joy. By this time we were past our second match of elims and got subbed out. Not a problem, I would have done the same thing.
Before we bagged we replaced the radio power cable with the one we used on our practice bot and replaced the VRM with a brand new out of the box new one.
Have not had a chance to test since.
Does anybody have any idea what else we should try if this does not work?
Thanks

Re: COMM lost way to many times
 

have you guys checked the mini automotive fuse. my team just unbagged our robot yesterday and it drove fine until we tried to go over any defenses. those fuses can come loose but still look like they are seated. hope this helps.

Re: COMM lost way to many times
 

We had a similar problem and discovered that one of the fuses on the PDP was loose. (I believe one of the small fuses at the bottom of the PDP goes to the VRM?)
After making sure it was securely pushed in, we didn't have further problems.
That might be something to look at

Re: COMM lost way to many times
 

Did you check that the yellow and red fuses were pressed all the way into the PDP? When they are inserted fully, there should only about 1/4 inch of the plastic showing. In Kansas City, there were a lot of robots with similar issues. The CSAs at the event checked all of the robots Friday afternoon and a lot of the roboRIO and Radio issues went away.

Re: COMM lost way to many times
 

we had several folks look at the fuses and they were deemed OK but I will pull them and insert new ones Thursday.
Thanks

Re: COMM lost way to many times
 

When does the radio reboot occur? Can you post links to matches where the radio is rebooting?

1. Since you've replaced everything else, consider replacing the wire from the PDP to the VRM. Be sure you're using stranded wire.

2. Make sure that all of the wire along the path is tacked down with zip ties (use zip tie bases if needed). You want to minimize wiggling.

3. How is your radio fastened to the robot?

Re: COMM lost way to many times
 

I know I am a flame thrower on this topic so I should probably just sit on my hands and watch this thread, but I will say this much, as a fan in the stands at UMass Dartmouth over the weekend, it was galling to watch team after team (and probably more veteran teams than rookies) sit dead on the field for close to a minute while the radio reboots (or reconnects or whatever -- their dead and that orange light isn't flashing so you can be sure that they are not coming back alive for half the match if at all).

I know, I know, I KNOW, it is almost always a power issue for the radio. I hear you, FIRST, but seriously, you need to stop blaming the teams on this matter.

When you run a remote controlled robot tournament, the first and primary contract you make with the participants that you promise them a solid data link to their robot.

I believe that FIRST is failing in this promise.

I argue that we are at the point where we need to either

find an idiot proof solution (that a team can do whatever they want and the radio link stays alive)

or

inspect in the set of behaviors that gets to a solution (e.g. every robot going on the field gets inspected for the proper wiring connections, strain reliefs, etc.).

Blaming teams was never a great plan but this year with such long reconnection times, it is just unsupportable.

Dr. Joe J.

Re: COMM lost way to many times
 

The problem is solved with hot glue.

Re: COMM lost way to many times
 

I would LOVE to see FIRST legalize opening up the Radio and soldering a power cable directly to the PCB pins. Not having a solid, locking connector is the biggest issue with the radio, and if teams could solder to create their own locking connector, maybe with a powerpole that would be great.

Re: COMM lost way to many times
 

1712 had too many simultaneous fixed to truly isolate which mattered and which didn't (we didn't have time to only change one variable at a time in the heat of competition). But I do believe that getting better strain relief on wires was a big part of us solving our roboRIO power issues, and may help you with your radio issues. Ensure your wiring has a sufficient bend radius near either end, and then get it tied down securely. Even momentary losses of power can cause reboots or brown outs. Check (and possibly replace) both your power and cat5 connectors. We also added an elastic band (hairtie) around our radio to help all the connectors stay plugged in. Further still, we cushioned our radio mount with foam to help absorb shock.

Re: COMM lost way to many times
 

This sounds like a power quality issue. Bad wires, bad crimps, bad connections, bad barrels all can contribute to this.

I would replace all of the wires from the batter to the radio, especially the black wires that go directly into the radio. Then I would reprogram your firmware on the radio in particular. Inspect your fuses, many people don't notice when they are bad. Drive the robot around at home, drive over bumps see if you can replicate these issues. Make sure all of your wires are anchored down appropriately and nothing can shake loose. Make a good effort to replicate the driving conditions on the field, move around fast and crash frequently.

With how difficult the field is on robots this year, i'm not surprised that we are seeing a rise in power quality related issues on the communications.

Re: COMM lost way to many times
 

What is the battery voltage showing on the logs? How old are your batteries?

[Bad advice retracted].

Try driving the robot in a stall condition by putting it against a wall and driving forward. See if your radio restarts.

Re: COMM lost way to many times
 

Please don't do this. Even doing this once is enough to damage the terminals on the battery even if the crimps are lose. Its much better to take them off of the battery, and test the crimps off the battery.

Also teams, if you are crimping your own Andersen connectors, you can actually purchase a hydraulic crimper from Harbor Freight for not too much money. We bought it a few years ago, and love it. Haven't had a lose crimp in years, and we even cut a few apart to inspect the test crimps and they are really solid.

http://www.harborfreight.com/hydraul...ool-66150.html

Re: COMM lost way to many times
 

That is a valid point. I would recomend pull testing them a better way.

Re: COMM lost way to many times
 

with velcro not the regular stuff that kind that sticks to itself whatever that is called.
Been doing that for years

Re: COMM lost way to many times
 

hot glue where?

Re: COMM lost way to many times
 

ya but the robot is in the bag

Re: COMM lost way to many times
 

we have all kinds of batteries but we lost comms some times when we were just going under the low bar. No big current draws, no big bumps.

Re: COMM lost way to many times
 

We had some issues with our radio and/or FMS at the Tech Valley Regional this weekend.

One of them was on us (tug test failed on our radio power connector), the other was indeterminate. (CSA/FTA was blaming robot power, as a whole, but we have video showing no power loss on the RoboRio.)

Regardless of what the problem is, even as a rookie team, the instrumentation provided by the RoboRIO/PDU/FMS/DS completely fails during problem events.

During the aftermath of our issues, we attempted to use the DS logs to trace voltage, power draw, comms, and log messages from the RoboRIO. The problem is, you can make "guesses" about your voltage state in the fraction of a second leading up to the event, but once it happens, you're blind.

We're already redoing our robot code library for next year to address some of these shortcomings. We'll be logging voltage and PDU stats, as well as bridge/radio and FMS connectivity via ethernet directly on the RoboRIO. In addition we'll be using a cached-logger, such that if DS communication is lost we can back-fill in our logs.

We are attempting to address the issue of the complete lack of FMS logging, and the logging dropout during any communication loss event. We're also pushing the logging closer to the source of the data. Our goal is to be able to document the voltages, RoboRIO, PDU, and Radio/FMS/DS status in an effort to diagnose issues.

Re: COMM lost way to many times
 

Do a pull test on all wiring connections. That means where a wire goes into a Weidmuller or Wago connector, where a wire goes into a crimp lug, where a wire goes into an Anderson connector and where the wires go into the plug for the RoboRio power. If wire slips out, re-do the connection.

Check all screw/lug connections such as the battery cables on the battery and on your main breaker. The wire and lug should not be able to rotate relative to the battery or breaker. Tighten the screws if they move.

Use a flashlight and examine closely where the wires go into the small Weidmuller connectors (i.e. RoboRio power on your PDP). Look for stray strands on any wire that is not in the connector. These stray strands can touch adjacent wires which is always the opposite polarity on the PDP, VRM and PCM. If necessary, cut off the "crinkly" stripped ends, strip to the length shown in the Users Manual for the PDP, VRM and PCM, twist the exposed strands so they lay neatly together then re-insert the wire in the Wiedmuller connector carefully and ensuring that ALL the strands go in.

Re: COMM lost way to many times
 

What in the video establishes that there was no power loss on the RoboRIO? If you're just looking at the Robot Signal Light, it's possible that your RoboRIO browned out due to power loss, but didn't completely shut off.

Re: COMM lost way to many times
 

We solder and heat shrink where we can, and hot glue everywhere else. So the connections to the Voltage Regulator are all hot glued. We were seeing all sorts of radio drops out just as you've described on both our comp and practice bots. Since hot gluing the radio wires into the voltage regulator we haven't lost comms. (~19 matches).

We use hot glue on nearly every connection, including our encoder cables, limit switches, arduino plugs, etc.

If you want something less permanent electrical tape tends to work well enough, but I haven't seen anything better than hot glue this year.

Re: COMM lost way to many times
 

So everyone knows, the DS logs are able to tell you if the roboRIO rebooted.

When a connection is established between the roboRIO and the DS the roboRIO will send a message saying how many seconds have passed since it last booted up, the DS then stores this information in the log. Look at the messages when communication is reestablished to find this information.

Re: COMM lost way to many times
 

Why is everyone missing the obvious solution? Use passive Power Over Ethernet (POE) on the port that supports it. The Ethernet connector snaps firmly into place. I've never seen one come loose due to bounces or vibration this year.


On the other hand, more than half the robots I checked had loose terminals either on the battery or the main circuit breaker. All of the "lost communication" events I followed up on at both of the events I worked at were associated with a faulty power connection. All.

(I'm counting only ones where there was actually a loss of communication, not the few the drivers called "lost comm" but where the problem was that robot code crashed or a joystick came disconnected or the Driver Station was being interfered with by a software update process.)

Re: COMM lost way to many times
 

I'm not familiar enough with PoE to be comfortable without some details. I sure don't want to fry a radio. With a quick search I haven't found out the best way to wire a PoE for this this radio. I assume you can use the 12 v port on the VRM into a spliced ethernet cable but what pins are used and which port does it go into in the router. I apologize if this has already been discussed. I didn't see it in my searching, but might have been looking for the wrong things.

Re: COMM lost way to many times
 

FIRST should consider allowing the use of a small secondary battery to power the radio should it lose connection with the VRM/PDP. The battery would be used exclusively for the radio and prevent any reboots due to power loss or brownouts elsewhere on the bot.
Consider that batteries and the PDP are generally well-hidden on the bot (the battery to lower CG and the PDP to shorten total wire lengths. However the radio needs good clearance from metal and motors to ensure a solid RF connection, leading to longer wires and generally less secure mounting (subject to more vibrations potentially). This all introduces risk of wire/connector failure. Allowing a small battery placed alongside the radio as a parallel backup would be beneficial.

Re: COMM lost way to many times
 

All the information I needed I found through the OpenMesh web site and a simple search for "passive POE". I summarized it here.

Re: COMM lost way to many times
 

Hot glue at the connection between the power plug and the radio.

I was the mentor from 379 that was trying to help. At the time we probably couldn't wait for a hot glue gun to heat up We reset all wires at the VRM. Checked other connections, inspected contacts, and zip tied the battery connection. Not knowing the robot, it was hard to inspect.

I do feel that it's something dealing with an electrical connection. Try replacing the Anderson battery/PDB connection, wire and all, from the robot. I wish we could have figured it out, you guys have a great design, good luck this season.

Re: COMM lost way to many times
 

Which port on the radio supports POE?

Re: COMM lost way to many times
 

I highly recommend against hot gluing the plug into the radio (and I also advocate against hot gluing or taping ethernet cables and power cables pretty much anywhere). It makes replacing those wires later more difficult.

The radio's plug is particularly problematic with hot glue. You can no longer easily detach the radio from your robot to program it at events, and swapping out radios should yours need to be replaced for any reason suddenly becomes much more difficult.

Furthermore, the issues I have seen are generally not that this plug comes loose. The problems are generally that the wires going into the Weidmuller connections on the VRM have too much copper showing and are shorting out, or that there are loose wires elsewhere in the chain.

I had a team that had mummified their Ethernet ports on both the radio and roborRIO along with the USB cables going into the roboRIO and the power plug on the radio with electrical tape drop out during a match. Seriously, there was a large mound of the stuff over everything. They asked me if I thought they should add more electrical tape since they were still dropping out. At that point I demonstrated that the wires going to their radio were just falling out of the VRM with very little effort on my part.

Understand the problem before trying to apply bandaids. You might not even be putting the bandaids in the right spot.

Re: COMM lost way to many times
 

I was going on the basis that our battery voltages were very good (and trending up), and our load was low at the time (and trending down). It was the CSA & FTAs opinion that we lost power completely and the RoboRIO rebooted, however the video of the lights on the RoboRIO (and RSL) didn't concur with that assessment.

Is it possible it was a brown-out, sure, but my point is that there isn't enough data because of the streaming-only nature of the logging mechanisms.

Re: COMM lost way to many times
 

Great callout Bryan. As he said, you should not have to apply hot glue on the connection between the radio and the plug, and that can definitely be problematic. We used hot glue to aid the Weidmuller connectors on the voltage regulator, but it certainly has drawbacks.

I'm a big fan of Alan's PoE idea for this particular problem

Re: COMM lost way to many times
 

Both Ethernet ports on the OM5P-AN support POE, each in a different way. The one nearest the power connector is labeled "18-24v POE", but the bottom of the device says "PoE Input: 12-24V Passive PoE" and it definitely works with the 12 volt output of the VRM. The other connector works with the 802.3af POE standard, which uses 48 volts -- higher than what is permitted by the robot rules.

It's a shame the radio programming kiosk specifies the second port. It would have been handy to use a cheap passive power "injector" and only require teams to connect the Ethernet cable in order to make it work.

Re: COMM lost way to many times
 

I imagine am certain that a wire properly inserted into a Weidmuller connector will hold more strongly than hot glue.

I just tested it by putting an 18 gauge wire in the +12Vin connection of a VRM, gluing the wire to a scrap of Lexan, waiting several minutes for the glue to harden, then easily pulling it off by lifting the VRM. I did it again with a piece of aluminum. I had no problem holding on to the hardened blob of glue and using the VRM to apply enough tension to pull the wire through the glue.

Most of the wires I saw falling out of Weidmuller connectors were not stripped nearly long enough. It should be 8mm (about 5/16") of exposed conductor.

Re: COMM lost way to many times
 

If you have one of the power cables for the radio that has a 90 degree plug (right angle plug), you can wrap a long zip-tie around the radio and the plug. That would be better than gooping up the power plug with hot glue.

Re: COMM lost way to many times
 

One common problem (that we had last year) which can lead to temporary power loss long enough to restart your radio is the main breaker. From what I've heard, there are a few every year that have this issue.

Try tapping your finger on the red button on the breaker, (the one you would use to power off your robot) if it causes your robot to power cycle, that's your issue - at least one of them. :)

Re: COMM lost way to many times
 

Does that video also show the Radio not losing power and rebooting? How long was the robot down for? What was the RSL doing at the time? A radio reboot is about 50ish seconds and then 10-15 or so more for the roboRIO to attach to the DS.

Do you have a link to the video itself? I'd like to see it for my own knowledge when I'm monitoring the field for my last 3 events this year.

Re: COMM lost way to many times
 

Not for sure if somone already said this; apparently the breakers that came in some of the kit if parts are bad. Our team had the exact same problem along with a couple others. We didn't know until a team informed us on our last match, we swapped it, then it worked pirfectly. Wiggle the red switch button to test it if all/some of your communications go down.

Re: COMM lost way to many times
 

We were having this same issue at the Australia Regional. After a while, we narrowed it down to the following 2 things:

a) The Circuit Breaker (and the wire that connects it to the PDP)

b) The Anderson Battery Connectors going from the Breaker/PDP to the Battery's Anderson Connector.

Replacing these seemed to solve our problem for all our future matches. Some breakers can be very sensitive to sudden stops (i.e. crashing into a defense), and faulty Anderson connectors was discussed in another thread.

EDIT: We diagnosed this by looking at the DS. If the comms go down, then comms go up and (later) code goes up (checking DS logs to confirm the signal loss was indeed with the radio), it means the total system did a reboot (radio AND rio), so you can usually diagnose that as the main power feed to the PDP (or the PDP itself), as the RIO and VRM (connecting the radio to power) are on separate connectors.

Re: COMM lost way to many times
 

OK here goes a list of problems and fixes...
Battery terminals even if tight can be intermittent for a variety of reasons. A simple fix is to add a #10, external tooth star washer between the wire terminal and the battery terminal when assembling the battery wiring. This simple addition will bite through any surface crud on the terminals and lock the two terminals together so that they cannot rotate. Use the provided locking hardware in addition to this one lock washer.
Push the PD fuses all the way in. When properly installed the fuse will be just 1/8" above the PDP surface. It should be hard to remove.
Strip the wires that are inserted into the various Weidmuller connectors to the required length. Often teams make these too short and the result is the wire is not properly retained. Do not tin the wire, these connectors are intended for bare stranded wire. If you pull the wire and it pulls out of the connector suspect the connector. I have seen these damaged by aggressive installers.
A small number (<1 in 100,000) of main breakers have a manufacturing defect. While the robot is on, lightly tap the red button. If your lights flicker, replace the main breaker.
Make sure that the terminals you use to feed the PDP input seats all the way down into the PDP terminals. Some terminals are a little too wide to fit the terminal and will not seat all the way down. This raises the series resistance considerably. Inspect these terminals with a bright light.
Use proper strain relief at each end of the wire. We use self adhesive wire tie blocks or wire tied directly to frame to keep everything from moving around.
This year's radio has antennas behind the vents on either side of the radio. Do not mount your radio near metal objects so that the vents are near the metal objects on your robot.
Observe when your radio reboots. If you have the issue when traversing the defenses, then it is likely a poor connection. If it occurs when turning, then it is a brownout issue.
Never, never, not ever, lift the battery by the wiring. The terminals on the battery are simply soldered in place inside the battery. This is a manufacturing method that allows attaching various terminal styles to the same basic battery. Lifting the battery by the wire will break the solder connection internal to the battery. You will never know you have permanently damaged the battery until you really need it.
I do not recommend hot glue for many reasons. It might give you a false sense of security and will make you look elsewhere when the problem is really under the hot glue all along. If you think about it for a few minutes you can come up with a much better solution.
If you do not use the power plug that came with the radio, then it is very likely you have the wrong diameter plug. They look alike but the center pin varies.

Re: COMM lost way to many times
 

Oh god this brings back bad memories from a couple weeks ago... At Q31(?) of the NYC regional we had some mounting issues with our top plate. Before we had proper time to fix it, we had to queue up for Q37(?), so we decided to go the match without it. After our partner pushed us out of the moat, our pressure release valve (which was normally mounted to the top plate) swung and hit the red button on the breaker.

That was a sucky way to lose a match, but hey. It happens.

Re: COMM lost way to many times
 

This is a great list, and pretty much the way I trouble shoot robots at competition with "loss of COMMS".

I will also add, Check the Ethernet connection on the laptop. IE, give it a wiggle. I found this was the source of intermittent comms drop for a couple teams at Centerline this past weekend.

This laundry list of items shows why it is such a frequent problem. While the folks at CTRE and NI and FIRST have done a great job at reducing the possible Idiot proofing, there are still so many ways that you can get an issue that low probability of many many things turns into a reasonably high probability that something will get you.

I.E. a 10% chance on any one item, but 10 items with that 10% chance turns into a pretty high probability that something will go wrong.

Pro-tip, when forming your alliance, ask if you can go through your partners robot with a fine tooth comb, and ask them to do the same for you. Often an extra set of eyes can help find issues and errors. Make you own checklists and review them often. Not sure what to put on the list? How about the good advice being dispersed in this thread.

These events the robots are going over are pretty violent. Even the weight of a wire or cable flopping around can be significant when the robot see a 20G pulse (1 lb acts like 20 lbs). Proper restraint and strain relief goes a long way.

Re: COMM lost way to many times
 

is it possible that the main breaker problem you reference can cause the radio to reboot without the RIO rebooting?
All indications are that our RIO is not rebooting.

Re: COMM lost way to many times
 

we had this issue at wpi 2 weeks ago, in all of our qualification matches except for two we lost communication we changed and checked everything, fortunately we were alliance captains, and going into our final quals match we simply decided to move the radio to a different spot and didn't have an issue with it after that something so simply cause suck grief .

Re: COMM lost way to many times
 

Ok I read that link and I want to make sure I understand how you are using this.
You take a port saver cord, open the cable somewhere and solder a red wire to the blue and white with blue stripe wires and a black wire to the brown and white with brown stripe wires then you stick those wires into the VRM, plug the port saver into the POE port on the radio and the ethernet cable from the RIO into the end of the port saver cable you just modified so you are using only ONE of the ethernet ports on the radio, correct?
Do you cut the wires in the port saver so power only goes to the radio or do you just solder to those wires so power goes to both the radio and the RIO?

Thanks

Re: COMM lost way to many times
 

We had this exact problem at TVR in our first four matches.

It wasn't the fuses.
It wasn't the wire connections.
After isolating and ruling out all other issues (and reproducing the fault by giving the PCP a shake), we swapped the PCP and the problem didn't repeat. [Side note: the PCP, barely used since coming out of the kit, rattled like a maraca when shaken. Back home, the simplest wiring setup possible, very secure, would lose voltage when shaken.]

In our second-to-last qualifying match, half of our pneumatic system failed to actuate. We checked our wiring and all seemed fine, and it worked in the pit (even after some jostling)--we couldn't reproduce the failure. It failed the next game as well, and we couldn't reproduce the failure. After QF 1 we swapped the PCM for a new one and it worked fine through our last two matches. We have not at this time tried the PCM in other situations.

It's aggravating to have failures in KOP devices that we're required to use on our robot, but cannot repair or upgrade to something more reliable. FRC requires robust components, and expecting teams to deal with internal failures of KOP parts used in a manner consistent with their intended purpose as "part of the game" is not acceptable--quality control on FRC KOP components should be top notch, and it simply isn't.

(The "tested" sticker on the back of the PCP was a bit of an insult-to-injury chunk of irony. They said it was tested...but never said whether or not it worked.)

Re: COMM lost way to many times
 

We plug the port saver into the roboRIO, not the radio. The power wires only need to reach from near the roboRIO to the VRM. Usually the radio is mounted some distance from the rest of the control system components, and it's convenient to run a single cable from the port saver up to the radio.

The Ethernet port next to the coax power input is the only port on the radio that accepts this scheme.

The blue and brown pairs from the "plug" side of the port saver are indeed disconnected in the one we have installed on the robot, but that was mostly for ease of construction. I have used a different "passive injector" during testing and verified that having power going to those pins on the roboRIO as well does not break anything.

Re: COMM lost way to many times
 

bump

Re: COMM lost way to many times
 

Extremely unlikely. Can you elaborate on the indications? The implication you were provided by the FTA\CSA earlier in the thread that the roboRIO takes longer to boot than the radio is false.

Re: COMM lost way to many times
 

Correct that the roboRIO boots quicker, but if both of them went down, then communications between the roboRIO and DS aren't going to come back until 10-15 seconds after the radio is fully back up (50 seconds or so itself). Net is 60-65 seconds before the robot moves again. That may be what the FTA was trying to say.

Re: COMM lost way to many times
 

Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?

I get the feeling that the problems are worse this year but I freely admit that I am a biased observer. If FIRST has data that shows that this problem is small and getting smaller then, well, I won't say I'll be happy, but I will at least feel better knowing that the ball is moving toward the right end zone.

Dr. Joe J.

Re: COMM lost way to many times
 

We had issues with our development robot. It would drop the radio almost any time we went over a defense. All of our other components stayed powered. The RIO never rebooted. We did several things to harden the control system:
1) made an improvised shock plate to mount the radio to. This dampens the shock transferred to the radio itself. There seems to be a strong indication that high G shock, as opposed to vibration, is an issue for the new radio.

2) We use a plug with screw terminals for the radio power. Once the wires were screwed into the terminals on the plug adapter, we soldered the wires in and insulated the leads. We verified isolation between the leads. Then zip tied the plug adaptor to the shock plate such that: 1) the tension relief moved as a system with the shock plate and radio, 2) would not move relative to the radio itself. We also zip tied the Ethernet cables to the shock plate.

3) For all wiring connections, we made tension reliefs, out of semi-rigid lexan and non-conductive foam tape, that could be zip tied to the component with the connections. This greatly reduced the movement of the wires in the connectors relative to their connected components.

4) We made sure the radio had physical separation from other CAN bus connected components. We found, as some other teams, that if the radio was placed too close to the PCM it appeared to interfere with the CAN transceiver in the device. At least that is how the issue was manifesting from observation without opening the PCM itself. When we relocated the radio, the issues went away. This was not a problem with last years radio.

5) We use 45amp power pole connectors from our TalonSRX controllers to the motors.

6) We use star rings on the main breaker and ensure the connections are tight. We fortunately did not have a faulty main breaker, but that is something that can be checked as described by others in this thread.

7) We perform a check on the robot before it leaves the pit prior to every match.

In the STL regional we didn't have any loss of communication with the robot during any of our matches.

We did have a problem before our first qualification match where the Ethernet port on our driverstation laptop was accidentally broken. But that had nothing to do with the robot. We fixed that issue with a new laptop that uses a full size port as opposed to the low profile. We also incorporate an Ethernet dongle that is fastened to the carrier.

All that said, the game has been fairly brutal this year. Definitely need to keep inspecting and repairing after every match to avoid failures due to damage.

Matt

Re: COMM lost way to many times
 

Out of curiosity, have you contacted CTR-E to see if they can help determine the root cause for the PDP and PCM? They are very easy to work with and very smart, I wouldn't be surprised if they end up helping find a root cause for both and then helping you get it fixed. Additionally how long did it take for you to replace what you assumed (and later verified) are bad components? I'd like to take this time to remind everyone that Spare Parts has multiple of these required components specifically for teams who are having issues. It's understood that you cannot open or modify them, so swap them out for an event and see if that helps.

Joe, I get you are trying to stay unbiased in this, but you are also trying to point fingers at FIRST for any dropped communication issues that appear on the field. That isn't a fair assessment. There's still a level of responsibility for the teams to take care of their own hardware and ensure that their robots are capable of playing the game at hand. Teams that take the time to make their wiring clean and ensure proper connections and relief and provided to their electronics rarely have issues. For example, one of the teams who dropped comms constantly at a previous event I attended had a rats nest of wires and relied on gravity and double sided tape to hold pieces together. Every time they lost comm we were able to determine what had come loose. Sometimes comms drop and a cause cannot be determined, that's just a fact of life when you are on the field. Even on my high school team this happened, eventually we found the issue or found a work around to prevent it (looking at you, 2009 bot who wouldn't boot properly every time). My point is, there are a lot of troubleshooting steps teams can and should take prior to entering the field to play. These include tug tests, checking for loose connectors, etc. You wouldn't believe the number of teams I've told to do that very thing (even at the drivers meeting) and they don't.

I don't have any data to back me up, but here are my general impressions of the control system year over year. Note that when a robot drops I attempt to do everything in my power (and allotted time) to find the issue on the field itself. I hate phantom issues showing up in the middle of a match. It's not inspiring to the students, it's not fun, it may provide a challenge to the teams, but they shouldn't have to worry about these types of issues during a 2.5 minute span. I would say I have about a 90% success rate and determining and replicating the issue on the field.

2011 - 2013 - I remember hating the control system during these years, I just can't remember exactly why.

2014 - Horrible comm drops due to multiple issues and with no way to easily explain why. I remember issues with the radio and crio rebooting in the middle of a match being most common. This was caused primarily by brown outs to either, or loose wiring. The brown outs occured most frequently in the last 60 seconds of a match, especially when the robot started playing defense. Pushing matches stressed the control system beyond what it could handle. Additionally I remember parts of the control system having to be swapped out randomly due to shorts and failures (digital side car, radio power converter, etc.)

2015 - Hardly any comm drops compared to previous years. This was a refreshing turn after concerns of switching to a new control system. There were multiple factors that helped including the low stress on the robots, stronger connections (when wired properly), brow-out protection being built into the roborio, and the vrm keeping the radio alive during short voltage drops that were low enough to take out the rio. There were multiple instances of robots dropping below the rio's threshold, causing a reboot, but the radio stayed up and connected. This was the most solid I've ever felt our control system has been.

2016 - I feel like this year is some-what in between 2014 and 2016. Issues from wires coming loose or pulling out are back due to the intensity of the game, but we still have many of the protections in place that helped the control system in 2015 feel solid (vrm and brownout protection). The big issue I see this year is just how long it takes to get communication back with your robot. A team can die at the start of teleop, then start moving with 20 seconds left. One of the most annoying aspects of this year's control system is starting robot code after the DS has full communication back up. It can take the 50 seconds it takes the radio to boot up to nearly 70+ seconds just for the robot to be enabled again. In other words, if you drop, the chances of coming back in time are slim.

Re: COMM lost way to many times
 

I initially had radio power problems during the build. I decided to strip the 24 AWG wire in the power twice as long as recommended (~3/4in), fold it in half, and point solder the end back the beginning of the strip (to make it more manageable during insertion). I know the VRM is rated for the 24 AWG as its minimum, but I feel that this loop provides a more reliable connection than just 24 AWG stranded. Just be careful not to force it down too much during install.

Also, if you mounted your radio with the barrel connector facing toward the floor, I suggest you rotate it so gravity is not working against you.

I continue to have major power problems using the old DLINKs and I am considering soldering directly to the PCB. So I think the problem is not specific to the new radio.

Re: COMM lost way to many times
 

I haven't contacted them as yet--too busy preparing for FLR this week!

It took maybe 20 minutes to replace the PCP, and ten to replace the PCM.

Our gear boxes were an accessibility issue that is, ultimately, our fault; the robot is packaged so tightly that we'd have to tear it apart to get the gearboxes off.

Re: COMM lost way to many times
 

Hey Alan, I've been following this thread, and I wanted to make sure I understand everything correctly. Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?

Thanks,
Joe

Re: COMM lost way to many times
 

I feel like we've had an absurd number of coms issues ever since we went to WiFi-based radios. I miss the old pre-2009 control system that would boot in 5 seconds and connect instantly. Basically the only coms issues I ever had with those systems was when something was physically damaged during a match. We should have stuck with something like that and just used WiFi for on-board video streaming, that way a connection failure doesn't shut down the whole machine for the entire match.

[/rant]

Re: COMM lost way to many times
 

What you show will work, but it's backwards from how I did it. Wiring the power into the cable connected to the radio fails to realize one of the main advantages of Power Over Ethernet. If you wire it the other way around, with the power wires going into a short extension plugged into the roboRIO, you can put the radio at any distance from the power source by using a standard patch cable between it and the powered Ethernet jack.

Re: COMM lost way to many times
 

This matches the PoE configuration we shop tested last night. Being paranoid now due to prior issues, we verified that we can provide power to the radio over both PoE and the standard barrel jack. We can unplug either individually and the radio won't lose power. So we now can have redundant sources of power to the radio from the VRM or pick and choose them individually. Overkill? Yes, but when your radio is intermittently rebooting during match play it's good to have a few extra tricks in the bag.

Re: COMM lost way to many times
 

Oh, duh! It was late last night. Haha! Thanks Alan!

Re: COMM lost way to many times
 

I don't know what to say. I can't think of a single case in my engineering career where I got to say to management, "yes our customers are having a crummy experience but really, we just have dumb customers. If we had smarter customers we wouldn't have these problems."

I am trying not to flame out here but it is hard because I know how much pressure most teams are under and I sympathize with them when they have to watch their robots sit dead for so many matches. Nobody wins in this case.

Do I blame these teams? Yes. Yes, I do. But I blame FIRST too. And if my (anecdotal*) experiences at NEF-Reading and NEF-UMassD are any guide, this seems like a pretty big deal and I have not observed an appropriately big deal response from FIRST (e.g. a tech update insisting that no robot will pass inspection without X, Y, & Z features in place).

Dr. Joe J.

*which is why I was asking if there was anyone with data that could shed light on this. I am suspicious that I am just generalizing a few bad experiences to the general population.

Re: COMM lost way to many times
 

At present I have only my observations, with no solid data in hand to back up what I "feel" is the case. I will be asking around in the next few weeks to see if anyone I know has that data. But what I observe is that communication problems are very rare this year compared with the last two years.

I do not count cases where power was lost (loose connections, faulty crimps, a main breaker being accidentally hit by a stray robot part, etc.) or where an error in a team's programming caused the robot software to crash. Those can have similar symptoms to a "comm problem", but the underlying cause is unrelated to the field-to-robot communication method.

(Power issues this year are extremely prevalent, though. Teams simply aren't checking regularly to make sure their battery and breaker connections are tight, or that wires are properly inserted into the Weidmuller connectors.)

Re: COMM lost way to many times
 

Joe -- I helped a lot of teams with comm issues at week 1 and 2 events. In general, the robots I helped had a lack of care in wiring -- gobs of electrical tape, lack of strain relief, improperly stripped wire ends, loose connectors, etc. Electrical and programming on FRC robots the last thing on the task list for many teams, and gets shorted in the rush to get the robot into the bag.

If a team with this lack of care appears at the field at 7:50 PM on practice day, asking for their connection test, there's a decent chance you'll see those same numbers with a solid RSL in a match the next day.

In the CSA role, in the heat of the competition, it's hard to convince a team to make changes if you don't have a match failure to point to. Example: one team I worked with recently had a painfully bad radio wiring job. Yet it took me three failed matches to convince them to go to spare parts and replace the 12v pigtail.

Re: COMM lost way to many times
 

It looks like there's been a lot of discussion on the radio, breaker, connectors, etc. being the issue. I think the likely culprit here isn't the PDP, but rather the VRM.

At the NC Wake County event, I helped out a team (5544 I believe) that had a failed VRM. The symptoms were:

• Both VRM lights were solid.
• The wires connecting the VRM to the PDP were solid and passed the tug test.
• The VRM and PDP were properly mounted.
• The Radio properly booted when the robot was idle.
• The VRM was located on the opposite side of the robot to the Radio.

We discovered that whenever the team drove the robot in the pits, the radio would enter a boot loop (radio power light flickering at a high frequency, not the ~2 Hz blinking you see when programming the radio). The final point shown above is important because many teams don't realize that this is an issue.

The VRM is very noisy. Locating it anywhere near your radio will cause increased packet loss and trip time. Another team at the same event reported laggy controls and communication issues. They mounted their VRM under their radio and the logs showed a direct correlation between current consumption and packet loss. The VRM is also very low cost. The testing I described would have never been done had teams not been given a reason to suspect the VRM. I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.

I highly recommend that, once the debugging procedures outlined by many others in this thread has been performed, try swapping the VRM and see if that solves the issue. In subsequent events I attended (NC Guilford, NC Wake, NC Asheville) the rate of teams having radio issues has plummeted mostly due to awareness and meticulous checking on behalf of the robot inspectors, CSAs, FTAs, and FTAAs. While I do believe that steps could have been taken to decrease the number of radio issues seen on the field, I also believe that teams & mentors should learn from these issues and test their robots under competition conditions wherever possible.

Re: COMM lost way to many times
 

That's a reasonable supposition. The VRM's "2 amp" output is actually likely to be damaged by anything higher than a continuous 1.5 amp draw. One of the Beta Test teams did mortally wound a VRM during testing by using it as labeled rather than as documented.

Re: COMM lost way to many times
 

well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr

Re: COMM lost way to many times
 

Would anyone be interested in doing a DFMEA style deep dive on this?

One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.

I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.

Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.

As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?

Re: COMM lost way to many times
 

DFMEA????

Re: COMM lost way to many times
 

Design Failure Mode and Effect Analysis. It's a Six-Sigma process that has you go through the system listing all the things that could go wrong, how bad it would be if they did, and how hard it is to detect them. After you do that, you take the ones that score the worst and develop plans to remove or mitigate the failure, then rinse and repeat.

Re: COMM lost way to many times
 

Weidmuller connector length
What is the required length?

The PDP, PCM, and VRM documents list 3/8". The FIRST video on using Weidmuller connectors says 5/8"

I believe the spec is actually 8mm but I just want to make sure since the official FIRST video says something different.

Radio PoE
Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set didn't work in my testing.

Re: COMM lost way to many times
 

Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?

Re: COMM lost way to many times
 

How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.

Re: COMM lost way to many times
 

Wow that would probably be my problem I just did a quick test using the barrel we already had connected, didn't realize the power and ground were reversed. I'll cut it tomorrow and make sure it works.

Re: COMM lost way to many times
 

The radio is far from the VRM and as far away from any motors as we can get them. The closest one is a drive CIM and that is something like 12 inches away.
After we swapped out the barrel connector we had no issues the rest of Thursday and Friday all the way to the last seeding match on Saturday.
then 3 times in like 8 matches.
Had all the orange and yellow hats looking. Nobody could find an issue.
One thought I had was this. We are using these wide polyurethane beach tires.
We notices when we were on the practice field we sold get a strong static electricity shock when we touched to bot after driving a while and we could also hear a discharge when going over the defenses. Never seen this before with a robot.
We always lost comms not after a violent move but rather when beginning to cross a defense.
Now I am wondering if we have two problems. Radio power, fixed by POE and the new barrel connector and a static discharge causing another failure mode?

Re: COMM lost way to many times
 

here is a pic


you can see the radio on the outside and away from most every thing electrical

Re: COMM lost way to many times
 

Re: COMM lost way to many times
 

hope this works

Re: COMM lost way to many times
 

The page that video is embedded on contains this note:

Re: COMM lost way to many times
 

From the VRM users guide

Strip wire back ~0.375” (3/8”)

Re: COMM lost way to many times
 

Make sure you smash them in there. Excessive force is necessary for them. Also, make sure the power connections to and from the VRM are done properly. When they're secured the right way, you should be able to pull on them with all of your weight without them moving.

Re: COMM lost way to many times
 

Ok, not on topic, but I have to ask what kind of tires are those?

And on topic, we too lost comms on our first match this past weekend. We were told that we should not have tinned the wiring going into the VRM....

Re: COMM lost way to many times
 

Bruce,
I know that this doesn't help you at this point but there doesn't seem to be any strain relief on the power plug for the radio. The barrel connector depends on the "spring force" of the outside contact to hold everything in place. I highly recommend using an adhesive backed wire tie point on the top of the radio. Loop the power wire up to the tie point and then secure with a a small wire tie. This small step keeps the connector from bouncing around and prevents pullout. Tape over the connector to the radio body also works. The antennas for this radio are on the sides of the radio, inside the vents. It doesn't help to have the stainless pneumatic tanks on that side of the radio but shouldn't be a big problem.
There is also the possibility that severe vibration has caused the power plug to pull out of the circuit board. This is easily repaired if the circuit board has not been damaged. The connector is a part of the circuit board that a good hit on the connector could cause a crack in the board. This has the same effect as broken wires. It may work most of the time, but vibration will make it lose the power.
Tinning wires is not recommended for Weidmuller terminals. They depend on the deformation of the strands to give maximum surface area for current to flow. With tinned wire, only the extreme outside surface of the wire touches the contact producing a connection equivalent to a #36 or smaller wire.

Re: COMM lost way to many times
 

We experienced the same thing at both the Arkansas and Rocket City Regionals. After speaking to other teams at Rocket City, the consensus was the power connection at the radio. It isn't as reliable as the D-Link. I found ours had backed out a little over an 1/8" and most likely lost connection briefly when we struck the wall in autonomous. Bomb Squad mentioned that they had experienced some issues prior to competition and had hot glued their connector, they didn't have any further issues with that connection.

It's extremely frustrating to say the least....Especially when Bob starts fussing! (cause usually he's right)

Re: COMM lost way to many times
 

One of the other things that hasn't been metioned is the use of ferrules with the weildmuller connectors. We used them on nearly all of our competition robot connections and have had zero issues.

Here is the white paper from Weildmuller recommending their use with stranded wires.

We import ours from China using Aliexpress but you can also get them from US electronics vendors, ebay, or Amazon.

Make sure you are using the right crimp tool and gauge ferrules. Also get 8mm long ones.

Re: COMM lost way to many times
 

This thread inspired me to write up this document. Anyone have any more suggestions to include.

https://docs.google.com/document/d/1...XrCy1gzk/edit#

Re: COMM lost way to many times
 

they are 9.4 inch beach tires. 3 psi and they litterally float over the defenses

Re: COMM lost way to many times
 

Al that was an old pic just to show the location of the radio. At the time of our latest comm losses during elims we had a new tighter barrel jack taped in place AND we were running POE. so we had two power sources.
this bot had been gone over by nearly every orange and yellow hat at both Cleveland and Milwaukee. After replacing the barrel jack and installing POE in Milwaukee we had no issues until the last match of quals then it started all over again. Had Pete Thomas doing everything he could think of during elims but he found nothing.

is it just the 706 curse? 6 time finalist in the last 4 years.

Thanks

Re: COMM lost way to many times
 

one other thing
we NEVER lost comms during extreme contact with another robot or flying over defenses. It always happened when just starting to cross the defense and even happened when crossing the low bar once.
I am beginning to think a static discharge may be part of the problem as we noticed a static discharge when crossing defenses on the practice field.

Re: COMM lost way to many times
 

Do you have the driver station logs from those matches that you can post?

There is also a system log file you can pull up using the RoboRIO webpage as well. We had a problem at Bayou were we were seeing the RoboRIO just crash for no reason (not our code but the whole RoboRIO, as in it stopped even showing up as a USB network adapter to computers once that crash happened, we never found the problem and just swapped RoboRIOs)

Re: COMM lost way to many times
 

I will see if I can have our programmers post those from finals matches.

Re: COMM lost way to many times
 

We used ferrules on almost all of our weidmuller connections and found that the crimps eventually work their way out after heavy competition. We are using the correct tool (ratcheting style) and gauge for our wires but getting consistent crimps never seemed possible. We are considering moving to mostly solidcore wire to avoid troubles in the future.

If you have gotten consistent crimps that hold up to this year's game, do you think that our crimping tool might be bad? I'm not sure where the failure is, but right now we aren't able to rely on ferrules.

Re: COMM lost way to many times
 

I believe you are on to something. Static problems can be so elusive because changes in humidity and surface contact materials are so difficult to track.I can relate one situation where static buildup definitely was the culprit and why it may relate to your problems.

We have used dozens of UsDigital MA3 absolute encoders for steering feedback over years so we know their characteristics pretty well. We have also used aluminum AM Performance wheels that contact the carpet even with tread attached. One off season we decided to replace the performance wheels with AM Plaction wheels for a demonstration robot. Within a couple of hours of run time, we had two encoder failures. The outputs would start to jitter and eventually get too bad to use. After realizing the wheels were Polycarbonate and probably isolating the chassis to allow static buildup, I attached a chain to drag the ground and never had another problem.

Many people are running pneumatic wheels this year without problems but most tires have additives that are conductive. Looking at the beach tires, they don't appear to have additives that other tires have. Is it possible that they are insulating enough to allow a charge to build on your robot? We have also had more static problems when running belts. They tend to create static like a Van De Graaff generator.

I'm not sure if a chain is the best idea this year with the defenses but something to reduce the potential between the carpet and the frame might avoid the big discharge when you touch the metal defense. If you can find a static gun some of this can be more scientifically proven.

For those that might say that the control system is immune to static, after working with plastic extrusion for 30+ years I have seen enough electronic equipment affected and destroyed from static buildup to believe that nothing is immune. Some designs are more tolerant but at some point a large enough discharge will cause damage. Also, many devices will continue to operate after being zapped but are degraded and prone to erratic performance and failure until replaced.

Re: COMM lost way to many times
 

The ESD theory is interesting. Our driver asked how he could keep from getting shocked when he was shutting off the robot at the Kansas City Regional.

Re: COMM lost way to many times
 

Were they shocked previously?

I am interested in this ESD theory. It sounds very plausible.

Re: COMM lost way to many times
 

We are currently using 20 AWG ferrules for everything (we haven't purchased a full set yet). We had to adjust the tension on our crimper to get a tight hold and we crimp the insulation in addition to the terminal. We tug test every wire and didn't have any failures at Bayou. We are using generic cheap crimps and a crimper from china.

Re: COMM lost way to many times
 

I told him to have the sophomore turn off the robot that way he wouldn't get shocked. He did not say anything about getting shocked at the Colorado Regional. This may have been due to being on the floor with insulation over the ice arena.
We had one reboot during the Colorado Regional. It was in in our last qualification match. The problem was due to a loose battery connection.

I was the one of the CSAs in Colorado. Most of the issues with radios and roboRIOs were due to loose connections or components.

One robot had too much strain relief on the Ethernet cable and when the robot frame twisted, it pulled the RJ-45 all the way to one side and disconnected.

One robot had a battery cable move and unplug the Ethernet cable on the radio when going over a defense

Re: COMM lost way to many times
 

Hhm, adjusting the tension and crimping the insulation as well could make a difference for us. Thanks for the tips, I'll look into it.

Re: COMM lost way to many times
 

I guess I need to complete my theory on how static might create radio problems. The static voltage buildup doesn't cause the problem, it is the sudden discharge of that voltage (ESD) as Greg stated that can create problems.

Years before RFID was mainstream technology I bought and installed a series of antennas to read tags tied to material conveying hoses. They were used to identify where the hoses were plugged. Almost immediately I had lockups and failures. I finally determined that static discharges were being received by the antennas and were actually damaging the receivers. They wouldn't fail immediately but damage the inputs progressively. I was able to tell the damage by measuring input resistance and predict the failures. The problem was totally eliminated by creating paths for the charges to dissipate farther away from the antennas.

We are dealing with a router with antennas here also. The higher the gain the more possibility that the input can be overdriven by large magnetic pulses. The technology has advanced drastically over recent years but theoretically the radio would be the most vulnerable device unless the discharge were to the actual control system. Since it is supposed to be isolated from the frame that is less likely.

This is all just speculation and not based on anything tested with this equipment. Consumer electronic devices are tested for these kinds of events but at some reasonable level. I have seen many fail with the levels of discharge easily produced in plastics handling. The trick is to reduce the potential to an acceptable level.

We need to watch for signs of affects caused by discharges.

Re: COMM lost way to many times
 

Ah, yes.... like the old lightning straps we used to have on our cars.... We need to find those old rubber straps. lol

What's next? Rockers? :yikes:

Re: COMM lost way to many times
 

wish I had another regional to go to to test this theory.
I can see where these beach tires may have uses in other games and I sure dont want these issues again.
We are running a belt drive on AM plastic belt sprockets. Is that what is creating the charge? And the polyurethane wheels are insulating it so it can build up?
What would the best way to prevent the discharge be?