COMM lost way to many times

[Bruceb]

We competed at Buckeye this past weekend and suffered comm issues in most of our later matches on Friday and Saturday.
We had lots of folks trying to help us figure it out from the FTA's and other field personnel to mentors from other teams.
Here is what I know.
From looking at logs and reported observations from the FTA's they see the radio rebooting. Immediately after they see the radio return they see the RIO so the belief is that the RADIO is the only think rebooting as the RIO takes longer than the radio and would not show up immediately after the radio if it was also booting.
Even so. The initial thought, before we found out the above, as that we had loos battery connections. We went over all our batteries and did find some that were marginal so we tightened ALL our battery connections regardless if they were loose or not. This did not solve the problem. The problem could not be duplicated in the pit no matter what we wiggled and banged on. We build a practice bot and had no issues with that all preseason. We had the radio from that along so we swapped it in and no joy with that either.
We pulled and re-seated all the wires in the weidmuller connectors on the PDP and VRM and tug tested. Again, no joy. By this time we were past our second match of elims and got subbed out. Not a problem, I would have done the same thing.
Before we bagged we replaced the radio power cable with the one we used on our practice bot and replaced the VRM with a brand new out of the box new one.
Have not had a chance to test since.
Does anybody have any idea what else we should try if this does not work?
Thanks

[kyle.h]

have you guys checked the mini automotive fuse. my team just unbagged our robot yesterday and it drove fine until we tried to go over any defenses. those fuses can come loose but still look like they are seated. hope this helps.

[Joe Johnson]

I know I am a flame thrower on this topic so I should probably just sit on my hands and watch this thread, but I will say this much, as a fan in the stands at UMass Dartmouth over the weekend, it was galling to watch team after team (and probably more veteran teams than rookies) sit dead on the field for close to a minute while the radio reboots (or reconnects or whatever -- their dead and that orange light isn't flashing so you can be sure that they are not coming back alive for half the match if at all).

I know, I know, I KNOW, it is almost always a power issue for the radio. I hear you, FIRST, but seriously, you need to stop blaming the teams on this matter.

When you run a remote controlled robot tournament, the first and primary contract you make with the participants that you promise them a solid data link to their robot.

I believe that FIRST is failing in this promise.

I argue that we are at the point where we need to either

find an idiot proof solution (that a team can do whatever they want and the radio link stays alive)

or

inspect in the set of behaviors that gets to a solution (e.g. every robot going on the field gets inspected for the proper wiring connections, strain reliefs, etc.).

Blaming teams was never a great plan but this year with such long reconnection times, it is just unsupportable.

Dr. Joe J.

[mwtidd]

The problem is solved with hot glue.

[Lil' Lavery]

1712 had too many simultaneous fixed to truly isolate which mattered and which didn't (we didn't have time to only change one variable at a time in the heat of competition). But I do believe that getting better strain relief on wires was a big part of us solving our roboRIO power issues, and may help you with your radio issues. Ensure your wiring has a sufficient bend radius near either end, and then get it tied down securely. Even momentary losses of power can cause reboots or brown outs. Check (and possibly replace) both your power and cat5 connectors. We also added an elastic band (hairtie) around our radio to help all the connectors stay plugged in. Further still, we cushioned our radio mount with foam to help absorb shock.

[protoserge]

What is the battery voltage showing on the logs? How old are your batteries?

[Bad advice retracted].

Try driving the robot in a stall condition by putting it against a wall and driving forward. See if your radio restarts.

[Thad House]

Originally Posted by protoserge

If you haven't already, pick your battery up by the cables. If it pulls the wires out, your crimps are bad. Don't make a habit of lifting batteries by the cables though.

Please don't do this. Even doing this once is enough to damage the terminals on the battery even if the crimps are lose. Its much better to take them off of the battery, and test the crimps off the battery.

Also teams, if you are crimping your own Andersen connectors, you can actually purchase a hydraulic crimper from Harbor Freight for not too much money. We bought it a few years ago, and love it. Haven't had a lose crimp in years, and we even cut a few apart to inspect the test crimps and they are really solid.

http://www.harborfreight.com/hydraul...ool-66150.html

[protoserge]

Originally Posted by Thad House

Please don't do this. Even doing this once is enough to damage the terminals on the battery even if the crimps are lose. Its much better to take them off of the battery, and test the crimps off the battery.

Also teams, if you are crimping your own Andersen connectors, you can actually purchase a hydraulic crimper from Harbor Freight for not too much money. We bought it a few years ago, and love it. Haven't had a lose crimp in years, and we even cut a few apart to inspect the test crimps and they are really solid.

http://www.harborfreight.com/hydraul...ool-66150.html

That is a valid point. I would recomend pull testing them a better way.

[bdaroz]

We had some issues with our radio and/or FMS at the Tech Valley Regional this weekend.

One of them was on us (tug test failed on our radio power connector), the other was indeterminate. (CSA/FTA was blaming robot power, as a whole, but we have video showing no power loss on the RoboRio.)

Regardless of what the problem is, even as a rookie team, the instrumentation provided by the RoboRIO/PDU/FMS/DS completely fails during problem events.

During the aftermath of our issues, we attempted to use the DS logs to trace voltage, power draw, comms, and log messages from the RoboRIO. The problem is, you can make "guesses" about your voltage state in the fraction of a second leading up to the event, but once it happens, you're blind.

We're already redoing our robot code library for next year to address some of these shortcomings. We'll be logging voltage and PDU stats, as well as bridge/radio and FMS connectivity via ethernet directly on the RoboRIO. In addition we'll be using a cached-logger, such that if DS communication is lost we can back-fill in our logs.

We are attempting to address the issue of the complete lack of FMS logging, and the logging dropout during any communication loss event. We're also pushing the logging closer to the source of the data. Our goal is to be able to document the voltages, RoboRIO, PDU, and Radio/FMS/DS status in an effort to diagnose issues.

[Bruceb]

Originally Posted by protoserge

What is the battery voltage showing on the logs? How old are your batteries?

[Bad advice retracted].

Try driving the robot in a stall condition by putting it against a wall and driving forward. See if your radio restarts.

we have all kinds of batteries but we lost comms some times when we were just going under the low bar. No big current draws, no big bumps.

[Conor Ryan]

This sounds like a power quality issue. Bad wires, bad crimps, bad connections, bad barrels all can contribute to this.

I would replace all of the wires from the batter to the radio, especially the black wires that go directly into the radio. Then I would reprogram your firmware on the radio in particular. Inspect your fuses, many people don't notice when they are bad. Drive the robot around at home, drive over bumps see if you can replicate these issues. Make sure all of your wires are anchored down appropriately and nothing can shake loose. Make a good effort to replicate the driving conditions on the field, move around fast and crash frequently.

With how difficult the field is on robots this year, i'm not surprised that we are seeing a rise in power quality related issues on the communications.

[Bruceb]

Originally Posted by Conor Ryan

This sounds like a power quality issue. Bad wires, bad crimps, bad connections, bad barrels all can contribute to this.

I would replace all of the wires from the batter to the radio, especially the black wires that go directly into the radio. Then I would reprogram your firmware on the radio in particular. Inspect your fuses, many people don't notice when they are bad. Drive the robot around at home, drive over bumps see if you can replicate these issues. Make sure all of your wires are anchored down appropriately and nothing can shake loose. Make a good effort to replicate the driving conditions on the field, move around fast and crash frequently.

With how difficult the field is on robots this year, i'm not surprised that we are seeing a rise in power quality related issues on the communications.

ya but the robot is in the bag

[Bruceb]

Originally Posted by mwtidd

The problem is solved with hot glue.

hot glue where?

[mwtidd]

Originally Posted by Bruceb

hot glue where?

We solder and heat shrink where we can, and hot glue everywhere else. So the connections to the Voltage Regulator are all hot glued. We were seeing all sorts of radio drops out just as you've described on both our comp and practice bots. Since hot gluing the radio wires into the voltage regulator we haven't lost comms. (~19 matches).

We use hot glue on nearly every connection, including our encoder cables, limit switches, arduino plugs, etc.

If you want something less permanent electrical tape tends to work well enough, but I haven't seen anything better than hot glue this year.

[Hadi379]

Originally Posted by Bruceb

hot glue where?

Hot glue at the connection between the power plug and the radio.

I was the mentor from 379 that was trying to help. At the time we probably couldn't wait for a hot glue gun to heat up We reset all wires at the VRM. Checked other connections, inspected contacts, and zip tied the battery connection. Not knowing the robot, it was hard to inspect.

I do feel that it's something dealing with an electrical connection. Try replacing the Anderson battery/PDB connection, wire and all, from the robot. I wish we could have figured it out, you guys have a great design, good luck this season.