COMM lost way to many times

[Joe Johnson]

Originally Posted by Schnabel

<snip>

Joe, I get you are trying to stay unbiased in this, but you are also trying to point fingers at FIRST for any dropped communication issues that appear on the field. That isn't a fair assessment. There's still a level of responsibility for the teams to take care of their own hardware and ensure that their robots are capable of playing the game at hand. Teams that take the time to make their wiring clean and ensure proper connections and relief and provided to their electronics rarely have issues. <snip>

I don't know what to say. I can't think of a single case in my engineering career where I got to say to management, "yes our customers are having a crummy experience but really, we just have dumb customers. If we had smarter customers we wouldn't have these problems."

I am trying not to flame out here but it is hard because I know how much pressure most teams are under and I sympathize with them when they have to watch their robots sit dead for so many matches. Nobody wins in this case.

Do I blame these teams? Yes. Yes, I do. But I blame FIRST too. And if my (anecdotal*) experiences at NEF-Reading and NEF-UMassD are any guide, this seems like a pretty big deal and I have not observed an appropriately big deal response from FIRST (e.g. a tech update insisting that no robot will pass inspection without X, Y, & Z features in place).

Dr. Joe J.

*which is why I was asking if there was anyone with data that could shed light on this. I am suspicious that I am just generalizing a few bad experiences to the general population.

[Alan Anderson]

Originally Posted by Joe Johnson

Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?

I get the feeling that the problems are worse this year but I freely admit that I am a biased observer.

At present I have only my observations, with no solid data in hand to back up what I "feel" is the case. I will be asking around in the next few weeks to see if anyone I know has that data. But what I observe is that communication problems are very rare this year compared with the last two years.

I do not count cases where power was lost (loose connections, faulty crimps, a main breaker being accidentally hit by a stray robot part, etc.) or where an error in a team's programming caused the robot software to crash. Those can have similar symptoms to a "comm problem", but the underlying cause is unrelated to the field-to-robot communication method.

(Power issues this year are extremely prevalent, though. Teams simply aren't checking regularly to make sure their battery and breaker connections are tight, or that wires are properly inserted into the Weidmuller connectors.)

[MrRoboSteve]

Joe -- I helped a lot of teams with comm issues at week 1 and 2 events. In general, the robots I helped had a lack of care in wiring -- gobs of electrical tape, lack of strain relief, improperly stripped wire ends, loose connectors, etc. Electrical and programming on FRC robots the last thing on the task list for many teams, and gets shorted in the rush to get the robot into the bag.

If a team with this lack of care appears at the field at 7:50 PM on practice day, asking for their connection test, there's a decent chance you'll see those same numbers with a solid RSL in a match the next day.

In the CSA role, in the heat of the competition, it's hard to convince a team to make changes if you don't have a match failure to point to. Example: one team I worked with recently had a painfully bad radio wiring job. Yet it took me three failed matches to convince them to go to spare parts and replace the 12v pigtail.

[juchong]

It looks like there's been a lot of discussion on the radio, breaker, connectors, etc. being the issue. I think the likely culprit here isn't the PDP, but rather the VRM.

At the NC Wake County event, I helped out a team (5544 I believe) that had a failed VRM. The symptoms were:

• Both VRM lights were solid.
• The wires connecting the VRM to the PDP were solid and passed the tug test.
• The VRM and PDP were properly mounted.
• The Radio properly booted when the robot was idle.
• The VRM was located on the opposite side of the robot to the Radio.

We discovered that whenever the team drove the robot in the pits, the radio would enter a boot loop (radio power light flickering at a high frequency, not the ~2 Hz blinking you see when programming the radio). The final point shown above is important because many teams don't realize that this is an issue.

The VRM is very noisy. Locating it anywhere near your radio will cause increased packet loss and trip time. Another team at the same event reported laggy controls and communication issues. They mounted their VRM under their radio and the logs showed a direct correlation between current consumption and packet loss. The VRM is also very low cost. The testing I described would have never been done had teams not been given a reason to suspect the VRM. I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.

I highly recommend that, once the debugging procedures outlined by many others in this thread has been performed, try swapping the VRM and see if that solves the issue. In subsequent events I attended (NC Guilford, NC Wake, NC Asheville) the rate of teams having radio issues has plummeted mostly due to awareness and meticulous checking on behalf of the robot inspectors, CSAs, FTAs, and FTAAs. While I do believe that steps could have been taken to decrease the number of radio issues seen on the field, I also believe that teams & mentors should learn from these issues and test their robots under competition conditions wherever possible.

[Alan Anderson]

Originally Posted by juchong

I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.

That's a reasonable supposition. The VRM's "2 amp" output is actually likely to be damaged by anything higher than a continuous 1.5 amp draw. One of the Beta Test teams did mortally wound a VRM during testing by using it as labeled rather than as documented.

[Bruceb]

well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr

[IKE]

Would anyone be interested in doing a DFMEA style deep dive on this?

One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.

I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.

Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.

As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?

[Bruceb]

Originally Posted by IKE

Would anyone be interested in doing a DFMEA style deep dive on this?

One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.

I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.

Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.

As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?

DFMEA????

[vhcook]

Originally Posted by Bruceb

DFMEA????

Design Failure Mode and Effect Analysis. It's a Six-Sigma process that has you go through the system listing all the things that could go wrong, how bad it would be if they did, and how hard it is to detect them. After you do that, you take the ones that score the worst and develop plans to remove or mitigate the failure, then rinse and repeat.

[AllenGregoryIV]

Weidmuller connector length

Originally Posted by Al Skierkiewicz

Strip the wires that are inserted into the various Weidmuller connectors to the required length. Often teams make these too short and the result is the wire is not properly retained.

What is the required length?

The PDP, PCM, and VRM documents list 3/8". The FIRST video on using Weidmuller connectors says 5/8"

I believe the spec is actually 8mm but I just want to make sure since the official FIRST video says something different.

Radio PoE
Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set didn't work in my testing.

[juchong]

Originally Posted by Bruceb

well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr

Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?

[Alan Anderson]

Originally Posted by AllenGregoryIV

Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set didn't work in my testing.

How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.

[AllenGregoryIV]

Originally Posted by Alan Anderson

How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.

Wow that would probably be my problem I just did a quick test using the barrel we already had connected, didn't realize the power and ground were reversed. I'll cut it tomorrow and make sure it works.

[Bruceb]

Originally Posted by juchong

Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?

The radio is far from the VRM and as far away from any motors as we can get them. The closest one is a drive CIM and that is something like 12 inches away.
After we swapped out the barrel connector we had no issues the rest of Thursday and Friday all the way to the last seeding match on Saturday.
then 3 times in like 8 matches.
Had all the orange and yellow hats looking. Nobody could find an issue.
One thought I had was this. We are using these wide polyurethane beach tires.
We notices when we were on the practice field we sold get a strong static electricity shock when we touched to bot after driving a while and we could also hear a discharge when going over the defenses. Never seen this before with a robot.
We always lost comms not after a violent move but rather when beginning to cross a defense.
Now I am wondering if we have two problems. Radio power, fixed by POE and the new barrel connector and a static discharge causing another failure mode?

[Bruceb]

here is a pic


you can see the radio on the outside and away from most every thing electrical