Re: COMM lost way to many times
 

I don't know what to say. I can't think of a single case in my engineering career where I got to say to management, "yes our customers are having a crummy experience but really, we just have dumb customers. If we had smarter customers we wouldn't have these problems."

I am trying not to flame out here but it is hard because I know how much pressure most teams are under and I sympathize with them when they have to watch their robots sit dead for so many matches. Nobody wins in this case.

Do I blame these teams? Yes. Yes, I do. But I blame FIRST too. And if my (anecdotal*) experiences at NEF-Reading and NEF-UMassD are any guide, this seems like a pretty big deal and I have not observed an appropriately big deal response from FIRST (e.g. a tech update insisting that no robot will pass inspection without X, Y, & Z features in place).

Dr. Joe J.

*which is why I was asking if there was anyone with data that could shed light on this. I am suspicious that I am just generalizing a few bad experiences to the general population.

Re: COMM lost way to many times
 

At present I have only my observations, with no solid data in hand to back up what I "feel" is the case. I will be asking around in the next few weeks to see if anyone I know has that data. But what I observe is that communication problems are very rare this year compared with the last two years.

I do not count cases where power was lost (loose connections, faulty crimps, a main breaker being accidentally hit by a stray robot part, etc.) or where an error in a team's programming caused the robot software to crash. Those can have similar symptoms to a "comm problem", but the underlying cause is unrelated to the field-to-robot communication method.

(Power issues this year are extremely prevalent, though. Teams simply aren't checking regularly to make sure their battery and breaker connections are tight, or that wires are properly inserted into the Weidmuller connectors.)

Re: COMM lost way to many times
 

Joe -- I helped a lot of teams with comm issues at week 1 and 2 events. In general, the robots I helped had a lack of care in wiring -- gobs of electrical tape, lack of strain relief, improperly stripped wire ends, loose connectors, etc. Electrical and programming on FRC robots the last thing on the task list for many teams, and gets shorted in the rush to get the robot into the bag.

If a team with this lack of care appears at the field at 7:50 PM on practice day, asking for their connection test, there's a decent chance you'll see those same numbers with a solid RSL in a match the next day.

In the CSA role, in the heat of the competition, it's hard to convince a team to make changes if you don't have a match failure to point to. Example: one team I worked with recently had a painfully bad radio wiring job. Yet it took me three failed matches to convince them to go to spare parts and replace the 12v pigtail.

Re: COMM lost way to many times
 

It looks like there's been a lot of discussion on the radio, breaker, connectors, etc. being the issue. I think the likely culprit here isn't the PDP, but rather the VRM.

At the NC Wake County event, I helped out a team (5544 I believe) that had a failed VRM. The symptoms were:

• Both VRM lights were solid.
• The wires connecting the VRM to the PDP were solid and passed the tug test.
• The VRM and PDP were properly mounted.
• The Radio properly booted when the robot was idle.
• The VRM was located on the opposite side of the robot to the Radio.

We discovered that whenever the team drove the robot in the pits, the radio would enter a boot loop (radio power light flickering at a high frequency, not the ~2 Hz blinking you see when programming the radio). The final point shown above is important because many teams don't realize that this is an issue.

The VRM is very noisy. Locating it anywhere near your radio will cause increased packet loss and trip time. Another team at the same event reported laggy controls and communication issues. They mounted their VRM under their radio and the logs showed a direct correlation between current consumption and packet loss. The VRM is also very low cost. The testing I described would have never been done had teams not been given a reason to suspect the VRM. I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.

I highly recommend that, once the debugging procedures outlined by many others in this thread has been performed, try swapping the VRM and see if that solves the issue. In subsequent events I attended (NC Guilford, NC Wake, NC Asheville) the rate of teams having radio issues has plummeted mostly due to awareness and meticulous checking on behalf of the robot inspectors, CSAs, FTAs, and FTAAs. While I do believe that steps could have been taken to decrease the number of radio issues seen on the field, I also believe that teams & mentors should learn from these issues and test their robots under competition conditions wherever possible.

Re: COMM lost way to many times
 

That's a reasonable supposition. The VRM's "2 amp" output is actually likely to be damaged by anything higher than a continuous 1.5 amp draw. One of the Beta Test teams did mortally wound a VRM during testing by using it as labeled rather than as documented.

Re: COMM lost way to many times
 

well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr

Re: COMM lost way to many times
 

Would anyone be interested in doing a DFMEA style deep dive on this?

One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.

I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.

Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.

As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?

Re: COMM lost way to many times
 

DFMEA????

Re: COMM lost way to many times
 

Design Failure Mode and Effect Analysis. It's a Six-Sigma process that has you go through the system listing all the things that could go wrong, how bad it would be if they did, and how hard it is to detect them. After you do that, you take the ones that score the worst and develop plans to remove or mitigate the failure, then rinse and repeat.

Re: COMM lost way to many times
 

Weidmuller connector length
What is the required length?

The PDP, PCM, and VRM documents list 3/8". The FIRST video on using Weidmuller connectors says 5/8"

I believe the spec is actually 8mm but I just want to make sure since the official FIRST video says something different.

Radio PoE
Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set didn't work in my testing.

Re: COMM lost way to many times
 

Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?

Re: COMM lost way to many times
 

How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.

Re: COMM lost way to many times
 

Wow that would probably be my problem I just did a quick test using the barrel we already had connected, didn't realize the power and ground were reversed. I'll cut it tomorrow and make sure it works.

Re: COMM lost way to many times
 

The radio is far from the VRM and as far away from any motors as we can get them. The closest one is a drive CIM and that is something like 12 inches away.
After we swapped out the barrel connector we had no issues the rest of Thursday and Friday all the way to the last seeding match on Saturday.
then 3 times in like 8 matches.
Had all the orange and yellow hats looking. Nobody could find an issue.
One thought I had was this. We are using these wide polyurethane beach tires.
We notices when we were on the practice field we sold get a strong static electricity shock when we touched to bot after driving a while and we could also hear a discharge when going over the defenses. Never seen this before with a robot.
We always lost comms not after a violent move but rather when beginning to cross a defense.
Now I am wondering if we have two problems. Radio power, fixed by POE and the new barrel connector and a static discharge causing another failure mode?

Re: COMM lost way to many times
 

here is a pic


you can see the radio on the outside and away from most every thing electrical