Intermittent Loss of Communications During Play

[Mike Betts]

Symptom:
Robot loses communication momentarily after impact with another robot or with field element.

Disclaimer:
As with any real life, non-trivial issue, this is likely a multi-problem problem and the solution described may not mitigate all instances.

Discussion:
I was a robot inspector at the Florida Regional last weekend. There were numerous complaints of momentary loss of communications but no single clearing house to vet all problems… I helped quite a few teams and came to the conclusion that no robot which used the power supply cable which came with the DAP-1522 radio had this issue. In fact, while working with one team, we succeeded in getting the DAP to be completely unpowered even though it appeared securely plugged in. We were able to restore power to the DAP by touching the power connector ever so slightly (virtually a feather’s touch).

All of the teams that I assisted which were experiencing problems were using the power connector from last year’s robot or a connector that “seemed like it fit” from Radio Shack.

It is my (largely subjective) conclusion/theory/assumption that the barrel accepting the male pin in the connector is a few mils smaller on the DAP’s BOM connector than on the connectors giving is trouble. Also, the momentary loss of power to the DAP is just enough to corrupt its driver and/or firmware circuitry but not enough to force its CPU into a reboot condition.

Solution:

• Find the AC power supply cord that came with the DAP-1522 and cut it in half.
• Note that the wire has two conductors. The positive conductor is identified by a white stripe printed on the black insulation.
• Install .250 spade disconnects on both ends so that you can revert to using the cable as an AC source in the future.
• Now store the half connected to the AC power supply away for future use and wire the other half into your robot. Note that the white stripe conductor gets connected to the yellow lead of the 360-CPR.

Identification:
The power connector which came with the DAP is a molded plastic, right angle connector whereas last year’s connector was an axial leaded barrel connector.

Epilogue:
I strongly suggest that you at least pack the DAP-1522 power supply cable with your spares before departing to your competition. In that way it will be available if you encounter this problem.

Regards,

Mike

[pfreivald]

Thanks for the heads up! Unfortunately, I can be the exception to your observation. We suffered this problem, too (and once when we were NOT impacted by anything at all), and are using the DAP cable (taped in, no less!)

[Ivan Helmrich]

Patrick, you are not the only exception. We had this with both practice robot and our competition robot. We did use the D-link supplied power cord, it was hot glued in competition. The only things common between the two robots was the voltage converter and the power cord.

At least this suggests a place for us to look for the problem. We replaced the D-link so we have an extra cable.

Thanks for the info.

Ivan

[pfreivald]

Interesting. If you find out anything of use, please let us know. We'd hate to have this happen again.

[Greg McKaskle]

As I mentioned in another thread. I was volunteering in NJ in week one and saw issues with power on quite a few robots. There wasn't one root cause, but several that are pretty easy to differentiate between. The causes were very similar to previous years with the new control system.

One diagnostic I use is to count how long the robot stays disabled and observe what the BFL looks like.

If the robot halts for less than ten seconds, it is a watchdog or motor controller over-current situation. This isn't long enough for a reboot of any other component.

If the robot halts for between ten and forty seconds, I look at the cRIO reboot. It could be due to code or due to cRIO power. SW, especially Java and C++ can cause a reboot with a null reference, bad pointer, etc. The cRIO supports redundant power supplies. Not terribly useful in FIRST, but it means two positive and two negative connections. Despite this, the most common situation is to put the black and red connections on adjacent pins. There is often copper showing and it just takes a wiggle of the cables for the copper to touch and the cRIO to reboot. As with all of these, also check the other end of the cable.

If the robot halts for more than forty seconds, but not for the entire match, I look at the radio power. I saw one team with a loose barrel connector, I think Radio Shack, and probably five robots that were wired to the unboosted PD voltage rather than the boosted radio supply. Adding in the voltage regulator is certainly a good thing, but now there are additional connections that could be loose, or shorted.

If the robot halts for longer than a minute or for the entire match, I look at the radio switch and the SW. FTAs definitely fixed a number of cases where radio switch had been jostled into AP or auto. In other cases the issue was that the SW never switched from Auto to TeleOp. The BFL pattern indicate the mode of the packets being processed, not the SW that is running, so that isn't really helpful, but this issue is usually duplicated by a practice match in the pits or practice field.

If you happen to be watching the robot when the halt happens, sometimes the BFL will give hints. It will often dim with low battery level, and is helpful in spotting cRIO reboots.

Greg McKaskle

[Mike Betts]

Originally Posted by Greg McKaskle

...probably five robots that were wired to the unboosted PD voltage rather than the boosted radio supply...

Greg,

This is a robot inspection checklist item and should have only been found by you on Thursday before most robots had passed inspection.

Mike

[Mark McLeod]

Originally Posted by Mike Betts

This is a robot inspection checklist item and should have only been found by you on Thursday before most robots had passed inspection.

A few got missed that day...

[pfreivald]

Originally Posted by Mark McLeod

A few got missed that day...

Which is inexcusable...

[Mike Betts]

Originally Posted by pfreivald

Which is inexcusable...

Pat,

Excuse me but it is very excusable...

FIRST makes it hard when they have us measuring the stroke width of team numbers instead of concentrating on wiring issues...

The events make it harder when inspectors are not allowed in the playing field area to gather info on problems teams are having...

The teams make it harder still when they wait until the last moment to even start inspection on Thursday...

All of those things are inexcusable...

Bottom line: I would suggest that you walk a mile in our shoes and volunteer to be an inspector at your next event...

Regards,

Mike

[Fletch1373]

I agree Mike. I've worked with you for the last 2 years and have seen first hand what inspectors go through. This year, I was FTAA at WPI so I was able to troubleshoot issues from the field side, giving me a fairly unique perspective on the issue. Luckily for us, we had an inspector or 2 on the field that we could dispatch to teams needing assistance with rule questions or just in general. I must say, overall, the WPI regional went very smoothly from all sides(barring a few minor issues early Thursday), and I'm very grateful to all the inspectors there that helped get the teams legal and in working order to make my job easier.

In a few weeks, I'll be inspecting at the Northeast Utilities Connecticut Regional(or whatever it's called these days) and I hope that event is able to run as smoothly, if not more so, than WPI did.

[Brian C]

Originally Posted by pfreivald

Which is inexcusable...

That's easy to say. However I would ask that you take a step back and look at the overall job that inspectors do at an event.

All of us would LOVE to begin inspecting teams robots at 8:30AM on Thursday. The reality is that many (not all) teams at an event view inspection as a second thought.

As I'm sure you've noticed at competitions you've attended the inspection queue really starts to get crazy at around 2:00PM on Thursday and by that time we inspectors have already created a hit list of teams that need extra attention.

These issues are only magnified at Week 1 Regionals.

Volunteering at an event will definitely open your eyes to how things really work.

I guarantee that you will bring back some improvements to how your team operates after your event volunteeer experience.

[Mike Betts]

Pat,

I would like to apologize for my last post. I have left it unedited as I feel the words have merit. However, I feel that my tone was a bit strong.

All any of us are trying to do is to help each other out. That is why I started this thread and it also appears to be why you posted here to begin with.

I have noted with some dismay how several other threads have devolved into "righteous indignation" against referees, et al, and I got a bit hot under the collar when it appeared to be starting here.

It is the aim of all of the volunteers at these events to do their job as well as they can and to help provide a meaningful experience for the students attending.

Once again, sorry if I barked too loud...

Regards,

Mike

[pfreivald]

No worries -- I have an extraordinarily thick skin and am nearly impossible to offend.

To clarify, it is inexcusable on two fronts:

#1. Teams get the checklist ahead of time, and there is no reason why every team should not be passing inspection out of the box (or bag) on their first try right when they get to an event. (1551 pretty much always does).

2. It's a checklist. If it isn't right, it shouldn't be checked, and the robot shouldn't pass.

Thus, the inexcusable-ness is most certainly not all on the shoulders of the inspectors, though they share some of the culpability.

---------------

On volunteering: I'd love to, but I can't justify taking even more time out of my AP physics class. I lose five days for FIRST as it is, and a full week without their teacher is a heavy enough burden for my students to bear!

[Mark McLeod]

There are a lot of layers of people to catch issues.
It never depends on a single person, nor has one person the whole responsibility.
Expect problems to be caught and identified at all levels.

1. Rules/Q&A/Team Updates/Bill's Blog/FIRST emails
2. Teams
3. Inspectors
4. FTA/FTAA/CSA/Field Crew
5. Friendly neighboring teams

They all play a part and cross-support each other.

[Al Skierkiewicz]

Originally Posted by Mike Betts

The events make it harder when inspectors are not allowed in the playing field area to gather info on problems teams are having...

Mike,
All events should allow at least the LRI and on ocassion any RI to observe from behind the table. The FTA cabinet is a great area from which to be close to robot displays while watching for issues during match play. During finals, I recommend two inspection people watch for issues so that teams do not get penalized for things like loose robot parts, bumpers out of the zone and frame perimeter issues. The lead needs to be present for any questions from the head ref or FTA.