Storm Robotics Team, 2729 had to do a fair amount of repairs for faults during the 2013 competition season so far, all of which carry some good and simple lesson's learned I think so I'll share briefly. The issues are not nightmares like some of the others in the thread by any means. [This is turning out to be way longer than I intended, but hopefully serves a purpose to someone out there
]
First, nearly all the issues are related to the ruggedness (or lack of) on our electrical system. In the 5 years of Storm we haven't had issues like this. Luck be it, or possibly because we were defended on much more this year, we had several cRIO reboots, router reboots, etc.
At MAR Hatboro Horsham, at the end of the competition (last match) our cRIO 'died'. 24V was verified but no LEDs came on when the bot was powered on. We had to pull the cRIO, bag it and give up. We later found that a single copper strand of wire, about 1/4 inch long had fallen into one of the cRIO module DB9 slots and lodged itself between a few of the pins, shorting them and the overall cRIO boot (would love to know the pinout of that DB9 connection to see how that is even possible). This was
under the black DB9 cap which was installed. Our best guess is that the strand was sitting in the cap when it was installed. The cap had been vacuumed out once and then found in the shop vac, so possibly the strand stuck in the cap from static charge, was installed with the strand in it, and then the strand fell out later and lodged in the pins. Or it was sabotage.
But for GP's sake we are going with bad luck
[I wonder why NI didn't put a female DB9 on the cRIO side anyway. When the pins break the cRIO is toast, which has happened on our earliest cRIO]
At the MAR Lenape regional we had a cRIO reboot early on which was traced in the pit later (after another luckily good match) to the 24V+ wire coming lose in the connector on the cRIO side. We tightened it best we could, hot glued it and continued on. Unfortunately later that day we had faults that looked similar, in another qual match and in the Finals Match 1. The Field folks said it was a router reboot.
After the competition we studied the DS logs and determined it to be a router reboot in both cases. Wiring seemed fine from the PDB through the regulator and to the bridge, but we noted to make it more rugged at MAR champs. After further log analysis we saw some deep voltage dips on the matches where the router rebooted. The best guess we came up with was bad connections on the batteries or bad batteries. The Good news - we keep battery logs during the competition to help ensure we use charged batteries and we can identify bad batteries. The Bad news - the log wasn't kept
after the competition, it never made it back to the school.
So we can't prove which batteries caused the issue, but we checked and had 4 batteries with loose wires at the terminals. We replaced with better suited hardware and had no router reboots or cRIO reboots at MAR champs.
[
Note - we plan on posting the full log analysis on CD with screen shots when we have time to clean it up and will use them as a workshop later this year with electrical and software.]
-
The moral to this one is - keep (long term) your battery logs to identify problem batteries, check your wires often doing a visual spot check between matches, and whenever you think you have made your wiring 'strong' enough during build season, do some more! Inevitably something will fail when 120 lb robots collide, but you can prevent a lot of it. We learned that the hard way this year.
At the MAR championship we seem to have fixed our HH and Lenape issues, but instead saw some new ones. In QF Match 2 we took a not too memorable hit from an opponent, 224 in this case. You can see Electra (Storm's robot name for 2013) roll away in a pitiful retreat, dead for the match. In the pit we found that the power pole between the PDB and the regulator unplugged. Surprising though as the hit was quite gentle compared to others survived this year.
-
The moral to this one - even though power poles are strong - zip tie them or use the locking pins! A simple zip tie would have absolutely prevented that failure.
We had another failure at MAR champs, with only about 10 minutes to find and fix. After a match Electra had a bad "let the magic smoke out" smell. We ran through tests on every system and all were functioning without obvious issue. We continued to persistently search for a sign, and finally caught it when we saw a shiny corner (from melted plastic) on a Victor 884 that controls the drivetrain. We removed it and opened it up, and it turns out that the Victor FETs had self destructed horribly, but it was still functioning at a unit test in the pit. Interestingly the fan 12V wire looked like it had shorted to an FET. Not sure what happened first, the FET failures or the fan short (or maybe the short caused the failure).
-
The moral to this one - Don't ignore your nose, I doubt a smell that bad can
ever be a good sign. Study your robot until you find the source, nobody knows it better than you. Never be complacent or the failure will just pop up later.