Let's Collaborate: FRC Electrical/Systems Troubleshooting Manual

Technical Technical Discussion
1

I’ve had this idea for a couple years now, but due to a whirlwind of a season last year haven’t had the chance to pursue it.

Background:
Each year it seems like an electrical/networking snafu happens to my team. We’ll either lose connectivity, a PWM cable will come loose, the a sensor will trigger prematurely, etc, etc, etc. Students, mentors, FTAs, and who knows who else will take a look at the robot and say, “Do this, this, this, aaaannd… that.” After about 10 people come by to help out, the problem seemingly fixes itself. The team is happy. Then someone (usually a freshman) asks, “So what did we fix?” to which the entire team scratches their heads and replies, “aldkjf4Q#@(%asfj[imagine many voices speaking at once here]@#($ASJfdasf4#$1”.

I’m sure many others out there know what I’m talking about.

And each year this happens, I say to myself, “I really need to sit down and write an in-depth troubleshooting manual where we could have gone step-by-step through an order of operations, so we could have known exactly what was the problem.”

Proposal:
If a magical manual that could solve this issue exists, please post it here and the mods can close this thread. If not…
I’ve decided to crowd source this thing. Some of the smartest and most experienced FIRSTers are on CD, so this is a great opportunity to create a quality resource everyone can use.

Maybe we divide the manual up into sections:

  • Robot won’t boot
  • Loosing comms.
  • Watchdog not fed
  • Robot drives, but not how I want it to!
  • and more?

Next steps
If you’re interested in working on this, or just have any other thoughts about this, reply to this thread. Don’t just start posting informal lists of “this is what team so-and-so does”, because who ever is going to work on this needs a better way to collaborate than CD - maybe I could set up a wiki on the COMETS’ website or use some free web service. Suggestions are most welcomed.

2

I like the idea. An issue I would suggest is a motor controller not getting signal.

3

Ooooh!! That’s a good one.

4

Generally it can be solved easily though process of elimination and a little luck to make it go faster. You need to always determine if the problem is mechanical, electrical or programming.

I like to blame the programmer first but you must start with mechanical.

  1. You can use another system to check that the mechanics of your system is working. Power up motors to see if they are working.
  2. Check from one end to the other that all items move as they are suppose to.
  3. Check and recheck that all sensors are working constantly. After three uses it may change or the speed it is actuated may not be detected by the computer randomly.
  4. Check and follow every single wire and make sure all polarity is correct.
    This is why it is important to label and organize your wiring. Keep the 12 volt separate from the low volt. It will make it faster to trace.
    5. If all the mechanics and wiring check out than start investigating the program…

An effective manual would be difficult to make but could be helpful. There are so many variables to why something is not working.

5

You might find it helpful to browse other similar threads, such as this one: http://www.chiefdelphi.com/forums/showthread.php?t=83634

6

Thanks for the link. Definitely helpful, but the vision I have for this thing is that it be a comprehensive guide for communication and basic electrical issues.

I’m thinking I’ll start a wiki. Thoughts?

7

As part of a team plagued with such issues and as such I really like the idea of a comprehensive electronics/systems troubleshooting guide.
However I really don’t think I could help much because I am rather weak (I lnow how to set up the stuff but really don’t know much else) in the electronics portion in the robo; part of the reason that we have so many problems.
This sounds like a great idea.:smiley:

8

I agree, a wiki would be an excellent tool. I would definitely be willing to contribute anything I find/learn.

9

For a ‘user’ manual, the key point made here is: check your connections, or swap to better connections ahead of time.

Case 1
Most of our electronics problems were solved in 2009 by these guys (we don’t buy them there; that’s just the first place I found a picture). Combined with the appropriate crimp tool, our electronics board is not only modular, it is incredibly easy to fix a connection and we haven’t had a mysterious random shutoff since 2008. We mainly went to the mini Anderson PowerPoles when we realized that we kept loosening the wires within the default KOP crimps while changing a motor or victor. The other issue we had, even with good electronics toolbox organization, was that we would recrimp a wire with the wrong-sized connector (which caused this little guy in Atlanta 2007 – luckily our post-match checkups caught this before it caught fire). So for a random subsystem shutdown, check the leads. It seems like an obvious solution, however finding the problem connection is sometimes VERY challenging.

Case 2
The other thing to check is your BATTERY connection. The battery wire leads should be SOLID, NOT MOVING, and COVERED. Picking up a battery by its wires will eventually deteriorate the connection and cause the described random shut off. The “fix” your team did may have been to simply replace the battery. As dumb as that may sound, we’ve had many matches (ranging from FTC to VEX to FRC) where a bad battery connection has been the culprit of random system-wide shutdowns; the cause could be traced back to battery mishandling.

If the drive train stops responding, the problem may be a combination of a bad battery connection combined with improperly-designed drive train or control code. For example, the drive train may shut off during long sweeping turns if the code only tells one side of the drive train to go forward, yet the motors on that side must pull an excessive amount of current to generate the torque necessary to make the robot go forward and turn (depending on wheel traction and wheel base). If the battery has a bad connection, it may only allow 30-50 amps of power; this is enough allow the drive train to go full-forward and other robotic systems to function, yet in a sweeping turn the drive train may pull more current than that connection allows is physically capable of handling. Thus the drive train stops responding. The solution here is twofold, though the first is a stop-gap measure while the second is a must-fix issue: disable long-sweeping turning in the code, and inspect/fix the battery connection.

PWM Cables
Use clear nail polish. It’s non-conductive, comes of cleanly, and is easy to cleanly apply. Stick the PWM cable into the Victor/Jaguar connection, verify that it works properly, then put a small dot of clear nail polish where the black plastic PWM connector touches the plastic on the Victor/Jaguar. The FRC electronics designers finally got something correct with adding lock tabs to the digital/analog sidecars, yet IFI/Luminary Micro still miss this (IMO, critical for anything industrial) function.

10

Jesse,
I have to agree on the rest of your post but I am firmly against the use of permanent or semi-permanent attachments for critical connections. Adhesives (hot glue, nail polish, super glue) should be avoided when you may need to change a defective part quickly during a finals match. I recommend that teams find a way to attach the PWM cable near the device with a tywrap. The Jaguar has some clips molded into the body for this purpose. If the cable is holding down the connector, i.e. no slack, then the connector cannot pull out. A simple cut with a small side cutter is all it takes to remove the cable. We use punched aluminum sheet to mount electrical items and the holes are perfect for this purpose.

For my two cents, learn early that a voltmeter does not tell you the condition of the battery unless it is under load. Dead or near dead batteries will cover about 50% of all electrical problems but will read 12 volts on a voltmeter with no load. If many things do not work on the robot that have before, change the battery. Don’t believe that a student gave you a charged battery try another one. The Crio senses battery voltage through the jumper on the analog card for module one only. If the battery voltage falls to 5.5 volts, the Crio disables all outputs. When the battery voltage rises, the Crio will start operating again until the battery falls to 5.5 volts. Watch the drivers display to check battery voltage. The power supplies in the PD will fail at about 4.5 volts. When the 24 volt supply fails the Crio reboots, the 12 volt supply failure causes a reboot of the gaming adapter and a five volt failure causes the camera to reboot. They do not all fail at the same time, so as luck would have it, you may not have communication re-established following a power brownout.
Learn to listen to the robot, it is talking to you. LEDs will tell you a lot about the health of all the systems, sounds will tell you if something is wrong mechanically, and for what you can’t see or hear, smell will tell you if something is overheating or has burned. You will never forget the smell of a burned FP motor or a bad speed controller. While we are on the subject, the LEDs on the sidecar can glow dimly with current leaking through the data connectors. Make sure that they are bright and supplied with a good external power connection. Use your voltmeter at the power in connector to insure that the sidecar has power.
For your benefit and those of your inspection team, don’t hide electrical parts where they can’t be seen. If we can’t see the LEDs on everything, we can’t help you play.

11

I wanted to add to the battery comments. Many many of the times things were not quite working right ended up being due to the battery not being charged or good.

As Al mentioned a voltmeter does not give a state of health on the battery. In Atlanta we found out that we only had brought 4 solid batteries out of the 8 we had shipped. The only way we found out was through a load tester that 1114 gratiously let us borrow. Needless to say the first thing we bought when we got home was the exact same battery tester as 1114.

I’ll list it here: its an Midtronics SCP-100 SCP 6/12
http://www.midtronics.com/home/findaproduct/byproducttype/batterytesters/scp100.aspx

12

Joe, thanks for posting my thread. I knew I had done something like this, but had stopped because it was a lot of work to lay out clearly in flow diagrams. :slight_smile:

A lot of problems originate from plugging things in wrong. You know that the same style of connector is used for Analog inputs, Digital inputs, Digital outputs, PWM outputs, and Relay outputs. It’s not keyed, and it doesn’t come labeled.
I highly reccomend getting a adhesive-tape labeler and labeling each data cable on both ends. Also, mark up your motor controllers, sensors, Digital Sidecar, and Analog Breakout so it’s obvious which way to plug the cable in. (It’s not fun when you fry a $60 gyro because the cable was in backwards and 5v and 0v were reversed)
Here’s an example of a marked-up Jaguar:
http://kamocat.com/photos/jag_colorcode.png

13

I’m by no means a programmer/electrical person, but if you do this I have a possible request. This may already exist or have some completely different fix/be impossible, but here goes:

I end up intermittently coordinating our programming sub-team. (why? good question. we’re working it…) Unfortunately, we have trouble understanding, not to mention fixing, all the error messages that can come up on the dashboard. We’ve started to develop a quasi-symptom based list: e.g. “this one means reboot the computer”, “that one means push OK really fast until it disappears”, “that one you should just drag into the corner and ignore”. …This doesn’t strike me as the best way to handle the situation. It’s (rather unsurprisingly) gotten us into/not out of some difficult spots before. :o

So if it’s possible, we’d really appreciate a reference list of error and explanations and/or fixes. Keep in mind that I honestly don’t know what this entails, all I know is that my team (who does have a better grasp of programming) really wants it. I’d appreciate any direction you’d like to offer.

[/non-programmer-electrical talk]

14

Good idea. What error messages do you see though? I’m especially curious about the ones that produce popup dialogs - don’t seem to recall any of those.

I’m going to talk to my team at our next meeting (two days from now), and probably create a wiki somewhere this week.

15

Thanks, good question. I’m currently away at school, but I’ll see if we can’t get a list together. Thank you.

16

I posted these in a different thread so I am copying them here…
Here’s a couple of hints that are common problems…

  1. Whiskers at the termination point. A stray strand of wire, in a connector, touching an adjacent wire in a connector (PD to Crio and sidecar power). This is easily handled by not stripping the wire too long. Twist the wires before insertion. If there is any copper showing when you are done, hold the wire in one hand and pull down on the insulation towards the connector with your other hand. The insulation will stretch and cover the exposed copper.
  2. Use heatshrink to cover up exposed wiring. It is cheap and fun to use.
  3. Insulate the battery terminals when you are finished terminating the power wiring. The battery is capable of 600+ amps when fully charged. It won’t kill you but it can cause a halacious fire in the right setting.
  4. Use the “tug” test on crimped terminals. Pull on the terminal with all your might, it should not pull off and it should not move. A ratchet crimper will cure most of these problems and costs about $50.
  5. Tie down all wiring near the termination (PD, connector, speed controller, Spike, motor, etc.) The robot moves, sometimes violently, so don’t depend on the terminal or connector to take care of itself.
  6. Inspect every connection after every match. Use your eyes, ears and nose.
  7. Color coding everything makes troubleshooting so much easier. See some of my other posts for details and our motor sheet is on here somewhere as well. A sheet with color codes, motors, servos, PWM inputs, breakers and whatever else the team needs for distribution to electrical, mechanical and software keeps everyone on the same page.
  8. Make sure that every student that works on the robot is knowledgeable and trained or is working with someone who is. It takes time but wins matches.
  9. The robot rules for wiring includes a color code. It is there for a reason. If a sponsor wants to donate wire make sure they know, you can only use certain colors.
  10. See #6 above, repeat as often as you need to feel safe and secure.