troubleshooting

[kamocat]

One of the biggest problems I see in people with troubleshooting is not knowing where to start.
There's people that will blame it all on the programming, or blame it all on the mechanical. There's people that will ask you "what's the problem" as if you knew what it was and simply were slow to fix it.

Troubleshooting is difficult, and it usually requires a lot of experience. In FIRST, we're lucky to have almost identical control systems. Most robots use the same concepts. (e.g. limit switches, joystick drive, button toggles, pistons and motors) Most of us run into the same sorts of problems, if not the same exact problems. However, many of us do not get adequate testing before the regional. We end up finding out at the regional that major components of our robot don't function.

It seems reasonable that we could make some sort of document to help people in troubleshooting. You may think "they can just search the forums", but as stated in Bill's Blog, wifi hotspots at regionals are disallowed this year. In addition, forums aren't very printer-friendly – they are post-centric, not content-centric – and don't lend themselves to definitive and well-organized content.
While it's true that there are many teams at the regionals that are more than eager to help, it's much harder to communicate the process of finding the problem than to communicate the problem. Saying "you plugged your PWM cables in backwards" doesn't help them find a problem in the future, except for that one thing. Because of this, rookies often have to rely on veterans for help and troubleshooting. Ideally, we would like to shift from reliance to collaboration as the team progresses. Usually this happens, but this can take a couple years. The faster a team can switch from reliance to collaboration, the earlier it can provide an engineering experience to its students.

There are other resources that would aid in troubleshooting. (e.g. documentation and clarity, how to use instrumentation, predicting how your robot will fail, compartmentalizing and isolating the parts of the design). This thread is about creating a document to describe what problems are often run into, and a checklist to methodically check common points of error or failure.

As for why I'm posting it here: I want to make the resources I create available to the FIRST community. Collaboration and sharing of resources is an important part of FIRST, and as we improve the resources available, we can overcome greater technical challenges.


I welcome input, collaboration, questions, and other resources.
Below I've listed some categories that problems fall under:

• low battery
• no communication
  • Robot radio
  • Driver station
  • Main router
  • programming laptop
  • cables
• watchdog
  • no communication / intermittent communication
  • code executing too slow (user watchdog)
  • robot disabled
• Power wiring
• Data wiring
• Mechanical
  • friction / binding
  • leverage
  • opposing forces
  • motor speed / torque
• programming
  • forgot to implement
  • error in coding
  • concept doesn't work

[Al Skierkiewicz]

Marshall,
I repair and troubleshoot systems everyday, sometimes down to the component level. On occasion it even results is a redesign of a system or device. People who know me or have watched me work know I take the following approach even though it may sound odd when you first read it. Listen to the device, it is trying to tell you what is wrong. I know people look at me strange when I say this but it is often true. In the case of our current system, there are LEDs on everything. They tell you a good deal about what is working or not. Take the Crio for example, just the LEDs next to the LAN connector can tell you if there is a connection and if it passing data. The LEDs on the PD tell you what power supplies are up and active and this year, what breakers are tripped or missing. Your transmissions tell you when they are stressed, binding, misaligned. The same goes for motors, servos, wheels and camera.
Case in point, I was called in to analyze a teams robot at Championship a few years back. The team was sure the breakers were tripping when the robot went between forward and reverse, but were not sure why. I had the team put the robot up on blocks so that the wheels were not on the floor and had them demonstrate forward and reverse. The noise that emanated from the frame was unreal. When I told the team they had a mechanical problem, they looked at me like I was crazy. I asked them what they were using to drive the wheels. They had a chain drive to a six wheel system with no provisions for adjusting for chain stretch. As the team went between forward and reverse, the chain would throw a link up into the frame and jamb the drive temporarily. (there was less than 1/4" clearance between the sprockets and the frame) It was so loud, there should have been no doubt as to what the cause was, but no one was listening. They still didn't believe me so I had them look for scratches on the inside of the frame and sure enough there was the evidence.
So your best tools in the box are your eyes and ears. Use them and believe in them.

[DonRotolo]

I work in the automotive industry, where troubleshooting is a career.

A good place to begin is with a symptom. Specifically, what is the system doing or not doing that makes you want to troubleshoot? Then, we work down a list of "possible causes" in rough order of probability or likelihood until we find the actual problem. "Working down a list" also assumes one knows how to check that something is correct or not.

We call this the "Three Cs": Complaint, Cause, Correction

So, if anyone wants to contribute, one way would be to state a symptom, list the possible causes, and how to identify if that potential cause is 'good' or 'bad'.

Example:
Complaint: motor connected to Victor not working
Possible Causes/Corrections:
1. Victor has no power (LED is not lit at all).
1a. Verify +12 volts in correct polarity at Victor Input screws. (If none, see "power")
2. Victor is not receiving PWM commands (LED is flashing orange, should be solid orange).
2a. Check that PWM cable is plugged into correct port of Digital Sidecar (verify correct port with Programming team) and is oriented correctly (red wire is +, white/yellow wire is SIG).
2b. Check that PWM cable is correctly plugged into the Victor. These cables are tricky to get plugged in correctly, you need to get a feel for it. It can look correct and not be connected!
2c. Check the PWM wire by swapping it with another, ideally one that is known to be good. You can also unplug the PWM cable form a Victor that is working correctly and plug it into the compliant Victor, it should then show a solid orange LED.
2d. Check if robot is disabled (no PWM commands are sent when disabled)
2e. Verify that Programming is incorrect (see Programming section)
2f. Check if Victor is faulty (swap with a known-good Victor)

(This would continue for all Victor problems we know of)

Now, if we can just build up a library of checklists like that, teams can troubleshoot and also learn the process. A whitepaper, with an index in the front (or a searchable PDF?) might help simplify things.


For anyone interested: This is a form of Failure Mode Effects Analysis (FMEA).

.

[Jon Stratis]

I'm with Al on this one - the very first thing you should do is look and listen, then apply your knowledge. Generally speaking, we follow these steps:

1. Look at the status lights. Everything has a status light that tells you what it's doing - if the light isn't doing what you think it should, you can start at that light and work backwards.

2. Start with checking the mechanical contraption involved. Check it for binding, odd noises, high friction, anything that could possibly cause your symptoms. If possible, move the contraption through it's motions manually (with the power turned off) - if you can't do this, be able to explain why (for example, the window motor can't be back-driven by hand due to the built in screw gear). Check for any points where something might get stuck. If you can, move it through by hand (or use a drill to run it) at full speed, something might "jump" out at you, as in Al's example.

3. Try manually powering the mechanism. Take a battery and hook it up directly to the motor leads by hand - does it move as you would expect? Can you measure the current being used to perform the motion? Bonus points: Use something like the Anderson PowerPole series to perform all of your power connections, and create a jig with a PowerPole output on one end, and Anderson quick connect for the battery on the other, and a momentary push button to enable power, allowing you to have fine motor control by hand.

4. Move on to the electronics. Starting as close to the mechanical contraption as possible, start working your way back with the multimeter. Check the voltages the motor is getting. Check the voltages the speed controller has for input and output. Check the PWM signal. As soon as you can say "this is correct at this point", checking anything further "upstream" of that pathway drops to the bottom of your list.

5. Move on to programming. At this point, you should be absolutely confident that the expected signal is not being produced by the digital side car, and all the wiring is correct. Check the slot and channel numbers, check the logic, use print statements, create a small test program that simply turns that mechanism "on" or steps through moving it.

These are basic steps to take to localize the problem to the point where specific experience can help guide you. It can certainly be made more specific (for example, exactly where to stick the multimeter to test a PWM cable) A series of lists like Don's could certainly help... but they also involve no critical thinking and as such don't help build up the problem solving skills teams need. Further, due to the amazingly wide range of contraptions people come up with, it's going to be impossible to list out troubleshooting steps for every possible situation.

The one issue i have with Don's example is that it completely discounts any possible mechanical problem - a "motor not working" may be due to excessive friction stalling it, and not be an electrical problem at all.

[kamocat]

Thank you! Let me try to combine those into an example troubleshooting sheet for a specific mechanism and a clear problem. (I realize this lacks some of the 'solution' parts on the hardware side, but I think it narrows it down enough so that "fix it" is a clear task.)

Our kicker is pneumatically actuated, with a spring return. The piston has two magnetic switches; one for full-forward, and one for full-reverse. It has a timed actuation determined by axis 3 (throttle) and actuated by button 1 (trigger) of the Joystick in USB port 2 (of the Status tab of the Driver Station).
Currently our pneumatic solenoid is normally open, so when the trigger is pressed, it should flash OFF. (this is technically a safety hazard, because the solenoid outputs are disabled when the robot is disabled, causing the kicker to fire if there is enough pressure. We'll fix it when we get a new solenoid.)

Here is a troubleshooting sequence for our kicker. Please tell me if I've missed anything.
(Problem: doesn't kick when trigger is pressed)


Is the robot enabled?

• Is there code on the robot?
• Do you have communications?
  • Have you waited a minute for the robot to boot up?
  • Can you ping the robot?
    • Can you ping the robot radio or router?
      • Check your IP address
    • Check the cables and IP addresses of the router and robot radio
  • Can you load code on the robot?
    • Does the cRIO target have the correct IP address in the Project Explorer?
    • Can you "connect" to the target?
      • Reboot the robot.
    • Try loading code again
  • Reboot the robot. (If it has been E-stopped, this is the only option)
• Is the kicker LED (on the solenoid module) ON by default?
  • Modify "kicker begin.vi" to turn the solenoid ON when it is initialized.

Does the kicker LED flash OFF when trigger is pressed?
(It's possible for it to flash too quickly to see. Make sure it's actuated for at least 100ms)

• Is the piston fully retracted?
• Are the magnetic switches on the piston tuned and and wired correctly?
• Is the Joystick in the correct port? (look in the Setup pane of the Driver Station, and drag it to the position it should be in)
• Run Robot Main and show execution for the kicker. Is the kicker being actuated?
  • Work outwards, showing execution in VIs, until you find the problem.
• Is the "Solenoid set" VI executing without errors?
  • It's probably a conflict with the IO being opened multiple times, or the cRIO needs to be re-imaged.

Is there adequate pressure after the regulator?

• Does the compressor turn on?
  • Is it wired correctly?
  • Is there code to start the compressor?
• Is there a leak in pneumatics?

When actuated, does the solenoid make a quiet "click" sound or light an LED?

• Is the solenoid wired up correctly?
• Does the solenoid breakout have power?

Does manually actuating the solenoid actuate the kicker?

• Is the solenoid correctly connected to the cylinder?
• Does the solenoid provide enough force to overcome the spring return?

The solenoid is not actuated long enough to extend the kicker.

[kamocat]

Does the lack of reply mean I've missed nothing?

[Jon Stratis]

No, it probably just means that it was the weekend after ship and we were all catching up on our sleep

It seems pretty complete, but my team hasn't used pneumatics since our rookie year, so I can't say for sure. You have a bunch of yes/no questions listed, but it's not always clear what should be done if the answer is yes or no. What might make more sense is to use some form of decision diagram for the layout - every question asked gets lines going off it for each answer, which leads either to more questions or a possible solution. I got it started to give you an idea (see attached image).

The nice thing about doing it this way - you can ensure the question is always answered in both directions with a quick glance, you can add in helpful statements/information at important steps to help guide the team, it will be obvious when you're done, as you'll be left with nothing but boxes (no diamonds), and it's easy enough anyone can understand it. The downside, however, is that it takes up a lot of room.

You can also divide it up with pointers to different sections. For example, "diagnosing robot communications" could be it's own separate diagram, and the "No" arrow from "Do you have communications" could simply reference it. That would help with the clutter and space needed for some of the more common problems that could exist with most items.