Gremlins and Troubleshooting

[Chief Hedgehog]

FRC 4607 had a terrible showing at our only regional this season (North Star).

I am the Head Coach/lead Mentor of FRC 4607. In the previous two seasons we had issues starting out, but our troubleshooting from the PIT team led us to find the problems and adjust quickly.

However, this season we had problems that plagued us throughout our only regional. Just to note, we do build a practice robot that we use to optimize our manipulators. The practice robot has an identical drive train, frame, etc. to our competition robot. Many of you saw our two release videos - and from our practice sessions we could stack 2 sets of 4 totes with ease and then cap with available time. Our competition robot worked like a dream before Bag'n'Tag - and our practice robot was a beast.

When we arrived at North Star we transferred our manipulators (intake and the forks) onto the competition robot. We ran the manipulators in the PIT, everything worked as they should. However, when we put the robot on the field, we had considerable problems connecting to the FMS. We spent 8 matches where the CSA's tried to get our robot to connect, and then after each match we had 2-5 CSA's in our PIT. We had the worst luck I have seen.

After 8 matches we eliminated the code, the robo-rio, the wiring, the VRM, and an errant 10amp breaker. The drive team finally had the opportunity to drive the robot at full functions in the 9th and 10th rounds at NS. However, it was too little too late.

So my question is this: What is your order of trouble-shooting the robot? FRC 4607 is lacking on this (as evident by our showing). How do we get beyond this in the future so we never deal with this problem again?

Thank you.

[tomy]

Did you guys decide to use a camera at all on your robot?

[rsisk]

For this particular type of problem, it is tough to identify before getting to the field at the competition.

It is always advisable to get to practice matches as soon as possible. Once you complete your safety inspection, get the robot to the field and confirm it can connect to the FMS. At least then you would have been troubleshooting during practice day rather than during the qualification rounds.

[Greg McKaskle]

I was CSA at three events this year. I almost always start with the DS logs to see what it is reporting. After being at a few events, I also started checking certain electrical connections for preventative measure.

The logs will show what is happening with electrical and code and will help determine where to start. By far, the most common failure I saw this year was loose electrical connections at the main breaker/switch. The connections were loose, often wobbly, and whenever possible, I would try to show the teams the arc marks. Loose battery terminals were also common. And probably 80% of the PDP fuses were not inserted fully. I would seat one, and have a student seat the other. You could see their look of recognition when they felt it insert.

On the laptop, the firewall is the biggest offender, though there were other odd networking setups we ran across. We saw SW updaters interfere with comms way more often than in previous years.

Coding issues were a mix of uncaught exceptions, deadlocks, and incorrect IP setup.

If you have questions about any of these, please ask.
Greg McKaskle

[Justin Montois]

How many practice matches were you able to get in on Thursday?

I think having a practice robot is great, but if it's stretching your resources so thin that you don't get enough time both before bagging your comp bot and on practice day to get your robot on the field and de-bug then it might not be worth it.

Great teams make great use of practice bots but it's an incremental process to utilize it most effectively.

If you weren't able to get in any practice on Thursday, investigate the reasons why and make changes accordingly.

[Alan Anderson]

Originally Posted by Chief Hedgehog

So my question is this: What is your order of trouble-shooting the robot?

Step zero: check the obvious power connections. Main breaker, PDP input, PDP fuses, wiring to roboRIO/VRM/D-Link. Plus the battery terminals.

If it works properly in the pit and when tethered on a practice field, I figure the robot itself is probably okay, and I start troubleshooting with the Driver Station logs. That will often point to an intermittent power connection or a communication issue. If power is okay, I look at the network settings. If they're good, I check for firewalls, antivirus scanners, software updaters, and other potentially disruptive background tasks.

Then I suppose I start following hunches instead of checklists.

[rod@3711]

Sound much like our experiences. Ours was due to using the HD3000 usb camera. When we finally removed the code that enabled the camera, we got rid of our gremlins and the CPU usage chart dropped from 95% to 50%.

[MrForbes]

As mentioned, having a practice robot can really mess up your ability to get your robot "finished" before bagging, and ready to be inspected and practiced with on the first day. I've seen a few teams really struggle their first regional because of this.

As for the gremlins...I'll just leave this here.

https://www.youtube.com/watch?v=DQU4UrWzQAs

[marccenter]

Dear CH,

Besides all the good advice above, you may want to add removal of all power saving features on the laptop running the driver station and dashboard. This was recommended by Omar, a CSA at the Troy MI district event.

Additionally, I moved some of my dashboard variables from the teleOp routine to a periodic 1000 msec routine. This anecdotally seemed to help with some radio dropouts on our system.

I definitely second getting rid of the USB camera unless your drivers really use it in the matches.

Best regards,

[philso]

Originally Posted by Chief Hedgehog

So my question is this: What is your order of trouble-shooting the robot? FRC 4607 is lacking on this (as evident by our showing). How do we get beyond this in the future so we never deal with this problem again?

I am interpreting the OP's question as "How does one do quick and effective troubleshooting?"

Start by calmly and coolly analyze the problem to get insight into the possible root cause. "Hurrying too fast" may lead one to miss important clues.

Try to observe the failure, multiple times, from different view points. Use tools to give you more view points; i.e. take voltage readings, check for glitches using an oscilloscope, use a camera with a high frame-rate.

Often your robot will have sub-systems that contain "closed loops" where some component is controlled by another and there are sensors monitoring the status of the sub-system. A fault of any kind at one point can manifest itself as a fault at another point in the loop. Probe the system (physically or in software) to systematically follow around the loop to find the fault. Do "sanity checks" for the values measured at various points around the loop.

Disable sub-systems, one at a time, to exonerate them and narrow down the location of the fault.

"Torture" or (over)load the system to cause it to exhibit the fault more frequently or more strongly.

Once the fault is narrowed down to a particular sub-system, work through it in a thorough and systematic way to identify the problem; i.e. signal is present on PWM output of RoboRio, signal is present at motor controller, motor controller has power, motor controller output has appropriate voltages for control input, voltage present at motor, motor shaft is spinning in the correct direction --> find one gear slipping on shaft in gearbox.

If a system is open-loop, it is sometimes possible to speed the trouble shooting process up by using a "binary search" technique. Start by probing at some point near "the middle" of the system and checking if the values seen are good. If they are good, probe at a point half way towards "the end". If they are bad, probe at a point half way back towards "the beginning". Eventually, one arrives at a point where the input to a component or sub-system looks good but the output looks bad.

Discuss the evidence with people with the relevant expertise (on your team and from others). Keep an open mind. A mechanical problem could cause a brief brown-out and cause something that appears to be a software problem.

A checklist is helpful to ensure that one covers all likely points of failure but it is no guarantee that one can discover the cause of a failure. A checklist is much more effective if one can get some insight regarding the possible cause of the failure so certain items can be skipped. The OP's post indicated software/configuration and electrical issues so it would probably have been fruitless to check things like chain tension.

Trouble shooting is a skill that has elements of nature and nurture. You will find that some people have the natural ability to quickly home in on the root cause of a problem. If you can identify such a person (student or mentor) on your team, make sure they are on your development team and your pit crew. One could say that the team member's trouble shooting abilities will be nurtured and developed as part of their FRC experience.

[GeeTwo]

A troubleshooting checklist only helps you identify problems you've already seen, or at least anticipated. For a 3+ year old automobile, following the manufacturer's checklist, you'll be covered 99.44+% of the time. For a robot that you started designing in January, not so much.

Not to say that you shouldn't have a checklist; you should. Include checks for things you've already seen, and things you might expect (as described by several posters above). This checklist may be able to find 75% to 90% of your problems, or even a bit more, depending on how thoroughly you made it. Part of this checklist may be to run the robot in "test mode". For this to work, you have to program a test mode which will provide very simple open loop inputs to each actuator so that you can verify that the output signal results in the appropriate behavior. If you use feedback sensors, the results of these sensors should be reported as well.

Once through the checklist, follow a method. Your individual techniques may be a bit more scientific or a bit more engineering-based depending on your predispositions, but the key points are:

• Describe what is going wrong (in terms of behavior, not cause).
• Run a test case or two with a few variables different to verify that behavior.
• Adjust your description if necessary.
• Form a hypothesis (or several) as to what is causing the problem.
• Test the hypotheses - look for a test which distinguishes among them (preferably which cuts the list in half or smaller based on the results). Get down to one thing, or at least a small set.
• Repair/replace/bypass/probe that one (small set of) thing.
• Repeat steps as necessary, eliminating one or more possible causes each pass.
• When you get it working, look for an update your troubleshooting checklist that would identify the issues you had or thought you might have.

[BeardyMentor]

Trouble shooting is very difficult. It usually takes a good understanding of the system, experience with other problems, and knowledge of the history of the system. I have done this for quite a while on a large variety of equipment ranging from military aircraft, industrial automation, CNC machining equipment, 3D printers ranging from $1,000-$2,000,000, among many others. Trying to codify how to do it is always frustrating as it is very difficult to hit all the bases. More so on a machine like an FRC robot where there is very limited experience with this particular situation.

The first step is usually to very clearly define what is happening or not happening. Try to run a function check of everything on the robot. This can reveal problems that are related and possibly give more clues to the cause.

Next write it all down. Our team keeps a binder with our prematch inspection checklists and every single problem we encountered any time we were operating the robot in a mostly completed configuration (from week 5 in build season or so). We have this year made it very granular, down to the level of recording every loose screw and broken zip tie. This has been invaluable in finding and tracking down intermittent problems.

Check the stupid stuff. No, really check the stupid stuff. Did you include all the appropriate libraries in the code? Is your Ethernet cable plugged in all the way? check it again anyway. Did you read the datasheet on how to properly operate the weidmuller connectors? Lightly tug on all the wires, anything pop loose? In my experience, at least 75% of the time it is some thing silly like an unplugged connection, bad crimp, blown fuse. Basically, define the problem, then start checking every part that can possibly contribute to it. Do not assume anything is fool proof or was done right. Even the stuff you did yourself.

Finally, design with the knowledge that everything will break eventually. Do not hope that the relay you tucked underneath the toughbox with 2 cims, an encoder, 5 million feet of wire between you and access to it will survive the whole season. You know it will not because it is the hardest part to access. Keep things neat and easy to access. Nothing sucks more than trying to track down that 5ft long PWM cable as it runs through a rats nest of wire. Remember Murphy's Law; What can go wrong will go wrong. Also, The Addendum to Murphy's Law; Murphy was an optimist.