Fire at Michigan State Champs

[Kevin Sevcik]

Originally Posted by Jaci

When a high amperage electrical circuit shorts, touching the metal chassis of the robot is a good way to become less alive.

You might be badly burned if the frame is hot, but Ohm's law plus physiology says you're extremely unlikely to suffer any injury from the electricity itself. Wikipedia says the resistance of wet skin is about 1 kOhms. So a 12V battery can push, say, 12 mA through you. The chart on that page notes that 12 mA of AC would be noticeable after a few seconds, but not deadly. It's important to note that DC is much less dangerous in this regard than AC. Unless you have wires connected directly to your heart[1], it takes a lot more DC to kill you than AC. If anyone is doubtful, I can hold a demonstration at Champs where I will bravely grab both terminals of a robot battery with either hand an suffer no ill effects.

Mind you there's plenty of other dangers from the battery, but they're all the fire, explosion, direct-short causing a welding arc sort.

TL;DR: The robot battery (and your car battery[2]) aren't going to electrocute you unless you stab a positive and negative wire from them into your heart. In which case you have plenty of other problems besides.

[1]I have before. It was weird.
[2]Had a dead battery once in college and got a jump from a doctor. He gave me the same electrocution warning. And he should know, because he's a doctor. I didn't argue because I wanted to get home.

[rsisk]

If there was a short with the frame, does this mean the frame isolation test during inspection was not done?

Or would this happen even if the frame isolation test passed?

[Jaci]

Originally Posted by rsisk

If there was a short with the frame, does this mean the frame isolation test during inspection was not done?

Or would this happen even if the frame isolation test passed?

It is possible even if circuits are electrically isolated from the frame before the match. If the outer shielding of the wire (or a solder joint, or wire crimp) becomes damaged, it is very possible for it to contact the robot chassis. With the rigors of Stronghold, this is entirely within the realm of possibility.

[GaryVoshol]

Originally Posted by Jaci

It is possible even if circuits are electrically isolated from the frame before the match. If the outer shielding of the wire (or a solder joint, or wire crimp) becomes damaged, it is very possible for it to contact the robot chassis. With the rigors of Stronghold, this is entirely within the realm of possibility.

From talking with some inspectors, that is the theory. The wire that caught fire went around a corner on a frame piece, and it is suspected that the insulation wore through because of robot movement.

In addition there were some other factors that may or may not have come into play. There may be a suggestion made on additional safety tests in inspection, or a change in wiring rules. I don't want to be more specific in case I am wrong.

[Ben Wolsieffer]

Thinking about this now, I realize that something like this could have happened to us. We had a almost unmeasurable frame short (later discovered to probably caused by a limit switch wire connected to a Talon SRX), which passed inspection because the resistance was higher than the required limit.

Last night and this morning we fried two Talon SRXes, and after a ton of troubleshooting we discovered that our LED ring power cable was being pinched and sometimes connecting positive to the frame. If the resistance was a little lower, I think the same thing could have happened to us. This appeared to be what destroyed the Talons.

[Jon Stratis]

Originally Posted by rsisk

If there was a short with the frame, does this mean the frame isolation test during inspection was not done?

Or would this happen even if the frame isolation test passed?

There's a lot of variables that can go into this, we can't say for certain if it was or was not done. It's possible it was done, but after the fact the insulation wore through. It's possible the frame short was intermittent, and only occurred when a wire was pinched in a mechanism a specific way. Or the team was doing a hasty repair and pinched a wire, or drilled through the insulation of a wire, or disconnected a motor and had the leads just dangling... There are any number of ways for the frame to short out either before or after a frame isolation test is done. And most of the time the frame isolation test is not performed during reinspection.

The reason we do the frame isolation test is to ensure that it takes two faults to cause a problem, not just one. Two faults obviously can still happen, but it's half as likely to happen as having one fault.

[jvriezen]

Originally Posted by Jon Stratis

wo faults obviously can still happen, but it's half as likely to happen as having one fault.

Is this true? If the likelihood of one fault occuring is 1/100, then the like likelihood of two at the same time is 1/10000.

[Analog]

See pic

[Koko Ed]

Originally Posted by Analog

See pic

Best Pit Prank ever!

[Mike Schreiber]

Originally Posted by rsisk

If there was a short with the frame, does this mean the frame isolation test during inspection was not done?

Or would this happen even if the frame isolation test passed?

I was the initial inspector on this robot. We did the isolated frame check and it passed with no issue. Wiring looked pretty clean actually.

[IKE]

Originally Posted by Mike Schreiber

I was the initial inspector on this robot. We did the isolated frame check and it passed with no issue. Wiring looked pretty clean actually.

As Mike said, the team had passed initial frame isolation check. They "dropped coms" earlier in the day, and after a few minutes, I went personally to check on them.
They said they had already found the issue, a loose Anderson connection on the battery as well as cleaned up the VRM input which had a possible whisker.
I verified those and was about to leave, but asked the team if this was the first time. They replied they had other drops at other events. I did a frame isolation check and found the positive side grounded to the frame. The team had LEDs, and it was one of the first items I asked about. The team told me they were disconnected*.
We then dug through the various areas and found a different item that would clear the frame isolation. Once cleared, I returned to the field. couple hours later, the fire occurred. I do not have first hand details of how the fire was handled as I was covering the other field, and a large group of people were gathered around, and I thought it was some robot with a typical sensor wire or PWM that had been pinched and smoked.

After the fire was out, the issue was diagnosed as a power cable that had previously fed the LEDs. It had a barrel connector that was disconnected earlier.
We suspect that the positive side frame grounding was intermittent and the barrel connector likely was exposed negative. This lead to the power cable catching fire when both sides grounded to the frame. It was relatively thin wire, so a short caused it to catch fire relatively quickly.

The team did an amazing job of keeping a positive attitude. They had a student lead the rebuild, and she did an amazing job of keeping students on task. One of the new Michigan LRIs working the event stayed with them to ensure it was rebuilt properly. After initial teardown, we inspected the remaining components and had the team not only change out most of the electronics, but most of the pneumatics as well. We were quite fortunate no one was injured, and the team only missed 1 match.

I know a report was sent in about the fire incident, and I plan to talk some more with other LRIs about possible ways of improving and/or mitigating future incidents.

*I missed asking the team exactly how they "disconnected" the LEDs. I did not see the barrel connector, and assumed the disconnection was done properly. I will be adding a more thorough line of questioning when "something is disconnected".

[JimBryson]

As lead engineering mentor for 3656, Dexter Dreadbots, I can confirm events are as Mike and Ike say above. I appreciate both of their help in inspecting and troubleshooting our robot.

Our failure analysis indicates that we had two shorts to our frame (usually you can assume that multiple failures won't happen at the same time--but as Ike says, there was foreshadowing, but we didn't catch the root cause). There was a 12V connection and a ground connection, both to the frame, leading to heating of narrow gauge wires and melting of the PVC insulation, and eventually to fire.

The ground connection was a no-longer used barrel connector wired on fine-gauge dual-extruded wire connected to a 20A breaker on the PDP (not so smart in hind-sight). The old wiring had not been removed, and the exposed barrel connector with GND external was in the robot, likely tucked away by a well-meaning team member. Excess length of wire was bundled together with a zip tie between the RoboRio and PDP. More on that later.

The disconnected LED power that Ike wonders about was our new wiring. The LEDs had been rewired (to another 20A breaker), but with powerpole connectors and GXL or TXL 20awg wire. These were unplugged while troubleshooting the ground fault. Checking wiring it seemed that pushing on the PCM bracket could make the ground fault come and go. A student rewired it, looking for "whiskers". The problem went away without a "smoking gun", and we assumed it was fixed. The unused barrel connector wires passed through the same area, so moving wires likely moved the barrel connector.

The 12v source was the battery anderson connector to the main breaker lug. The main breaker was mounted close (~1/4") to a vertical frame element. It is likely that with repeated battery installation and removals and a high impact game, that the lug may has spun slightly and have allowed current to the frame, by splitting the electrical tape or maybe the taping wasn't perfect. Difficult to tell after the fire. We could not recreate the 12v to the frame that Ike had seen, probably because we had unplugged the battery and moved the lug slightly while looking for the ground issue. Strain relief and more flexible wire for battery hookup would both be helpful with this, as well as additional clearance and better insulation.

I was confused how both could be happening at the same time, and assumed we were not understanding what Ike was saying. But we could not recreate the faults, and we were called to que.

During the match (see MIDC qual 36 video https://youtu.be/TfYDX4_zND8) our robot let out the smoke about 73s into the match. The driver hit the e-stop at 77s, and the FTA turned off the main breaker about 103s in. The robot appears to stop smoking. After the match was completed and the robot was on the cart around 250s after the match started (not on the same video) the video shot by Byting Bulldogs shows the active flames. Our drive coach had noticed that the RSL was back on about this time. This may be due to the 12v from the frame backfeeding into the PDP after all the insulation was burned off the paired wires.

It is likely that the short reoccurred after putting the robot on the cart, or that the heat had been slowly building. At some point the heat was enough to melt through a pneumatic line, adding oxygen to the "smoldering" insulation and allowing it to build to a full fire. Most of the fuel for the fire was from the PVC insulation on the LED wiring (coiled up for maximum heating and fuel potential) and pneumatic tubing. The RoboRio contributed some as well. Our standard for wiring is to use GXL or TXL wire, and the insulation on those wires scorched slightly, but it did not burn or melt.

Our students were great at repairing the robot. Many teams offered help and parts. We had students who could and did do the work, and had most of the parts ourselves. As a mentor, I was very proud to hand out tools, get spare parts and help clean up, but not to do the rebuilding. We moved the robot and tools outdoors to several tables because we would not all fit in the pit, and to allow the dispersion of the dry chemical residue. We decided to replace all the electronics and wire (not motor controllers though) and had to replace several plastic components and one pneumatic cylinder that may have been damaged.

It may look like a lot of time passed before the extinguisher was discharged, but heat damage was not extensive. We now have an extinguisher on our cart, and I am looking into procuring a CO2 one for less collateral damage. We have moved (and further insulated) the main breaker and are looking into designing and 3d printing an insulator box that would offer strain relief as well.

After 4 hours of rebuild we were on the practice field with a fully-functional bot having missed only one match, and are now on our way to St Loius!

[Tom Line]

Originally Posted by JimBryson

As lead engineering mentor for 3656, Dexter Dreadbots, I can confirm events are as Mike and Ike say above. I appreciate both of their help in inspecting and troubleshooting our robot.

Our failure analysis indicates that we had two shorts to our frame (usually you can assume that multiple failures won't happen at the same time--but as Ike says, there was foreshadowing, but we didn't catch the root cause). There was a 12V connection and a ground connection, both to the frame, leading to heating of narrow gauge wires and melting of the PVC insulation, and eventually to fire.

The ground connection was a no-longer used barrel connector wired on fine-gauge dual-extruded wire connected to a 20A breaker on the PDP (not so smart in hind-sight). The old wiring had not been removed, and the exposed barrel connector with GND external was in the robot, likely tucked away by a well-meaning team member. Excess length of wire was bundled together with a zip tie between the RoboRio and PDP. More on that later.

The disconnected LED power that Ike wonders about was our new wiring. The LEDs had been rewired (to another 20A breaker), but with powerpole connectors and GXL or TXL 20awg wire. These were unplugged while troubleshooting the ground fault. Checking wiring it seemed that pushing on the PCM bracket could make the ground fault come and go. A student rewired it, looking for "whiskers". The problem went away without a "smoking gun", and we assumed it was fixed. The unused barrel connector wires passed through the same area, so moving wires likely moved the barrel connector.

The 12v source was the battery anderson connector to the main breaker lug. The main breaker was mounted close (~1/4") to a vertical frame element. It is likely that with repeated battery installation and removals and a high impact game, that the lug may has spun slightly and have allowed current to the frame, by splitting the electrical tape or maybe the taping wasn't perfect. Difficult to tell after the fire. We could not recreate the 12v to the frame that Ike had seen, probably because we had unplugged the battery and moved the lug slightly while looking for the ground issue. Strain relief and more flexible wire for battery hookup would both be helpful with this, as well as additional clearance and better insulation.

I was confused how both could be happening at the same time, and assumed we were not understanding what Ike was saying. But we could not recreate the faults, and we were called to que.

During the match (see MIDC qual 36 video https://youtu.be/TfYDX4_zND8) our robot let out the smoke about 73s into the match. The driver hit the e-stop at 77s, and the FTA turned off the main breaker about 103s in. The robot appears to stop smoking. After the match was completed and the robot was on the cart around 250s after the match started (not on the same video) the video shot by Byting Bulldogs shows the active flames. Our drive coach had noticed that the RSL was back on about this time. This may be due to the 12v from the frame backfeeding into the PDP after all the insulation was burned off the paired wires.

It is likely that the short reoccurred after putting the robot on the cart, or that the heat had been slowly building. At some point the heat was enough to melt through a pneumatic line, adding oxygen to the "smoldering" insulation and allowing it to build to a full fire. Most of the fuel for the fire was from the PVC insulation on the LED wiring (coiled up for maximum heating and fuel potential) and pneumatic tubing. The RoboRio contributed some as well. Our standard for wiring is to use GXL or TXL wire, and the insulation on those wires scorched slightly, but it did not burn or melt.

Our students were great at repairing the robot. Many teams offered help and parts. We had students who could and did do the work, and had most of the parts ourselves. As a mentor, I was very proud to hand out tools, get spare parts and help clean up, but not to do the rebuilding. We moved the robot and tools outdoors to several tables because we would not all fit in the pit, and to allow the dispersion of the dry chemical residue. We decided to replace all the electronics and wire (not motor controllers though) and had to replace several plastic components and one pneumatic cylinder that may have been damaged.

It may look like a lot of time passed before the extinguisher was discharged, but heat damage was not extensive. We now have an extinguisher on our cart, and I am looking into procuring a CO2 one for less collateral damage. We have moved (and further insulated) the main breaker and are looking into designing and 3d printing an insulator box that would offer strain relief as well.

After 4 hours of rebuild we were on the practice field with a fully-functional bot having missed only one match, and are now on our way to St Loius!

Congratulations guys. It was really inspiring to see you out by yourself on the practice field running that robot so quickly. See you in St. Louis!

[Al Skierkiewicz]

Jim,
If you could provide any pictures of the before location and the after location I think that would help teams going to Champs and for the future. I was getting reports from the inspection team and when I heard you had everything back together in four hours, I had to smile knowing that a good team overcame a devastating fault and came back with a vengeance. Good luck to the team next week.
Al

[Jaci]

Originally Posted by Kevin Sevcik

You might be badly burned if the frame is hot, but Ohm's law plus physiology says you're extremely unlikely to suffer any injury from the electricity itself. Wikipedia says the resistance of wet skin is about 1 kOhms. So a 12V battery can push, say, 12 mA through you. The chart on that page notes that 12 mA of AC would be noticeable after a few seconds, but not deadly. It's important to note that DC is much less dangerous in this regard than AC. Unless you have wires connected directly to your heart[1], it takes a lot more DC to kill you than AC. If anyone is doubtful, I can hold a demonstration at Champs where I will bravely grab both terminals of a robot battery with either hand an suffer no ill effects.

Mind you there's plenty of other dangers from the battery, but they're all the fire, explosion, direct-short causing a welding arc sort.

TL;DR: The robot battery (and your car battery[2]) aren't going to electrocute you unless you stab a positive and negative wire from them into your heart. In which case you have plenty of other problems besides.

[1]I have before. It was weird.
[2]Had a dead battery once in college and got a jump from a doctor. He gave me the same electrocution warning. And he should know, because he's a doctor. I didn't argue because I wanted to get home.

That is a fair point, I should have thought a bit further before posting that last line. Looking back, I should have known a bit better[1]. Even so, I still stand that touching an electrically charged chassis is far from a good idea, even if it's just for peace-of-mind.

[1]I do a lot of work with electrical systems. Also got shocked by a 240 ~3-6A wall socket (for the second time) a few years ago, less than a fun experience.