View Full Version : Fire at Michigan State Champs
MooreteP
15-04-2016, 05:53
Was looking for a thread on this yesterday:
Fire at Michigan State Champs (https://www.reddit.com/r/FRC/comments/4et7zf/fire_at_michigan_state_championships/)
Surprised it wasn't here yet, so I am linking to the Reddit post.
Yes, this made quite the impression. I was afraid the team was going to be bombarded wroth safety captains all trying to be the first to put it out. Fire extinguisher on the cart isn't a bad idea. We're redesigning ours this year I think it's would be a good addition.
Cory Walters
15-04-2016, 07:37
The team was able to rebuild the robot yesterday after the fire. They ran multiple practice match's and would appear that it functioning again. Everyone around was cheering them on! A great moment.
Al Skierkiewicz
15-04-2016, 07:51
The report I have is that the team rebuilt in four hours and ran in a practice match. The pictures I have seen show some burned wiring near the PDP. Good luck guys!
GreyingJay
15-04-2016, 08:33
An interesting comment in the original link by a poster saying, essentially, "all the safety captains and meticulously memorized safety code rules, useless when something actually happens".
I can attest to that feeling. A number of years ago my car engine caught fire while it was parked at a camp. (The running theory is that a rodent chewed through enough wiring to cause a short). There was a good few moments of "uhhhh what's happening?" followed by, in that moment, not having a clue what to do.
That moment led to writing down a specific procedure "in case of fire", that is printed and posted in every building. It is unambiguous. Step 1, this person do this. Step 2, everyone else do that.
There was a robot fire at the Greater KC event too.
Not as much flame, but smoked-up the place pretty good. Had to wait a while to start the next match until the smoke cleared.
I understand that the robot was up and running again later.
Heard third-hand that two motor wires were routed through a drilled hole with battery wires nearby. Motor wires wore through and shorted, lighting a bunch of insulation on fire.
https://youtu.be/K2gLfBm0s8s?t=100
Yes, I think it sounds like a great idea to actually have a plan for who will do what actions during a fire.
Mark Holschuh
15-04-2016, 09:16
Does anyone know what was the specific heat source that started the fire, and what the combustible material was that sustained the fire?
Drivencrazy
15-04-2016, 09:26
Does anyone know what was the specific heat source that started the fire, and what the combustible material was that sustained the fire?
/u/sourec commented (https://www.reddit.com/r/FRC/comments/4et7zf/fire_at_michigan_state_championships/d23b740) on the reddit post detailing the fire and showing some more pictures.
An expensive, unfortunate incident.
Kevin Sevcik
15-04-2016, 09:42
An interesting comment in the original link by a poster saying, essentially, "all the safety captains and meticulously memorized safety code rules, useless when something actually happens".
I can attest to that feeling. A number of years ago my car engine caught fire while it was parked at a camp. (The running theory is that a rodent chewed through enough wiring to cause a short). There was a good few moments of "uhhhh what's happening?" followed by, in that moment, not having a clue what to do.
That moment led to writing down a specific procedure "in case of fire", that is printed and posted in every building. It is unambiguous. Step 1, this person do this. Step 2, everyone else do that.It's like CPR training. Step 1, notice there's a problem and decide to do something. Step 2, tell someone to call 911. Step 3, make sure they heard and understood that. Step 4, actually start CPR.
Preventative safety is great, since it's a lot easier to not have to deal with injuries in the first place, but it seems like training and planning for what to do in an actual emergency is slightly lacking.
We had a similar incident at the Long Island regional in 2009 with Team 263. That was the result of the catastrophic failure of a Victor speed controller, sustained by bumper material. The facility manager wouldn't let the robot back into the building unless they determined it was fixed and wouldn't happen again.
I hate to be 'that person', but it has to be said.
Why did it take so long to grab a fire extinguisher that should have been ready at the field?
Why is there a group of volunteers all standing around a robot with a roaring fire and a live battery inside it?
Why are there volunteers sticking their hands inside and/or near the robot, let alone fanning the fire?
It seems to me like this reflects a lack of training for what to do in this situation. I would expect volunteers to be properly inducted about what to do in the event of an emergency such as this one. We've got 6 highly dangerous machines on the field and dozens more behind the curtain, all with massive batteries and electrical currents running through them, but it seems that no one knows what to do in the (quite likely) event that something goes wrong.
We're always saying that "Safety comes first", but it's events like this that make me question "are we really living up to that?"
orangemoore
15-04-2016, 10:40
I hate to be 'that person', but it has to be said.
Why did it take so long to grab a fire extinguisher that should have been ready at the field?
Why is there a group of volunteers all standing around a robot with a roaring fire and a live battery inside it?
Why are there volunteers sticking their hands inside and/or near the robot, let alone fanning the fire?
It seems to me like this reflects a lack of training for what to do in this situation. I would expect volunteers to be properly inducted about what to do in the event of an emergency such as this one. We've got 6 highly dangerous machines on the field and dozens more behind the curtain, all with massive batteries and electrical currents running through them, but it seems that no one knows what to do in the (quite likely) event that something goes wrong.
We're always saying that "Safety comes first", but it's events like this that make me question "are we really living up to that?"
Training for an emergency and being in one feel completely different.
I don't believe you can fault volunteers who aren't exactly sure what to do when they go to maybe 3-4 events a year. If they were trained for an emergency like this the chances of this happening has to be close to 1 in 100 events. So it isn't like this is something that happens so often that they would know how to react if it did.
I do agree there should be a fire estinguisher at the field to be easily found. And that perhaps maybe they shouldn't have been standing so close to a fire.
What are your concerns with a battery and fire?
indieFan
15-04-2016, 10:43
Volunteer training has not included "fire safety," as long as I've been a volunteer. Perhaps a solution could be to ask local firefighters if they would be willing to volunteer for the event.
Sperkowsky
15-04-2016, 10:45
I hate to be 'that person', but it has to be said.
Why did it take so long to grab a fire extinguisher that should have been ready at the field?
Why is there a group of volunteers all standing around a robot with a roaring fire and a live battery inside it?
Why are there volunteers sticking their hands inside and/or near the robot, let alone fanning the fire?
It seems to me like this reflects a lack of training for what to do in this situation. I would expect volunteers to be properly inducted about what to do in the event of an emergency such as this one. We've got 6 highly dangerous machines on the field and dozens more behind the curtain, all with massive batteries and electrical currents running through them, but it seems that no one knows what to do in the (quite likely) event that something goes wrong.
We're always saying that "Safety comes first", but it's events like this that make me question "are we really living up to that?"
It seems to me that they were searching for a dry chemical fire extinguisher apposed to a more common co2 fire extinguisher in order to keep parts of the robot salvageable.
The likelihood hood of the fire spreading was very low but, it did slightly concern me when they started to poke around inside of the robot. I was also expecting with that size flame that the bumper fabric would eventually catch on fire. Overall it just seemed foolish to poke around.
The reaction time could have definitely been better and this is a situation that volunteers should be better prepared for but the likelihood is pretty low.
A more accessible fire extinguisher is really all they need.
Jon Stratis
15-04-2016, 10:48
The overall reaction there seemed a bit less prepared than this one for Lake Superior a few years ago https://m.youtube.com/watch?v=YOd3mF1tvwk
Fast forward to the last 25 seconds, you can see immediate action behind the robot, and a fire extinguisher was out and ready on the field rather quickly!
My assumption is that the early poking around near the fire was an attempt to find and hit the main breaker, which is usually the first line of defense before they go to the extinguishers. Even when clearly labelled, the smoke can make it really hard to see the breaker. Blowing on the flames, however, was probably not the best plan.
Later, they popped the cover to make it easier to get the fire extinguisher onto the fire.
What are your concerns with a battery and fire?
It's moreso that a battery rupture could cause massive damage to the internals of the robot and any close components assuming the bellypan of the robot isn't fully enclosed.
It was stated that the fire was caused by a battery short with a live chassis. If a robot is on fire, 9 times out of 10 it will be because of a battery short or electrical fault. When a high amperage electrical circuit shorts, touching the metal chassis of the robot is a good way to become less alive.
Doug Frisk
15-04-2016, 11:14
I hate to be 'that person', but it has to be said.
Why did it take so long to grab a fire extinguisher that should have been ready at the field?
Because someone screwed up. As part of field setup, the fire extinguisher is removed from case 8 and put by the side of the field. I usually put it in front of the FMS at the center of the table or on the end of the table where the FTAs are hanging out.
It's quite apparent that extinguisher was still in case 8. I bet all the FTAs get an email this week.
My assumption is that the early poking around near the fire was an attempt to find and hit the main breaker, which is usually the first line of defense before they go to the extinguishers. Even when clearly labelled, the smoke can make it really hard to see the breaker. Blowing on the flames, however, was probably not the best plan.
Later, they popped the cover to make it easier to get the fire extinguisher onto the fire.
At this point, hitting the breaker won't solve the problem. The fire is started electrically, which means that something, somewhere has shorted. Not only does this trip the breaker/fuse, but will also discharge the battery if the breaker isn't fast enough to catch it.
Because someone screwed up. As part of field setup, the fire extinguisher is removed from case 8 and put by the side of the field. I usually put it in front of the FMS at the center of the table or on the end of the table where the FTAs are hanging out.
It's quite apparent that extinguisher was still in case 8. I bet all the FTAs get an email this week.
I'm aware that fire extinguishers are standard with the field, but the fact that it was neglected/forgotten by the field setup crew is what worries me.
Doug Frisk
15-04-2016, 11:18
It seems to me that they were searching for a dry chemical fire extinguisher apposed to a more common co2 fire extinguisher in order to keep parts of the robot salvageable.
The likelihood hood of the fire spreading was very low but, it did slightly concern me when they started to poke around inside of the robot. I was also expecting with that size flame that the bumper fabric would eventually catch on fire. Overall it just seemed foolish to poke around.
The reaction time could have definitely been better and this is a situation that volunteers should be better prepared for but the likelihood is pretty low.
A more accessible fire extinguisher is really all they need.
You have that backwards. A dry chemical extinguisher will coat the robot in dust and crap, CO2 leaves no residue making it easier to salvage bits.
I'd also call dry chemical much more common. C02 extinguishers are not typically found at your big box home store.
I've never actually seen a CO2 unit in a pit at an FRC event. I'm sure some teams have them, but dry chemical is pretty much what everyone has.
Kevin Sevcik
15-04-2016, 11:27
When a high amperage electrical circuit shorts, touching the metal chassis of the robot is a good way to become less alive.You might be badly burned if the frame is hot, but Ohm's law plus physiology says you're extremely unlikely to suffer any injury from the electricity itself. Wikipedia says (https://en.wikipedia.org/wiki/Electric_shock#Body_resistance) the resistance of wet skin is about 1 kOhms. So a 12V battery can push, say, 12 mA through you. The chart on that page notes that 12 mA of AC would be noticeable after a few seconds, but not deadly. It's important to note that DC is much less dangerous in this regard than AC. Unless you have wires connected directly to your heart[1], it takes a lot more DC to kill you than AC. If anyone is doubtful, I can hold a demonstration at Champs where I will bravely grab both terminals of a robot battery with either hand an suffer no ill effects.
Mind you there's plenty of other dangers from the battery, but they're all the fire, explosion, direct-short causing a welding arc sort.
TL;DR: The robot battery (and your car battery[2]) aren't going to electrocute you unless you stab a positive and negative wire from them into your heart. In which case you have plenty of other problems besides.
[1]I have before. It was weird.
[2]Had a dead battery once in college and got a jump from a doctor. He gave me the same electrocution warning. And he should know, because he's a doctor. I didn't argue because I wanted to get home.
If there was a short with the frame, does this mean the frame isolation test during inspection was not done?
Or would this happen even if the frame isolation test passed?
You might be badly burned if the frame is hot, but Ohm's law plus physiology says you're extremely unlikely to suffer any injury from the electricity itself. Wikipedia says (https://en.wikipedia.org/wiki/Electric_shock#Body_resistance) the resistance of wet skin is about 1 kOhms. So a 12V battery can push, say, 12 mA through you. The chart on that page notes that 12 mA of AC would be noticeable after a few seconds, but not deadly. It's important to note that DC is much less dangerous in this regard than AC. Unless you have wires connected directly to your heart[1], it takes a lot more DC to kill you than AC. If anyone is doubtful, I can hold a demonstration at Champs where I will bravely grab both terminals of a robot battery with either hand an suffer no ill effects.
Mind you there's plenty of other dangers from the battery, but they're all the fire, explosion, direct-short causing a welding arc sort.
TL;DR: The robot battery (and your car battery[2]) aren't going to electrocute you unless you stab a positive and negative wire from them into your heart. In which case you have plenty of other problems besides.
[1]I have before. It was weird.
[2]Had a dead battery once in college and got a jump from a doctor. He gave me the same electrocution warning. And he should know, because he's a doctor. I didn't argue because I wanted to get home.
That is a fair point, I should have thought a bit further before posting that last line. Looking back, I should have known a bit better[1]. Even so, I still stand that touching an electrically charged chassis is far from a good idea, even if it's just for peace-of-mind.
[1]I do a lot of work with electrical systems. Also got shocked by a 240 ~3-6A wall socket (for the second time) a few years ago, less than a fun experience.
If there was a short with the frame, does this mean the frame isolation test during inspection was not done?
Or would this happen even if the frame isolation test passed?
It is possible even if circuits are electrically isolated from the frame before the match. If the outer shielding of the wire (or a solder joint, or wire crimp) becomes damaged, it is very possible for it to contact the robot chassis. With the rigors of Stronghold, this is entirely within the realm of possibility.
Because someone screwed up. As part of field setup, the fire extinguisher is removed from case 8 and put by the side of the field. I usually put it in front of the FMS at the center of the table or on the end of the table where the FTAs are hanging out.
It's quite apparent that extinguisher was still in case 8. I bet all the FTAs get an email this week.
Unless it changed this season, Michigan does not use the standard road case system that the rest of the country uses. There is no case 8, just a toolbox and a tote or two with the non-tool items that are usually in case 8 (paper towel glass cleaner, etc). I'm actually not sure where the fire extinguisher is stored with Michigan fields.
Disclaimer: I didn't do any events in Michigan this year, so it may have changed.
If there was a short with the frame, does this mean the frame isolation test during inspection was not done?
Or would this happen even if the frame isolation test passed?
Or maybe it happened after inspection? Wire coming loose during a match?
Doug Frisk
15-04-2016, 11:55
Unless it changed this season, Michigan does not use the standard road case system that the rest of the country uses. There is no case 8, just a toolbox and a tote or two with the non-tool items that are usually in case 8 (paper towel glass cleaner, etc). I'm actually not sure where the fire extinguisher is stored with Michigan fields.
You make a good point. Though the extinguisher that came out eventually did look like the standard CO2 extinguisher packed with the regional fields.
Jon Stratis
15-04-2016, 12:02
If there was a short with the frame, does this mean the frame isolation test during inspection was not done?
Or would this happen even if the frame isolation test passed?
There's a lot of variables that can go into this, we can't say for certain if it was or was not done. It's possible it was done, but after the fact the insulation wore through. It's possible the frame short was intermittent, and only occurred when a wire was pinched in a mechanism a specific way. Or the team was doing a hasty repair and pinched a wire, or drilled through the insulation of a wire, or disconnected a motor and had the leads just dangling... There are any number of ways for the frame to short out either before or after a frame isolation test is done. And most of the time the frame isolation test is not performed during reinspection.
The reason we do the frame isolation test is to ensure that it takes two faults to cause a problem, not just one. Two faults obviously can still happen, but it's half as likely to happen as having one fault.
Calvin Hartley
15-04-2016, 16:50
Unless it changed this season, Michigan does not use the standard road case system that the rest of the country uses...
This is correct. (Though I am far from an authority on the topic, I've helped with a handful of event setup and teardowns.)
Even so, I still stand that touching an electrically charged chassis is far from a good idea, even if it's just for peace-of-mind...
You are aware your car has an electrically charged chassis? In the States we bond one leg of of our power to ground. Touching a charged chassis really isn't an issue. You don't want to use your body to complete the electrical circuit though. :]
Before using your extingisher on an electrical fire, you want to remove the inigtion source. Powdered fire extinguisher are messy. Never my first choice when other options are present.
Kevin Sevcik
15-04-2016, 18:18
Before using your extingisher on an electrical fire, you want to remove the inigtion source. Powdered fire extinguisher are messy. Never my first choice when other options are present.On the other hand, it's not like it's wise to be choosy beyond making sure the extinguisher is rated for your particular emergency.
"No, no, it's not going to spread that much. Let's wait and see if we can find the CO2 extinguisher..."
On the other hand, it's not like it's wise to be choosy beyond making sure the extinguisher is rated for your particular emergency.
"No, no, it's not going to spread that much. Let's wait and see if we can find the CO2 extinguisher..."
:) I cannot argue with that.
RoboChair
15-04-2016, 20:35
You make a good point. Though the extinguisher that came out eventually did look like the standard CO2 extinguisher packed with the regional fields.
That was definitely a dry chemical extinguisher, the CO2 kind would have dissipated much faster.
GaryVoshol
15-04-2016, 21:13
It is possible even if circuits are electrically isolated from the frame before the match. If the outer shielding of the wire (or a solder joint, or wire crimp) becomes damaged, it is very possible for it to contact the robot chassis. With the rigors of Stronghold, this is entirely within the realm of possibility.
From talking with some inspectors, that is the theory. The wire that caught fire went around a corner on a frame piece, and it is suspected that the insulation wore through because of robot movement.
In addition there were some other factors that may or may not have come into play. There may be a suggestion made on additional safety tests in inspection, or a change in wiring rules. I don't want to be more specific in case I am wrong.
Ben Wolsieffer
15-04-2016, 22:16
Thinking about this now, I realize that something like this could have happened to us. We had a almost unmeasurable frame short (later discovered to probably caused by a limit switch wire connected to a Talon SRX), which passed inspection because the resistance was higher than the required limit.
Last night and this morning we fried two Talon SRXes, and after a ton of troubleshooting we discovered that our LED ring power cable was being pinched and sometimes connecting positive to the frame. If the resistance was a little lower, I think the same thing could have happened to us. This appeared to be what destroyed the Talons.
James Juncker
15-04-2016, 23:56
First the test only requires the resistance across your frame to be above 3k ohms
Secondly from what I've learned from the team the short was caused by an old unused LED controller that became grounded to the already charged chassis, which led to the heat and fire
Thirdly the battery had been removed (or disconnected unsure) from the robot before being Taken off of the field after seen smoking on the field
And finally it was definitely a dry chemical extinguisher, the residue was swept and vacuumed up immediately following the incident, but multiple fire extinguishers were brought over to aid the fire, the first extinguisher had its safety pin break off when someone attempted to pull it rendering the extinguisher useless increasing the time it took to extinguish the flames.
I would like to congratulate the team however on successfully redoing most of their robot as well as getting their shooting back up and rinning! A great turn around for them after an incident like that. Go team!
2012 - member of 2834
2013 - electrical and mechanical function
2014 - driver, mechanical, and drivetrain
2015 - driver, drivetrain, mechanical knot tying
2016 - captain, mechanical, drivetrain
On the other hand, it's not like it's wise to be choosy beyond making sure the extinguisher is rated for your particular emergency.
"No, no, it's not going to spread that much. Let's wait and see if we can find the CO2 extinguisher..."
Look closely to the right of the robot in the video. A volunteer in a black shirt brought a CO2 extinguisher, and you can see him pull the pin, while another volunteer(white shirt, towards the back of the robot) was holding a dry chemical extinguisher. As others have said, the dry chemical was applied to the fire; you can see the guy in the white shirt aim and discharge his extinguisher.
...the first extinguisher had its safety pin break off when someone attempted to pull it rendering the extinguisher useless increasing the time it took to extinguish the flames.
This is probably why the dry chemical was used instead of CO2. After seeing the pin break, he decided it would be better to put it out with the powder than to let it burn to search for another CO2, if there was one. (Good idea.)
The overall reaction there seemed a bit less prepared than this one for Lake Superior a few years ago https://m.youtube.com/watch?v=YOd3mF1tvwk
Fast forward to the last 25 seconds, you can see immediate action behind the robot, and a fire extinguisher was out and ready on the field rather quickly!
There was also an "Injury Risk" sign next to the field. That exercise ball seemed to be an effective fire blanket!
I just found another video on YouTube from the field, recorded by YouTube user zenith828: https://m.youtube.com/watch?v=6QyrZZ3zIX4
Mike Schreiber
17-04-2016, 02:29
If there was a short with the frame, does this mean the frame isolation test during inspection was not done?
Or would this happen even if the frame isolation test passed?
I was the initial inspector on this robot. We did the isolated frame check and it passed with no issue. Wiring looked pretty clean actually.
jvriezen
17-04-2016, 08:13
wo faults obviously can still happen, but it's half as likely to happen as having one fault.
Is this true? If the likelihood of one fault occuring is 1/100, then the like likelihood of two at the same time is 1/10000.
See pic :)
Best Pit Prank ever!
I was the initial inspector on this robot. We did the isolated frame check and it passed with no issue. Wiring looked pretty clean actually.
As Mike said, the team had passed initial frame isolation check. They "dropped coms" earlier in the day, and after a few minutes, I went personally to check on them.
They said they had already found the issue, a loose Anderson connection on the battery as well as cleaned up the VRM input which had a possible whisker.
I verified those and was about to leave, but asked the team if this was the first time. They replied they had other drops at other events. I did a frame isolation check and found the positive side grounded to the frame. The team had LEDs, and it was one of the first items I asked about. The team told me they were disconnected*.
We then dug through the various areas and found a different item that would clear the frame isolation. Once cleared, I returned to the field. couple hours later, the fire occurred. I do not have first hand details of how the fire was handled as I was covering the other field, and a large group of people were gathered around, and I thought it was some robot with a typical sensor wire or PWM that had been pinched and smoked.
After the fire was out, the issue was diagnosed as a power cable that had previously fed the LEDs. It had a barrel connector that was disconnected earlier.
We suspect that the positive side frame grounding was intermittent and the barrel connector likely was exposed negative. This lead to the power cable catching fire when both sides grounded to the frame. It was relatively thin wire, so a short caused it to catch fire relatively quickly.
The team did an amazing job of keeping a positive attitude. They had a student lead the rebuild, and she did an amazing job of keeping students on task. One of the new Michigan LRIs working the event stayed with them to ensure it was rebuilt properly. After initial teardown, we inspected the remaining components and had the team not only change out most of the electronics, but most of the pneumatics as well. We were quite fortunate no one was injured, and the team only missed 1 match.
I know a report was sent in about the fire incident, and I plan to talk some more with other LRIs about possible ways of improving and/or mitigating future incidents.
*I missed asking the team exactly how they "disconnected" the LEDs. I did not see the barrel connector, and assumed the disconnection was done properly. I will be adding a more thorough line of questioning when "something is disconnected".
JimBryson
17-04-2016, 22:37
As lead engineering mentor for 3656, Dexter Dreadbots, I can confirm events are as Mike and Ike :) say above. I appreciate both of their help in inspecting and troubleshooting our robot.
Our failure analysis indicates that we had two shorts to our frame (usually you can assume that multiple failures won't happen at the same time--but as Ike says, there was foreshadowing, but we didn't catch the root cause). There was a 12V connection and a ground connection, both to the frame, leading to heating of narrow gauge wires and melting of the PVC insulation, and eventually to fire.
The ground connection was a no-longer used barrel connector wired on fine-gauge dual-extruded wire connected to a 20A breaker on the PDP (not so smart in hind-sight). The old wiring had not been removed, and the exposed barrel connector with GND external was in the robot, likely tucked away by a well-meaning team member. Excess length of wire was bundled together with a zip tie between the RoboRio and PDP. More on that later.
The disconnected LED power that Ike wonders about was our new wiring. The LEDs had been rewired (to another 20A breaker), but with powerpole connectors and GXL or TXL 20awg wire. These were unplugged while troubleshooting the ground fault. Checking wiring it seemed that pushing on the PCM bracket could make the ground fault come and go. A student rewired it, looking for "whiskers". The problem went away without a "smoking gun", and we assumed it was fixed. The unused barrel connector wires passed through the same area, so moving wires likely moved the barrel connector.
The 12v source was the battery anderson connector to the main breaker lug. The main breaker was mounted close (~1/4") to a vertical frame element. It is likely that with repeated battery installation and removals and a high impact game, that the lug may has spun slightly and have allowed current to the frame, by splitting the electrical tape or maybe the taping wasn't perfect. Difficult to tell after the fire. We could not recreate the 12v to the frame that Ike had seen, probably because we had unplugged the battery and moved the lug slightly while looking for the ground issue. Strain relief and more flexible wire for battery hookup would both be helpful with this, as well as additional clearance and better insulation.
I was confused how both could be happening at the same time, and assumed we were not understanding what Ike was saying. But we could not recreate the faults, and we were called to que.
During the match (see MIDC qual 36 video https://youtu.be/TfYDX4_zND8) our robot let out the smoke about 73s into the match. The driver hit the e-stop at 77s, and the FTA turned off the main breaker about 103s in. The robot appears to stop smoking. After the match was completed and the robot was on the cart around 250s after the match started (not on the same video) the video shot by Byting Bulldogs shows the active flames. Our drive coach had noticed that the RSL was back on about this time. This may be due to the 12v from the frame backfeeding into the PDP after all the insulation was burned off the paired wires.
It is likely that the short reoccurred after putting the robot on the cart, or that the heat had been slowly building. At some point the heat was enough to melt through a pneumatic line, adding oxygen to the "smoldering" insulation and allowing it to build to a full fire. Most of the fuel for the fire was from the PVC insulation on the LED wiring (coiled up for maximum heating and fuel potential) and pneumatic tubing. The RoboRio contributed some as well. Our standard for wiring is to use GXL or TXL wire, and the insulation on those wires scorched slightly, but it did not burn or melt.
Our students were great at repairing the robot. Many teams offered help and parts. We had students who could and did do the work, and had most of the parts ourselves. As a mentor, I was very proud to hand out tools, get spare parts and help clean up, but not to do the rebuilding. We moved the robot and tools outdoors to several tables because we would not all fit in the pit, and to allow the dispersion of the dry chemical residue. We decided to replace all the electronics and wire (not motor controllers though) and had to replace several plastic components and one pneumatic cylinder that may have been damaged.
It may look like a lot of time passed before the extinguisher was discharged, but heat damage was not extensive. We now have an extinguisher on our cart, and I am looking into procuring a CO2 one for less collateral damage. We have moved (and further insulated) the main breaker and are looking into designing and 3d printing an insulator box that would offer strain relief as well.
After 4 hours of rebuild we were on the practice field with a fully-functional bot having missed only one match, and are now on our way to St Loius!
Tom Line
18-04-2016, 01:54
As lead engineering mentor for 3656, Dexter Dreadbots, I can confirm events are as Mike and Ike :) say above. I appreciate both of their help in inspecting and troubleshooting our robot.
Our failure analysis indicates that we had two shorts to our frame (usually you can assume that multiple failures won't happen at the same time--but as Ike says, there was foreshadowing, but we didn't catch the root cause). There was a 12V connection and a ground connection, both to the frame, leading to heating of narrow gauge wires and melting of the PVC insulation, and eventually to fire.
The ground connection was a no-longer used barrel connector wired on fine-gauge dual-extruded wire connected to a 20A breaker on the PDP (not so smart in hind-sight). The old wiring had not been removed, and the exposed barrel connector with GND external was in the robot, likely tucked away by a well-meaning team member. Excess length of wire was bundled together with a zip tie between the RoboRio and PDP. More on that later.
The disconnected LED power that Ike wonders about was our new wiring. The LEDs had been rewired (to another 20A breaker), but with powerpole connectors and GXL or TXL 20awg wire. These were unplugged while troubleshooting the ground fault. Checking wiring it seemed that pushing on the PCM bracket could make the ground fault come and go. A student rewired it, looking for "whiskers". The problem went away without a "smoking gun", and we assumed it was fixed. The unused barrel connector wires passed through the same area, so moving wires likely moved the barrel connector.
The 12v source was the battery anderson connector to the main breaker lug. The main breaker was mounted close (~1/4") to a vertical frame element. It is likely that with repeated battery installation and removals and a high impact game, that the lug may has spun slightly and have allowed current to the frame, by splitting the electrical tape or maybe the taping wasn't perfect. Difficult to tell after the fire. We could not recreate the 12v to the frame that Ike had seen, probably because we had unplugged the battery and moved the lug slightly while looking for the ground issue. Strain relief and more flexible wire for battery hookup would both be helpful with this, as well as additional clearance and better insulation.
I was confused how both could be happening at the same time, and assumed we were not understanding what Ike was saying. But we could not recreate the faults, and we were called to que.
During the match (see MIDC qual 36 video https://youtu.be/TfYDX4_zND8) our robot let out the smoke about 73s into the match. The driver hit the e-stop at 77s, and the FTA turned off the main breaker about 103s in. The robot appears to stop smoking. After the match was completed and the robot was on the cart around 250s after the match started (not on the same video) the video shot by Byting Bulldogs shows the active flames. Our drive coach had noticed that the RSL was back on about this time. This may be due to the 12v from the frame backfeeding into the PDP after all the insulation was burned off the paired wires.
It is likely that the short reoccurred after putting the robot on the cart, or that the heat had been slowly building. At some point the heat was enough to melt through a pneumatic line, adding oxygen to the "smoldering" insulation and allowing it to build to a full fire. Most of the fuel for the fire was from the PVC insulation on the LED wiring (coiled up for maximum heating and fuel potential) and pneumatic tubing. The RoboRio contributed some as well. Our standard for wiring is to use GXL or TXL wire, and the insulation on those wires scorched slightly, but it did not burn or melt.
Our students were great at repairing the robot. Many teams offered help and parts. We had students who could and did do the work, and had most of the parts ourselves. As a mentor, I was very proud to hand out tools, get spare parts and help clean up, but not to do the rebuilding. We moved the robot and tools outdoors to several tables because we would not all fit in the pit, and to allow the dispersion of the dry chemical residue. We decided to replace all the electronics and wire (not motor controllers though) and had to replace several plastic components and one pneumatic cylinder that may have been damaged.
It may look like a lot of time passed before the extinguisher was discharged, but heat damage was not extensive. We now have an extinguisher on our cart, and I am looking into procuring a CO2 one for less collateral damage. We have moved (and further insulated) the main breaker and are looking into designing and 3d printing an insulator box that would offer strain relief as well.
After 4 hours of rebuild we were on the practice field with a fully-functional bot having missed only one match, and are now on our way to St Loius!
Congratulations guys. It was really inspiring to see you out by yourself on the practice field running that robot so quickly. See you in St. Louis!
Al Skierkiewicz
18-04-2016, 08:00
Jim,
If you could provide any pictures of the before location and the after location I think that would help teams going to Champs and for the future. I was getting reports from the inspection team and when I heard you had everything back together in four hours, I had to smile knowing that a good team overcame a devastating fault and came back with a vengeance. Good luck to the team next week.
Al
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.