Blown talon srx modules

[GregGarner]

After discussing this with Mike at CrossTheRoadsElectronics, he and I have concluded that we probably damaged these Talon SRX drives with ESD (ElectroStatic Discharge) when we were assembling them.

Mike took apart the first two that were destroyed and reported that multiple chips inside the device had been damaged, and the only mechanism that he thinks fits the evidence is ESD. This is clearly evidence against my previous theory.

We are now up to 4 blown talon SRX devices. The last Talon SRX to die was under power and moving a bag motor, so it is somewhat different than the other 3 that died. It is clearly evidence against my previous theory.

We think that a combination of extremely dry conditions (i.e. no humidity) and assembling in a carpeted room may have contributed.

We will now start start following standard ESD handling rules when assembling these devices. This will include humidifer, grounded anti-static mats, and getting the kids to wear anti-static wristbands when wiring the robot. We may also spray anti-static spray on the carpet where we do our assembly.

I do want to give CrossTheRoadsElectronics Kudos for being extremely supportive and willing to extensively discuss this issue with us. I feel that they are a very good company.

Hopefully we will get past this issue.

GregGarner
3612 The Gearhogs

[magnets]

Originally Posted by GregGarner

After discussing this with Mike at CrossTheRoadsElectronics, he and I have concluded that we probably damaged these Talon SRX drives with ESD (ElectroStatic Discharge) when we were assembling them.

What do you mean by assembly? Just putting on connectors/pwm?

If so, I have a really hard time believing that ESD was the problem. Our team has used hundreds of Victors over the years, and has never heard of an ESD problem. We have used absolutely no ESD protection ever with the components, and many teams do the same. If the new controllers were super ESD sensitive, then many more people would complain.

[GregGarner]

The current theory is ESD on assembly.

If you are skeptical of this, then given the evidence, what is your theory?
I am certainly open to other theories, as my main goal is to figure this out and stop the bleeding!

Here is a quick summary:

1) 2 blown talon SRX devices when working on the robot, may or may not have been moving the motor (we just didn't notice exactly when they stopped working). These 2 were on different motors, and did not blow at the same time. When we replaced the talon we did not change any wiring or any motor, and the talons that were replaced are still working fine. The Talon SRX devices have no LED after they are blown, i.e. the CPU and the regulator were fried. These two were looked at by CrossTheRoadsElectronics and found multiple chips were fried (just nonworking and shorted, the tops were not blown off the chips).

2) 1 more blown Talon SRX, this one apparently went out while we were moving the motors manually, and it may have been powered or not powered on by the battery, not sure. Again the sympton is the LED does not come on at all. Changed the Talon, did not change anything else, replacement Talon is still functioning. This one has not yet been returned for post mortem.

3) Final Talon SRX blown when moving a bag motor. I will verify the symptoms (i.e. no LED) this evening, as I haven't seen it in person yet.
This one has not yet been returned for post mortem.

Note that we are using the PD board with circuit breakers, and we are using a standard FRC battery. This robot has been very carefully assembled, we did not connect up the Talon SRX modules backwards or get the battery polarity backwards. All connections were done with solder, covered by heatshrink (although we are switching back to high amperage connectors now because we suspect we may have to change out Talons in between matches if we can't solve this problem)



GregGarner
3612 Gearhogs

[philso]

Originally Posted by GregGarner

The current theory is ESD on assembly.

If you are skeptical of this, then given the evidence, what is your theory?
I am certainly open to other theories, as my main goal is to figure this out and stop the bleeding!

Here is a quick summary:

1) 2 blown talon SRX devices when working on the robot, may or may not have been moving the motor (we just didn't notice exactly when they stopped working). These 2 were on different motors, and did not blow at the same time. When we replaced the talon we did not change any wiring or any motor, and the talons that were replaced are still working fine. The Talon SRX devices have no LED after they are blown, i.e. the CPU and the regulator were fried. These two were looked at by CrossTheRoadsElectronics and found multiple chips were fried (just nonworking and shorted, the tops were not blown off the chips).

2) 1 more blown Talon SRX, this one apparently went out while we were moving the motors manually, and it may have been powered or not powered on by the battery, not sure. Again the sympton is the LED does not come on at all. Changed the Talon, did not change anything else, replacement Talon is still functioning. This one has not yet been returned for post mortem.

3) Final Talon SRX blown when moving a bag motor. I will verify the symptoms (i.e. no LED) this evening, as I haven't seen it in person yet.
This one has not yet been returned for post mortem.

Note that we are using the PD board with circuit breakers, and we are using a standard FRC battery. This robot has been very carefully assembled, we did not connect up the Talon SRX modules backwards or get the battery polarity backwards. All connections were done with solder, covered by heatshrink (although we are switching back to high amperage connectors now because we suspect we may have to change out Talons in between matches if we can't solve this problem)



GregGarner
3612 Gearhogs

Were the chips that CTRE found to be damaged connected to "the outside world" or were they only connected "internally"? The ESD would have been amazingly powerful to cause damage past the first device it hits. ESD can cause mis-operation of a device when injected into the system while it is running but I doubt that you would be touching those connections while running your robot.

Perhaps CTRE may want to check their supply chain for counterfeit parts. We have had several instances at work where counterfeit parts sort of work but caused failures where there was no "abuse" of the system.

[JamesBrown]

Originally Posted by philso

Perhaps CTRE may want to check their supply chain for counterfeit parts. We have had several instances at work where counterfeit parts sort of work but caused failures where there was no "abuse" of the system.

I would be surprised if this was a component issue, if it was I imagine the problems would be much more wide spread.

[mklinker]

Originally Posted by JamesBrown

I would be surprised if this was a component issue, if it was I imagine the problems would be much more wide spread.

We have had two Talon SRX controllers drop from the Can network after working for a day or longer each and then complete turn up dead......no signal lights ..... Not on the network....: just dead with after being fully functional. I have not contacted CTRE yet but will.

[GregGarner]

I have a new theory:

Last night it occurred to me that our team is probably doing one thing quite differently than any other team out there. We elected to make our own encoder/limit switch cables, rather than using the pre-made cables that come from Vex. The reason we did this is because our team is cash limited, and we could save a good chunk of change making our own cables.

Therefore we bought the .025 ribbon cable in bulk, and purchased the correct .050 spacing dual row IDC connectors for that cable. We then proceeded to make our own cable to go all the way from the Talon SRX out to the Encoder, which in some cases ended up being 8 feet long. We then terminated the ribbon cable with the correct 5 pin connector into the encoder.

Since the ribbon cable has all 10 conductors in it, this strategy implies that every signal on the Talon is being routed all the way out to the end of the 8 foot cable, even though there are some signals we didn't need. In particular, the 3.3V signal is connected to the ribbon cable. The 3.3V signal is not used by our encoder, and it is not terminated at the end of the cable. This effectively means that we have a 8 foot antenna on the 3.3V wire, which could pick up any signal in the area and connect it into the Talon 3.3V rail.

In addition, we put the motor wires into the same wiring duct with the encoder cables. This means that the current pulses in the motor wires could be inductively coupling into the encoder ribbon cable, and in particular the 3.3V line going straight to the Talon SRX CPU.

We have noticed that out of the 5 blown Talon SRX modules, at least 4 of them were on the longer encoder cables. We have not seen any Talon SRX module have a problem when it had a shorter encoder cable (i.e. The Talon SRX modules are physically closer to the drive motors so the cables are shorter. These closer Talon SRX modules have not failed).

We have seen the Talon SRX modules fail when functioning, and also when not functioning but we have been manually rotating the motors by manipulating the robot. Again the failures are only seen when there are long cables to the encoder.

My goal at this point is to shotgun the problem and try lots of different things at once to try to stop the problem. We have one week to bag and tag, and so it means we can't debug the problem in the standard way of changing one thing at a time.

I am going to do at least one more test before tearing apart the wiring. I will put a digital scope on the 3.3V line at the Talon and look and see if I can see any significant induced spikes on the 3.3V line when moving the motor under program control and under manual control, on the talon SRX with the longest attached ribbon cable. I will report back on what I see.

In order to try to stop the problem from happening, we intend to use shorter ribbon cable out of the Talon SRX, and then splice in a more robust wire to go on out to the Encoder. This means we will not be routing out all the unused signals from the Talon SRX to the rest of the robot, and in particular we will not route out the 3.3V line. Also, we will run the encoder cables in a separate wiring ducts, so there is physical separation between the encoder signals and the large current power wires for the motors.

Finally we do intend to follow industry standard ESD protocol as part of the shotgun approach.

I want to emphasize again that the guys at CrossTheRoadsElectronics have been very helpful, and are replacing the blown Talon SRX modules. In addition they are assisting us with trying to understand why this is happening. They have also sent us some cables and breakout boards to try, since that is what other teams are probably using.

GregGarner
Team 3612

[Alan Anderson]

Originally Posted by GregGarner

We think that a combination of extremely dry conditions (i.e. no humidity) and assembling in a carpeted room may have contributed.

Dry air, carpet, and electronics are not a good combination. I think your best remediation will be using an anti-static mat to cover the carpet. Wrist straps will help a lot, but I am confident that they will not always be used when appropriate.

[Michael Hill]

Originally Posted by Alan Anderson

Dry air, carpet, and electronics are not a good combination. I think your best remediation will be using an anti-static mat to cover the carpet. Wrist straps will help a lot, but I am confident that they will not always be used when appropriate.

I'm sure we differ in opinions, but I've always felt that any piece of consumer electronic that can be permanently destroyed by ESD is, to put it bluntly, crap. Also, odds are not many teams even have ESD mats and wrist straps and even fewer have them connected correctly. A product that is designed to be put on FRC robots should take carpet, dry air, high school students, heck mentors, into consideration. It's pretty inexcusable for a product used in this environment to be affected, especially permanently, by ESD.

[Alan Anderson]

Originally Posted by Michael Hill

I'm sure we differ in opinions, but I've always felt that any piece of consumer electronic that can be permanently destroyed by ESD is, to put it bluntly, crap.

We agree on that part. However, I don't consider Talons to be "consumer electronic" devices.

[Michael Hill]

Originally Posted by Alan Anderson

We agree on that part. However, I don't consider Talons to be "consumer electronic" devices.

They are, however, a device that are designed with this specific purpose in mind, which should require it to have even better ESD protection.

[Mike Copioli]

Originally Posted by Michael Hill

They are, however, a device that are designed with this specific purpose in mind, which should require it to have even better ESD protection.

Better than What? Do you even know what the ESD rating is for the Talon SRX?
This would be an important thing for you to know before claiming it is not sufficient.

[Michael Hill]

Originally Posted by Mike Copioli

Better than What? Do you even know what the ESD rating is for the Talon SRX?
This would be an important thing for you to know before claiming it is not sufficient.

Seeing as the data has not been released (that I have seen), of course I don't know what it is. I would expect it should be on the order of 15kV. Anything less than 8kV, I would say is probably insufficient for this application.

[Mike Copioli]

Originally Posted by Michael Hill

Seeing as the data has not been released (that I have seen), of course I don't know what it is. I would expect it should be on the order of 15kV. Anything less than 8kV, I would say is probably insufficient for this application.

So what happens if the ESD event is greater than 15 kV?

[Michael Hill]

Originally Posted by Mike Copioli

So what happens if the ESD event is greater than 15 kV?

Then it fails? I'm not sure what you're getting at.