Blown talon srx modules

[GregGarner]

The current theory is ESD on assembly.

If you are skeptical of this, then given the evidence, what is your theory?
I am certainly open to other theories, as my main goal is to figure this out and stop the bleeding!

Here is a quick summary:

1) 2 blown talon SRX devices when working on the robot, may or may not have been moving the motor (we just didn't notice exactly when they stopped working). These 2 were on different motors, and did not blow at the same time. When we replaced the talon we did not change any wiring or any motor, and the talons that were replaced are still working fine. The Talon SRX devices have no LED after they are blown, i.e. the CPU and the regulator were fried. These two were looked at by CrossTheRoadsElectronics and found multiple chips were fried (just nonworking and shorted, the tops were not blown off the chips).

2) 1 more blown Talon SRX, this one apparently went out while we were moving the motors manually, and it may have been powered or not powered on by the battery, not sure. Again the sympton is the LED does not come on at all. Changed the Talon, did not change anything else, replacement Talon is still functioning. This one has not yet been returned for post mortem.

3) Final Talon SRX blown when moving a bag motor. I will verify the symptoms (i.e. no LED) this evening, as I haven't seen it in person yet.
This one has not yet been returned for post mortem.

Note that we are using the PD board with circuit breakers, and we are using a standard FRC battery. This robot has been very carefully assembled, we did not connect up the Talon SRX modules backwards or get the battery polarity backwards. All connections were done with solder, covered by heatshrink (although we are switching back to high amperage connectors now because we suspect we may have to change out Talons in between matches if we can't solve this problem)



GregGarner
3612 Gearhogs

[philso]

Originally Posted by GregGarner

The current theory is ESD on assembly.

If you are skeptical of this, then given the evidence, what is your theory?
I am certainly open to other theories, as my main goal is to figure this out and stop the bleeding!

Here is a quick summary:

1) 2 blown talon SRX devices when working on the robot, may or may not have been moving the motor (we just didn't notice exactly when they stopped working). These 2 were on different motors, and did not blow at the same time. When we replaced the talon we did not change any wiring or any motor, and the talons that were replaced are still working fine. The Talon SRX devices have no LED after they are blown, i.e. the CPU and the regulator were fried. These two were looked at by CrossTheRoadsElectronics and found multiple chips were fried (just nonworking and shorted, the tops were not blown off the chips).

2) 1 more blown Talon SRX, this one apparently went out while we were moving the motors manually, and it may have been powered or not powered on by the battery, not sure. Again the sympton is the LED does not come on at all. Changed the Talon, did not change anything else, replacement Talon is still functioning. This one has not yet been returned for post mortem.

3) Final Talon SRX blown when moving a bag motor. I will verify the symptoms (i.e. no LED) this evening, as I haven't seen it in person yet.
This one has not yet been returned for post mortem.

Note that we are using the PD board with circuit breakers, and we are using a standard FRC battery. This robot has been very carefully assembled, we did not connect up the Talon SRX modules backwards or get the battery polarity backwards. All connections were done with solder, covered by heatshrink (although we are switching back to high amperage connectors now because we suspect we may have to change out Talons in between matches if we can't solve this problem)



GregGarner
3612 Gearhogs

Were the chips that CTRE found to be damaged connected to "the outside world" or were they only connected "internally"? The ESD would have been amazingly powerful to cause damage past the first device it hits. ESD can cause mis-operation of a device when injected into the system while it is running but I doubt that you would be touching those connections while running your robot.

Perhaps CTRE may want to check their supply chain for counterfeit parts. We have had several instances at work where counterfeit parts sort of work but caused failures where there was no "abuse" of the system.

[JamesBrown]

Originally Posted by philso

Perhaps CTRE may want to check their supply chain for counterfeit parts. We have had several instances at work where counterfeit parts sort of work but caused failures where there was no "abuse" of the system.

I would be surprised if this was a component issue, if it was I imagine the problems would be much more wide spread.

[mklinker]

Originally Posted by JamesBrown

I would be surprised if this was a component issue, if it was I imagine the problems would be much more wide spread.

We have had two Talon SRX controllers drop from the Can network after working for a day or longer each and then complete turn up dead......no signal lights ..... Not on the network....: just dead with after being fully functional. I have not contacted CTRE yet but will.

[GregGarner]

I have a new theory:

Last night it occurred to me that our team is probably doing one thing quite differently than any other team out there. We elected to make our own encoder/limit switch cables, rather than using the pre-made cables that come from Vex. The reason we did this is because our team is cash limited, and we could save a good chunk of change making our own cables.

Therefore we bought the .025 ribbon cable in bulk, and purchased the correct .050 spacing dual row IDC connectors for that cable. We then proceeded to make our own cable to go all the way from the Talon SRX out to the Encoder, which in some cases ended up being 8 feet long. We then terminated the ribbon cable with the correct 5 pin connector into the encoder.

Since the ribbon cable has all 10 conductors in it, this strategy implies that every signal on the Talon is being routed all the way out to the end of the 8 foot cable, even though there are some signals we didn't need. In particular, the 3.3V signal is connected to the ribbon cable. The 3.3V signal is not used by our encoder, and it is not terminated at the end of the cable. This effectively means that we have a 8 foot antenna on the 3.3V wire, which could pick up any signal in the area and connect it into the Talon 3.3V rail.

In addition, we put the motor wires into the same wiring duct with the encoder cables. This means that the current pulses in the motor wires could be inductively coupling into the encoder ribbon cable, and in particular the 3.3V line going straight to the Talon SRX CPU.

We have noticed that out of the 5 blown Talon SRX modules, at least 4 of them were on the longer encoder cables. We have not seen any Talon SRX module have a problem when it had a shorter encoder cable (i.e. The Talon SRX modules are physically closer to the drive motors so the cables are shorter. These closer Talon SRX modules have not failed).

We have seen the Talon SRX modules fail when functioning, and also when not functioning but we have been manually rotating the motors by manipulating the robot. Again the failures are only seen when there are long cables to the encoder.

My goal at this point is to shotgun the problem and try lots of different things at once to try to stop the problem. We have one week to bag and tag, and so it means we can't debug the problem in the standard way of changing one thing at a time.

I am going to do at least one more test before tearing apart the wiring. I will put a digital scope on the 3.3V line at the Talon and look and see if I can see any significant induced spikes on the 3.3V line when moving the motor under program control and under manual control, on the talon SRX with the longest attached ribbon cable. I will report back on what I see.

In order to try to stop the problem from happening, we intend to use shorter ribbon cable out of the Talon SRX, and then splice in a more robust wire to go on out to the Encoder. This means we will not be routing out all the unused signals from the Talon SRX to the rest of the robot, and in particular we will not route out the 3.3V line. Also, we will run the encoder cables in a separate wiring ducts, so there is physical separation between the encoder signals and the large current power wires for the motors.

Finally we do intend to follow industry standard ESD protocol as part of the shotgun approach.

I want to emphasize again that the guys at CrossTheRoadsElectronics have been very helpful, and are replacing the blown Talon SRX modules. In addition they are assisting us with trying to understand why this is happening. They have also sent us some cables and breakout boards to try, since that is what other teams are probably using.

GregGarner
Team 3612