Blown talon srx modules
 

Our team has blown 3 talon srx modules. We have been watching very carefully to try to understand how they got fried. The common element seems to be that they stop working when we move the motor connected to the talon manually. My theory is that the motor is generating a voltage that is backfeeding the talon and frying it. We believe at least one of the talons was fried when there was no 12v power on the talon.

We did verify that there is no short to ground or other kind of short. Each time the talon fried it happened on a different motor.

The motors are generating voltage when moved, and if there it's no load presented by the talon then the voltage can be very high.

Had anyone else seen this problem?

We did send the first two units back to crosstheroads electronics but have not heard back from them on the analyses. They did replace the first two.

We will send the third talon back for analyses as well.

The purpose of this thread us to try to understand why we are blowing talons, and to see if anyone else is having a similar problem.

Re: Blown talon srx modules
 

We have had one talon stop working. I know for a fact that it was connected to the battery backwards.

We have not had any other problems with talons though we haven't been pushing our robot around or back driving motors very much yet.

Hope this helps...

Edoga

Re: Blown talon srx modules
 

Thanks for the reply.

All of out talons that fried were previously working on the robot, so we are quite sure that none of them were hooked up backwards.

GregGarner

Re: Blown talon srx modules
 

Manually rotating the motors does generate current and can fry motor controllers. We have fried many Victors and Jaguars this way. We have adopted the following "rule" because of this:

Do not push the robot. Do not manually turn mechanisms. Pick up the robot or use a cart to move it.

Also, we've started adding connectors between the motors and the controllers. Among many other benefits, one is being able to disconnect the motor if we need to manually rotate the motor.

Re: Blown talon srx modules
 

We have used Victors for years and we have never had one fry because we moved the robot without using a cart.
You can actually see the status lights lite when we move the bot.

Re: Blown talon srx modules
 

Ditto that. Our kids have fun pushing them fast enough that the brake mode kicks on and tries to stop the robot.

Re: Blown talon srx modules
 

Can either of you offer an explanation as to why numerous teams have pushed numerous robots great distances with power on, or off, and have never suffered a blown motor controller? What about mechanisms (like the drive-train) that create the same back-current as they coast down?

If your theory cannot explain all present evidence, you may want to revise your theory. ;)

In all my years in FRC I have never blown a Victor. Jaguars, sure, never Victors though.

Re: Blown talon srx modules
 

Back in the IFI controller days, I used to jump start the RC to get it to power itself off the backup battery by pushing the robot so I could download code with out a battery. I've since learned this is a bad idea and in general we try to avoid pushing the robot quickly, though we haven't broken a speed controller because of this that I know of.

Re: Blown talon srx modules
 

Pushing a robot turns motors which generate voltage, and that voltage is passed through the motor controller's MOSFETs to the battery terminal which subsequently can power the robot. The current path from M+/M- to V+/V- in the motor controller is done through the MOSFET's reverse body diode and they act as a rectifier.

You don't want to pass current through these diodes since they're not intended to pass significant current (they're parasitic), and when they do pass current through them, the power dropped across them is the forward diode voltage times the current. This is why the Black Jaguar, Talon SR, and Talon SRX use synchronous rectification (SR) when driving the motor. SR turns the low side of the H-Bridge on during the "off" phase of the drive. Unfortunately, when pushing the robot motor controllers are not doing SR.

When testing Black Jaguars, the team used a test bench with two CIMs, the output shafts coupled to each other through an AM ToughBox. One CIM was driven from the motor controller and the other used as a generator with the leads connected across a switchable bank of 1 ohm power resistors. You'd be surprised at how much power you can draw out of a CIM when the shaft is turned.

Re: Blown talon srx modules
 

Turning a motor when it is unpowered does generate a voltage. That voltage is no different in principle from the back EMF that the motor generates when it is powered.

Unless you're backdriving the motor much faster than its free speed, I don't think you're going to create a situation where the speed controller is in any danger.

Me too!

Re: Blown talon srx modules
 

Back driving a motor on a Talon SRX, Victor SP or Talon SR will not cause it any harm. The back EMF that gets generate is no more harmful than the forward voltage that is applied during rectification.

I REALLY want this to be understood so teams do not needlessly adopt measures to prevent something that is not even a problem.

Re: Blown talon srx modules
 

After discussing this with Mike at CrossTheRoadsElectronics, he and I have concluded that we probably damaged these Talon SRX drives with ESD (ElectroStatic Discharge) when we were assembling them.

Mike took apart the first two that were destroyed and reported that multiple chips inside the device had been damaged, and the only mechanism that he thinks fits the evidence is ESD. This is clearly evidence against my previous theory.

We are now up to 4 blown talon SRX devices. The last Talon SRX to die was under power and moving a bag motor, so it is somewhat different than the other 3 that died. It is clearly evidence against my previous theory.

We think that a combination of extremely dry conditions (i.e. no humidity) and assembling in a carpeted room may have contributed.

We will now start start following standard ESD handling rules when assembling these devices. This will include humidifer, grounded anti-static mats, and getting the kids to wear anti-static wristbands when wiring the robot. We may also spray anti-static spray on the carpet where we do our assembly.

I do want to give CrossTheRoadsElectronics Kudos for being extremely supportive and willing to extensively discuss this issue with us. I feel that they are a very good company.

Hopefully we will get past this issue.

GregGarner
3612 The Gearhogs

Re: Blown talon srx modules
 

What do you mean by assembly? Just putting on connectors/pwm?

If so, I have a really hard time believing that ESD was the problem. Our team has used hundreds of Victors over the years, and has never heard of an ESD problem. We have used absolutely no ESD protection ever with the components, and many teams do the same. If the new controllers were super ESD sensitive, then many more people would complain.

Re: Blown talon srx modules
 

I would expect it to be similar to the amount of power you can draw out of the shaft of a CIM when you apply electrical power to the wires.

Re: Blown talon srx modules
 

The current theory is ESD on assembly.

If you are skeptical of this, then given the evidence, what is your theory?
I am certainly open to other theories, as my main goal is to figure this out and stop the bleeding!

Here is a quick summary:

1) 2 blown talon SRX devices when working on the robot, may or may not have been moving the motor (we just didn't notice exactly when they stopped working). These 2 were on different motors, and did not blow at the same time. When we replaced the talon we did not change any wiring or any motor, and the talons that were replaced are still working fine. The Talon SRX devices have no LED after they are blown, i.e. the CPU and the regulator were fried. These two were looked at by CrossTheRoadsElectronics and found multiple chips were fried (just nonworking and shorted, the tops were not blown off the chips).

2) 1 more blown Talon SRX, this one apparently went out while we were moving the motors manually, and it may have been powered or not powered on by the battery, not sure. Again the sympton is the LED does not come on at all. Changed the Talon, did not change anything else, replacement Talon is still functioning. This one has not yet been returned for post mortem.

3) Final Talon SRX blown when moving a bag motor. I will verify the symptoms (i.e. no LED) this evening, as I haven't seen it in person yet.
This one has not yet been returned for post mortem.

Note that we are using the PD board with circuit breakers, and we are using a standard FRC battery. This robot has been very carefully assembled, we did not connect up the Talon SRX modules backwards or get the battery polarity backwards. All connections were done with solder, covered by heatshrink (although we are switching back to high amperage connectors now because we suspect we may have to change out Talons in between matches if we can't solve this problem)



GregGarner
3612 Gearhogs

Re: Blown talon srx modules
 

If P2 is the maximum mechanical power you can draw from a CIM's output shaft for a given input electrical power P1,

and P3 is the maximum electrical power you can draw from a CIM whose output shaft is being driven with mechanical power P2,

then P3 is considerably less than P1.

Reason: motor losses work in both directions.

Re: Blown talon srx modules
 

Dry air, carpet, and electronics are not a good combination. I think your best remediation will be using an anti-static mat to cover the carpet. Wrist straps will help a lot, but I am confident that they will not always be used when appropriate.

Re: Blown talon srx modules
 

Were the chips that CTRE found to be damaged connected to "the outside world" or were they only connected "internally"? The ESD would have been amazingly powerful to cause damage past the first device it hits. ESD can cause mis-operation of a device when injected into the system while it is running but I doubt that you would be touching those connections while running your robot.

Perhaps CTRE may want to check their supply chain for counterfeit parts. We have had several instances at work where counterfeit parts sort of work but caused failures where there was no "abuse" of the system.

Re: Blown talon srx modules
 

I would be surprised if this was a component issue, if it was I imagine the problems would be much more wide spread.

Re: Blown talon srx modules
 

We have had two Talon SRX controllers drop from the Can network after working for a day or longer each and then complete turn up dead......no signal lights ..... Not on the network....: just dead with after being fully functional. I have not contacted CTRE yet but will.

Re: Blown talon srx modules
 

I'm sure we differ in opinions, but I've always felt that any piece of consumer electronic that can be permanently destroyed by ESD is, to put it bluntly, crap. Also, odds are not many teams even have ESD mats and wrist straps and even fewer have them connected correctly. A product that is designed to be put on FRC robots should take carpet, dry air, high school students, heck mentors, into consideration. It's pretty inexcusable for a product used in this environment to be affected, especially permanently, by ESD.

Re: Blown talon srx modules
 

We agree on that part. However, I don't consider Talons to be "consumer electronic" devices.

Re: Blown talon srx modules
 

They are, however, a device that are designed with this specific purpose in mind, which should require it to have even better ESD protection.

Re: Blown talon srx modules
 

And what "consumer electronics" come with bare wires to which you must connect power and or signalling cables?

Being aware of ESD and taking care not to zap electronic components is SOP and good things for students to learn. A better question is were they warned by the packing material or by a mentor? I don't remember seeing anything in the TalonSRX manual, a pretty big oversight. But all the electronics used in FIRST can be damaged by ESD (to some degree) - I warned our guys/girls just the other day to NOT touch the gyro board as it is really easy to zap and destroy.

Re: Blown talon srx modules
 

We also have a blown Talon SRX. During a practice session last weekend the talon stopped functioning. No lights no voltage on the output leads at all. We opened the talon and discovered that one of the capacitor leads to the circuit board was burned into. We soldered the lead back together but still does not work. Looks like something else on the board is fried. The only thing we think might have happened is the connectors on the leads from the talon to the motor were not correctly insulated and shorted to the aluminum frame. The beaker on the power distribution board was tripped.
We have ordered two more. It seems as though the new Talons are somewhat sensitive. Word of caution, make sure all connectors are secure and insulated properly and have a couple of spare talons on hand.

Re: Blown talon srx modules
 

Home Theater equipment, for one. But regardless, the wires didn't even have to be like that. It could have been some kid of female receptical (something like powerpole).

I'd like to see some high resolution pictures of the blown boards to see if there's anything in common.

Re: Blown talon srx modules
 

Better than What? Do you even know what the ESD rating is for the Talon SRX?
This would be an important thing for you to know before claiming it is not sufficient.

Re: Blown talon srx modules
 

Sensitive to what?


Have you contacted CTRE support regarding this issue?

Re: Blown talon srx modules
 

We have been working with Greg and will continue to work with him on this issue. This problem has never been reported or observed during Internal testing or during Beta testing. We will continue to work with him and any team that has an issue with a Talon SRX.

Due to the nature of symptoms and information provided during conversations with Greg, I believe the issue is unique to this particular robot configuration. Possibly related to the cabling strategies between the encoder and the data port.

Since this problem cannot be repeatedly reproduced, pinpointing the exact cause is difficult. If any team is experiencing a similar issue, please contact support@crosstheroadelectronics.com to report.

Re: Blown talon srx modules
 

Seeing as the data has not been released (that I have seen), of course I don't know what it is. I would expect it should be on the order of 15kV. Anything less than 8kV, I would say is probably insufficient for this application.

Re: Blown talon srx modules
 

So what happens if the ESD event is greater than 15 kV?

Re: Blown talon srx modules
 

Then it fails? I'm not sure what you're getting at.

Re: Blown talon srx modules
 

I have a new theory:

Last night it occurred to me that our team is probably doing one thing quite differently than any other team out there. We elected to make our own encoder/limit switch cables, rather than using the pre-made cables that come from Vex. The reason we did this is because our team is cash limited, and we could save a good chunk of change making our own cables.

Therefore we bought the .025 ribbon cable in bulk, and purchased the correct .050 spacing dual row IDC connectors for that cable. We then proceeded to make our own cable to go all the way from the Talon SRX out to the Encoder, which in some cases ended up being 8 feet long. We then terminated the ribbon cable with the correct 5 pin connector into the encoder.

Since the ribbon cable has all 10 conductors in it, this strategy implies that every signal on the Talon is being routed all the way out to the end of the 8 foot cable, even though there are some signals we didn't need. In particular, the 3.3V signal is connected to the ribbon cable. The 3.3V signal is not used by our encoder, and it is not terminated at the end of the cable. This effectively means that we have a 8 foot antenna on the 3.3V wire, which could pick up any signal in the area and connect it into the Talon 3.3V rail.

In addition, we put the motor wires into the same wiring duct with the encoder cables. This means that the current pulses in the motor wires could be inductively coupling into the encoder ribbon cable, and in particular the 3.3V line going straight to the Talon SRX CPU.

We have noticed that out of the 5 blown Talon SRX modules, at least 4 of them were on the longer encoder cables. We have not seen any Talon SRX module have a problem when it had a shorter encoder cable (i.e. The Talon SRX modules are physically closer to the drive motors so the cables are shorter. These closer Talon SRX modules have not failed).

We have seen the Talon SRX modules fail when functioning, and also when not functioning but we have been manually rotating the motors by manipulating the robot. Again the failures are only seen when there are long cables to the encoder.

My goal at this point is to shotgun the problem and try lots of different things at once to try to stop the problem. We have one week to bag and tag, and so it means we can't debug the problem in the standard way of changing one thing at a time.

I am going to do at least one more test before tearing apart the wiring. I will put a digital scope on the 3.3V line at the Talon and look and see if I can see any significant induced spikes on the 3.3V line when moving the motor under program control and under manual control, on the talon SRX with the longest attached ribbon cable. I will report back on what I see.

In order to try to stop the problem from happening, we intend to use shorter ribbon cable out of the Talon SRX, and then splice in a more robust wire to go on out to the Encoder. This means we will not be routing out all the unused signals from the Talon SRX to the rest of the robot, and in particular we will not route out the 3.3V line. Also, we will run the encoder cables in a separate wiring ducts, so there is physical separation between the encoder signals and the large current power wires for the motors.

Finally we do intend to follow industry standard ESD protocol as part of the shotgun approach.

I want to emphasize again that the guys at CrossTheRoadsElectronics have been very helpful, and are replacing the blown Talon SRX modules. In addition they are assisting us with trying to understand why this is happening. They have also sent us some cables and breakout boards to try, since that is what other teams are probably using.

GregGarner
Team 3612

Re: Blown talon srx modules
 

Speakers are outputs and muuuuuch easier to protect than any kind of input.

A connector for power and for CAN/PWM would have been nice but would require some sort of standardization for CAN/PWM and power interconnections.

Re: Blown talon srx modules
 

Interference is no longer an ESD problem.

Also, a wire that isn't terminated at one end isn't likely to result in much inductive coupling, and it will be much, much less than a wire that's terminated at both ends. Having it terminated at only one end will make inductive coupling almost disappear. There's no effective loop area if you don't have a loop!

You may have parasitic capacitive coupling though.

Re: Blown talon srx modules
 

The Talon SRX is tested to both EN 61000-6-1 and EN 6100-4-2, 8KV air gap and 4 kV direct contact. This is the standard that most consumer electronics are tested to.

We used the attached TVS diode in several places in the design. It is rated to >+/-30kV.

Re: Blown talon srx modules
 

I agree it should fail. But I would not suggest that the product is crap if it did fail because of this.

Re: Blown talon srx modules
 

Greg,

Could your custom cables be shorting adjacent wires and therefore be shorting things within the Talon? If misaligned, this could be a problem since they're insulation displacement connectors.

Moreover, if these cables are faulty, you'd continue to propagate failures.

Re: Blown talon srx modules
 

It can be very difficult to get useful information doing this. Your scope and it's probe can inject noise into the system since it acts like an antenna too. How you ground your scope probe to the system is critical to how clean signal you get. A lot of what you may see may be induced in the ground lead of the scope probe. You are probably better off putting a high-frequency current probe around the cable coming out of your Talon since the current probe will not be Gavanically connected to your system.

Secondly, the 3.3V line is probably a power supply line. The impedance looking into the Talon on that line will probably be quite low making it hard for any outside interference source to have a significant influence on the internal circuitry of the Talon. Basically the internal impedance of the 3.3V line forms a divider with the impedance coupling to your external noise source.


Yes.


Making a new cable assembly may make the problem go away if there is a short in your existing cable; i.e. the new one does not have the same fault. You can use the same ribbon cable material and the same IDC connector to make a cable with just the wires you need. The "more robust wire" will not be any more resistant to external interference. If the pins in the connector that you need are all adjacent, then rip the ribbon cable to have only the number of wires you need then crimp it into the IDC connector. If the wires you need are not adjacent, then crimp a width of ribbon cable that encompasses the wires you need, use a sharp knife to split out the wires you do need from the ones you don't need and cut off the ones you don't need near the connector.

You may want to twist the new cable assembly (2-4 twists per inch) to make it more resistant to external inductive noise.

Running the cable so that runs that are parallel to power wires are 2-3 inches away from the high power wires can also help reduce noise coupling.

Re: Blown talon srx modules
 

I'm not suggesting the Talon is crap. I'm suggesting it may not be due to ESD. It sounds like you've taken the proper precautions, but somehow, through the normal course of use, Talons are being blown; Not by one team, but by several. ESD should not kill products that are used in a manner in which they are intended to be used, if not also have tolerance for wiring mistakes (this is a competition full of people who make mistakes). My guess is something thermal is happening in combination with ESD. It looks like The protection diode is only rated up to 125 Celsius, which I think is normally fine. However, when put in an enclosed box like the SRX is and put inside of a robot with potentially not enough air flow to convection cool the fins, perhaps the diodes aren't able to offer the full protection they normally would. I don't have any temperature measurements to back this up, and I'm not exactly sure how you could accurately (and easily) measure the junction temperature while simulating the SRX enclosure.

That's really my best guess at the moment.

Re: Blown talon srx modules
 

When spinning the motors to feed power backward, you may have huge (kV range) voltage spikes from flyback from the motor.

The power distribution board current sensor is susceptible to damage from flyback voltage from motors, as our team has discovered the hard way.

This is speculation, but could a similar problem have affected this team? It seems strange that so many of the controller have failed...

Re: Blown talon srx modules
 

I doubt that a Talon would very exceed 100deg C, needless to say 125. I think they tend to stay fairly cool.

Re: Blown talon srx modules
 

The temperature of the case is different than the junction temperature.

Re: Blown talon srx modules
 

In normal operation, those transorbs have less than 100 nanoAmps flowing through them and are essentially out of circuit. The energy present in a standard ESD test, like the ones referred to in the datasheet linked by Mr. Copioli would not contain enough energy to cause the temperature of the transorb to rise that much. It would not pass those tests if it did.

Transorbs are a pretty rugged class of devices. I have severely abused similar devices by applying surge voltage test with much higher energy than the ESD test (several 100 x), without allowing the required cool down times and the transorbs worked properly up till the heat dissipated in them melted the solder and the part fell off the board. They continued to work after we soldered the parts back in the board.

Re: Blown talon srx modules
 

I'm not saying the temperature rose due to ESD. My guess is that the robot had been running prior to the event and heated up.

Re: Blown talon srx modules
 

This is an ESD issue, not related to thermal.

So far only two teams have reported this problem. Not several.

This is far from the truth. No amount of ESD precautions can guarantee protection from an ESD strike.

Re: Blown talon srx modules
 

It doesn't matter why the transorb got that hot. The point in my last paragraph was that these devices continue to work up to the point where we melted the solder, way beyond 125 C.

Why do you think that the Talon had heated up enough that the transorb was at or near 125 C?

Re: Blown talon srx modules
 

Double post

Re: Blown talon srx modules
 

During all the previous failures the Talon SRX modules were not warm at all. They were essentially at room temperature. We haven't noticed the Talon SRX modules getting warm even under fairly heavy loading. I attribute this to the extremely low on resistance of the H-Bridge inside the Talon, just not much power dissipation there. We also have the Talon's mounted to aluminum to keep them cool.

We are quite sure that the cables have no internal shorts or other faults, because we haven't replaced the cables and the replacement Talons connected to those cable are working properly.

I did hook up the scope last night, and I found that there is a large transient on the 3.3V wire at the Talon SRX. In particular, I do see large transients on the 3.3V line when running the motor, but I see no such transients when moving the motor manually. I think this is strong evidence that the 3.3V wire in our cable is coupling in noise from the motor wires. Again, we are possibly the only team making our own custom ribbon cables, so it is likely that most other teams aren't seeing this problem because they didn't run the 3.3V signal out to the end of a 8 foot cable in parallel with the motor wires.

I briefly looked at the 5V line and the transients seemed much lower, perhaps because there is a load on the 5V line in the form of the encoder.

Therefore we will now proceed to reworking the cables as previously described. I am hopeful that this will solve the problem. The breakout boards from CrossTheRoadElectronics just arrived (Thanks Mike!), so I should be able to get the team to make progress on this change tonight.

Thanks to everyone who is trying to help us solve this problem!

Here is the highest transient I saw on the 3.3V line at the Talon connector while running the motor. I think the peak voltage was probably around +-7V:


GregGarner
FRC GearHogs 3612

Re: Blown talon srx modules
 

Are you saying it's acceptable for your cell phone or your laptop to completely die from an ESD strike?

Re: Blown talon srx modules
 

Greg

Please post a photo showing how you have the scope probe connected to the Talon. The Ground clip of the probe should be connected to pin 10 of the Talon Analog Input connector. The wire for Ground clip of the probe forms a loop and will cause the probe to pick up noise from magnetic fields around the probe. Minimizing the area of that loop will minimize the spurious signals. The wire can be wrapped around the probe leaving the minimum length require to reach the Ground pin on the Talon Analog Input connector.

You can also connect the probe to the breakout board with no cables connected. If you still get noise, it is coming from inside the Talon or is being picked up "from the air" by your scope probe.

Please run the Talon with the same commands as used to get the scope shot but with the motor connected then disconnected.

You may want to twist the motor wires (about 2 twists per inch) and see if that reduces the noise.

Re: Blown talon srx modules
 

Yes.

If the strike's potential is higher than what the device is designed around, you should expect the device to fail.

Would you think your television design was unacceptable if it fried due to a lightning strike? I can almost guarantee it will die if this happens.

BTW lightning is ESD.

There are boundaries to every design. Just because you hit one of these boundaries does not make a product crap or unacceptable.

Re: Blown talon srx modules
 

You aren't comparing apples to apples here, the talon is not a consumer electronics product. It is a component in an electrical system. It is not unreasonable to expect the customer who purchased the product to take appropriate precautions during installation and in their application.

Re: Blown talon srx modules
 

Note that the wiring in this robot has been torn down, so I am unable to take any more pictures or scope shots until we get it rebuilt.

I can probably take some more scope shots when we get it rebuilt using the breakout boards in a day or two. I expect that we will so little to no noise on 3.3V rail since it will only have a short ribbon cable connected to it.

We are changing over to shielded wiring to go from the breakout board out to the encoders, so that should greatly help the 5V line to not pick up any noise, and of course the encoder cables will be run separately from the motor wires now.

I would like to re-emphasize that our original implementation was flawed, it is not good design practice to run signal wires in close proximity to high current motor wires for any significant distance. I also think that the idea of using this super tiny ribbon cable to go all the way out to the encoders was flawed, since this ribbon is just not very robust.

It is much better and more reliable to run the ribbon a short distance out of the talons and then use the breakout board to switch over to a more robust shielded cable to go the long distance to the encoder.

Hindsight is 20/20, but for any other teams out there that see this thread please think about this when designing and building your robot.

GregGarner
FRC3612 GearHogs

Re: Blown talon srx modules
 

Michael

It is expected by those who wrote the standards and those who follow them (customers and manufacturers), that applying levels beyond those called out in the standard can lead to failure of the product. I am not aware of any standards that require a product not fail with no limit on the level of the stimulus.

Re: Blown talon srx modules
 

Good news from Gearhogs Team 3612:

The shotgun approach worked, as our Talon SRX modules are all functioning nicely.

We bagged up our competition bot last night, fully functioning. We ran it quite a bit, so I am pretty confident that we solved the problem.

I haven't had a chance to look at any more waveforms.

Greg Garner