We noticed today that our test bed was browning out at current loads well below those that should be problematic.
Upon further testing, and reviewing telemetry logs from earlier characterization, we determined that the robot’s net resistance before the motor controllers (including internal resistance of the battery) was in the ballpark of .06 ohms - this seems to be about a factor of 3 higher than it ought to be.
Unfortunately, we find ourselves unable to easily diagnose where in the circuit the problem lies, as none of our multimeters can measure resistances below .1 ohms, and thus are redoing much of the wiring in a broad attempt to hopefully fix whatever is causing the extra resistance. This is extremely time-consuming and inefficient.
What strategies do teams use to diagnose problems such as these? Multimeters with high-resolution resistance measuring seem to be quite expensive, and I’m not sure how they play with resistance of the leads (which is undoubtedly significant).
It browns out when drawing more than around 100-120 amperes. As the brownout condition depends only on the voltage seen by the RIO, which depends almost entirely on the voltage sag caused by the system resistance, it does not much matter what exact robot behavior causes this current draw.
You’ve been at this for a while, but I’ve got to ask …
The terminations at the battery, main breaker, and PDP are all tight; i.e. terminals don’t move.
Are you using compression lugs on the heavy cables, are you using 6 AWG?
Have you given the terminations a good strong pull? If you can pull them apart, they will fail.
What type of motors and quantity?
What type of drives?
Have you run your robot up on a test stand or blocks?
Currents look okay in this condition?
The typical wire resistance we work with is difficult to measure with a multi-meter. The resistance measurement is better performed using a power supply. Set the power supply current limit to a known value - perhaps 1 Amp - connect the cable between the supply and the return of the power supply output and measure the Voltage drop in the cable. Hope this makes sense.
Eli, if you’re browning out, you’re dropping ~6V somewhere. Just use the current draw and ohms law to find the problem. Run the robot into a wall so you’re pulling a bit less than browning out. Put a voltmeter in volts mode and measure volts at the battery terminal to make sure it’s not a dud battery. Then start measuring volts along a wire. Start with measuring volts from the + battery terminal to the + PDP. Ditto for the - leg. Whichever one is dropping the most is probably the problem. Measure that leg and start working your way up the wire. When the volts drop goes from huge to normal, you’ve found the problem termination or component. Stabbing probes into a wire through the insulation is a great way to measure across a termination.
Turn your robot off. Check that ALL lugs with bolted connections (battery, breaker and PDP input) do not rotate even a little bit. Do a pull test on ALL crimped and clamped connections. Have a second, detail oriented person repeat these tests. Don’t waste your time getting out your DVM or any other diagnostic tools until you have found and fixed ALL the bad connections.
Another possible way is to use a thermal imaging camera (e.g. http://www.fluke.com/fluke/sgen/solutions/ti/thermal-imaging-electrical-systems). They’re quite expensive, but you may be able to find a place that rents them or (better yet) would be willing to donate some time to drop by with one (e.g. a local electrical contractor).
Can you compare the voltage the RoboRIO is seeing to the voltage the Talons are seeing? That would at least tell you if there’s a bad connection somewhere in the 6awg wire between the battery and PDP or if the problem lies between the PDP and the RoboRIO.
In the OP’s case there is likely to be multiple bad connections. The thermal camera will not find the loose connections that carry low currents. Those connections are just as important. He is better off doing a complete check of ALL the connections in his robot. Otherwise, one or more of the loose connections that cannot be found with the thermal camera will make itself evident at an inconvenient time.
This was the first thing we did. Nothing was obviously shoddy, but we re-did every connection anyway. We’re not sure if it fixed the problem yet, as we haven’t had time to test, but it took a while.
We’re almost certain the problem is not between the PDP and the RoboRIO, since our method for determining the resistance (measuring current and voltage drop at motors) does not care about the RIO circuitry at all, and the calculated value of the resistance is consistent with the brownout behavior we’re seeing.
It would be nice if there were a reasonable electrical check for individual connections; if they’re not obviously mechanically-bad, what is the logical next step for testing? Simply re-doing every major connection is hugely wasteful, it seems.
What is the difference between voltage read by a (e.g.) Battery Beak and the voltage as-read on the Driver’s Station?
Are you sure it’s not a bad crimp in the SB-50 that goes to the PDP? Popping the plastic housing off takes just a few seconds.
Even though software and calculations “say” there isn’t an issue before the RIO, have you physically checked everything?
Are you using mini-Anderson blocks between the motor controllers and the Talons? If so, might there have been a bad crimp somewhere?
What do the logs of the individual PDP ports say on the DS graphs? You may be able to use that at as a starting point, particularly for master/follower relationships.
Negligible; but this wouldn’t necessarily reveal a bad connection, as the current draw of a robot at rest is very small.
Are you sure it’s not a bad crimp in the SB-50 that goes to the PDP? Popping the plastic housing off takes just a few seconds.
We looked at these, and didn’t see any immediately-obvious problems. We have re-crimped them anyway.
Even though software and calculations “say” there isn’t an issue before the RIO, have you physically checked everything?
Yes.
Are you using mini-Anderson blocks between the motor controllers and the Talons? If so, might there have been a bad crimp somewhere?
What do the logs of the individual PDP ports say on the DS graphs? You may be able to use that at as a starting point, particularly for master/follower relationships.
The Talons all report similar numbers.
We are almost certain the source of the resistance is somewhere in the battery-PDP circuit. It just would be nice to have a reasonable method of figuring out precisely where.
Have you tried replacing the PDP? We’ve had a few failed PDPs that exhibited various weird pass-thrus and measurements like you describe.
Some were team caused, some were indeterminate root cause.
If you have some wire flexibility, the easiest way we have seen is to just temporarily lay one on top of the other and move the main power, CAN, and the few critical outputs you need up to the new one. If that works, add further branch circuits up to the new or replacement PDP. This can help you isolate the problem to a particular branch or the PDP itself.
Apparently, we had omitted the lock washers. I know lock washers are useless for actually locking, but I suppose they may do a fair job of improving electrical connectivity. We will be sure to add them.
I’ve seen contractors install washers and lockwashers between lugs and the stud they are mounted to. My thought was that this would account for a significant increase in resistance even though the individual terminations were secure.
All were at or below 20 amps each and the total was at or below 120, but the RIO still browned out when the draw was 120 amps, with a fresh and beak-tested battery.
Just be careful to not let the two probes cross and cause a short. Most of the high current components you would be taking measurements on are big enough that it is very difficult to cause a short this way. Poke the probes into the back of the SB connectors and into the connectors in the PDP. This naturally isolates your probe.
Loose connections are a manufacturing process issue. Proper training for how to make the connections (bolted and crimped) should reduce the incidence of bad connections to near zero. The person making the connections should perform a check after making each and every connection. Having a different person perform a QC check afterward should catch the few bad connections that remain. It should be possible to have good connections the first time. If the manufacturing process is not good enough, re-doing the connections is not a guarantee that the resulting connections will be good.
After ensuring you have good connections, it is possible you have a bad component or bad connector. While you are pulling high current;[ul]
[li]Measure the voltage on the battery terminals and compare it to the voltage on the inputs to the PDP.[/li][li]If the battery terminal voltage looks unusually low, swap in a known good, and charged, battery.[/li][li]If there is a significant difference between the battery terminal voltage and the PDP input voltage[LIST][/li][li]Measure the voltage across the input and output of the breaker[/li][li]Measure the voltage across the two positive contacts and the two negative contacts of the SB connector.[/ul] [/li][li]If there is little difference between the battery terminal voltage and the PDP input voltage[ul][/li][li]Measure the voltage from the positive input to the positive terminal at several of the PDP outputs[/li][*]Measure the voltage from the positive input to the positive terminal at several of the PDP outputs[/ul] [/LIST]