Problems with the Voltage Regulation Module VRM

[juchong]

Hey everyone,

We had an issue with one of the VRMs on our test robot last week where the radio would intermittently drop the driver station, but maintain communication with our debug PC. We're using LabVIEW on our robots this year.

The DS would drop while idling, moving, enabling, etc. which seemed odd. In past years teams would have issues with communication dropping because of a loose cable, a bad connection, bad soldering, etc. but I can't ever recall seeing a bad voltage regulation module at a competition.

I swapped radios and still experienced the same issue, so I decided to remove the VRM and replace it with last year's module (remember, this is a test robot).

I finally had a chance to test the module in the lab and found quite a few issues with it. I used a high power decade resistor box as my load, a calibrated Keithley power supply set at 12.3V/1.5A as my source, a calibrated Fluke multimeter to verify the resistance of the decade resistor box, and a calibrated Tek oscilloscope to capture the waveforms.

TEST 1: 5 V / 2A Output
Load: 3 ohms (measured) - Calculated current: 1.6A

• If the load is attached to the power supply while powering up the VRM, the VRM will not start up.
• Powering on the VRM with no load, then attaching the load will allow the VRM to stabilize.
• Ripple on the supply was hideous; about 5.4V p-p. This would definitely explain why our radio was rebooting.

Screenshot of test 1's waveform. Note that the oscilloscope is set to AC coupling.

Test 2: 5 V / 2 A Output
Load: 4 ohms (measured) - Calculated current: 1.25A

• If the load is attached to the power supply while powering up the VRM, the VRM will not start up. Same as the previous test.
• Powering on the VRM with no load, then attaching the load will allow the VRM to stabilize.
• Ripple on the supply was hideous; about 4.2V p-p. This would definitely explain why our radio was rebooting. It's also important to note that the maximum rated power draw from the radio is 1A, not too far from these test conditions.

Screenshot of test 2's waveform. Note that the oscilloscope is set to AC coupling.

Test 3: 5 V / 2 A Output
Load: 10 ohms (measured) - Calculated current: 500mA

• The VRM will stabilize after about 5 seconds and correctly output 5V on both supplies, but it will draw 1.5A (current limited) from the 12.3V supply. That's one heck of a start-up transient.
• Powering on the VRM with no load, then attaching the load will allow the VRM to stabilize quickly.
• Ripple on the output is still bad. 1.1V p-p. At such a low power rating, this is unacceptable. Ripple should be on the order of 50 ~ 100mV maximum for a bad supply, 10 ~ 30mV for a high quality supply.

Screenshot of test 3's waveform. Note that the oscilloscope is set to AC coupling.

Conclusion
It looks like there is something seriously wrong with this module. I'm planning on taking similar data on another VRM in order to compare the results. I really hope we don't have to walk around with oscilloscopes at champs!

[rich2202]

Originally Posted by juchong

a calibrated Keithley power supply set at 12.3V/1.5A as my source

I think you need more than 1.5A as your source. One output port alone on the VRM is rated for 12V/2A peak, and 12V/1.5A continuous.

Part of your power-up problem may be not having enough input current.

Given the minimum wiring size of 24 AWG, they are expecting peak draw of 3.5 amps.

Assuming maximum continuous power draw on all ports, that would be 0.5 and 1.5 amps at 12 volts, and 0.5 amps and 1.5 amps at 5 volts. That means a minimum draw of 2.83 amps at 12 volts continuous, and peak draw minimum of 3.55 amps at 12 volts. Assuming 80% efficiency, that would be 3.5 amps input to deliver the maximum rated continuous power draw.

[Jared]

Can you try the test again with a 12 volt FRC legal battery? The power distribution board used in previous years had power supplies that didn't play well when fed with another power supply and could be causing your problem.

[Alan Anderson]

Originally Posted by Jared

Can you try the test again with a 12 volt FRC legal battery? The power distribution board used in previous years had power supplies that didn't play well when fed with another power supply and could be causing your problem.

That would definitely be an issue with a switching power supply, but the Keithleys with which I am familiar are linear supplies.

It's certainly possible that you have a faulty VRM. Out of several thousand in the hands of robotics teams at the moment, I would not be surprised if one doesn't work properly.

[philso]

Please post a photo showing how you have connected your scope probe and it's ground clip to the VRM.

[juchong]

Originally Posted by rich2202

I think you need more than 1.5A as your source. One output port alone on the VRM is rated for 12V/2A peak, and 12V/1.5A continuous.

Part of your power-up problem may be not having enough input current.

Given the minimum wiring size of 24 AWG, they are expecting peak draw of 3.5 amps.

Assuming maximum continuous power draw on all ports, that would be 0.5 and 1.5 amps at 12 volts, and 0.5 amps and 1.5 amps at 5 volts. That means a minimum draw of 2.83 amps at 12 volts continuous, and peak draw minimum of 3.55 amps at 12 volts. Assuming 80% efficiency, that would be 3.5 amps input to deliver the maximum rated continuous power draw.

That's great and all, but I'm only testing the 5V / 2A supply rail. If we do a simple power calculation (P = V * I) we find that for this specific power rail, the maximum power consumption is only 10W. Now, if we calculate the maximum theoretical input power, the result (assuming a 12V input voltage and a maximum supply current of 1.5A) is 18W. This means that if I were to fully load the output of the 5V /2A rail on the VRM, it would consume (I = P / V) 0.83A on the input supply. Now, if we take into account 80% efficiency (which is terrible for a switching regulator), that puts the total current draw from the source supply at just about 1A. 1A < 1.5A, so we're good!

If I wanted to test maximum voltage regulation on all channels simultaneously, then yes, you are correct, I would require a much larger supply. In this case, I'm not testing that. I'm only testing the 5V / 2A output, and a 1.5A supply is more than adequate!

Originally Posted by Alan Anderson

That would definitely be an issue with a switching power supply, but the Keithleys with which I am familiar are linear supplies.

It's certainly possible that you have a faulty VRM. Out of several thousand in the hands of robotics teams at the moment, I would not be surprised if one doesn't work properly.

Agreed Alan, this is a lab-grade Keithley. I took some more data on another VRM and produced similar results. These measurements are starting to worry me. I'll post pictures of my setup tomorrow, but I can assure you that I've performed supply characterization tests like this before. 2 failures in ~3000 modules is terrible QC!

[rich2202]

Originally Posted by juchong

That's great and all, but I'm only testing the 5V / 2A supply rail.

My point is: If you don't let the VRM draw max current during power-up, then observations about how long it takes the power to stabilize output voltage is not indicative of normal operation. I'm guessing there are circuits in the VRM that need to be energized in order to provide stable output power. That start-up load can easily exceed 1.5 amps.

The VRM has a minimum input voltage of 5v. So, in order to supply 12 volts, it must first step-up the power to 13 or more volts, and then it can regulate it down to 12 volts. I know you are not using the 12v circuit in your test, but the VRM doesn't know that. So, it has to energize that circuit during power on.

In your Test 1, you say that when a 1.6 amp load is attached, the VRM will not start up. It may be because as soon as it starts to energize the circuit, it is drained away by the load. There is not enough excess power in the circuit that is allowed to build up to the operating level.

In Test 3, you finally have a situation where the the amount of power draining out is slow enough that the amount of power remaining in the VRM can build up to operating levels in 5 seconds. Now, whether that is normal or not, I don't know.

[ozrien]

Hey Juan,
I'd expect to be some ripple if you are drawing near the max continuous current (1.5A VRM User's Guide).

I didn't really understand this...

We had an issue with one of the VRMs on our test robot last week where the radio would intermittently drop the driver station, but maintain communication with our debug PC. We're using LabVIEW on our robots this year.

...so I grabbed my mecanum kit bot which uses a DAP1522(VRM1 - 20A fused) and M1011 Camera (powered from VRM2 - WAGO20A snapaction) and drove it around for a while. Ran a full battery down to the point where DS was in constant "Voltage Brownout" meanwhile watching the Axis Camera on the laptop running the DS. I didn't seem to lose camera or the router the entire time. Which seems like a good test of the power supply of the router and camera (each a VRM). Also ping times to the RIO seemed ok so I'm not sure this if the VRM has anything to due with your disconnects.

Plus Beta teams spent a lot of time beating up that aspect of the robot.

Can you explain the original problem a bit more?

[juchong]

Originally Posted by ozrien

Hey Juan,
I'd expect to be some ripple if you are drawing near the max continuous current (1.5A VRM User's Guide).

I didn't really understand this...

...so I grabbed my mecanum kit bot which uses a DAP1522(VRM1 - 20A fused) and M1011 Camera (powered from VRM2 - WAGO20A snapaction) and drove it around for a while. Ran a full battery down to the point where DS was in constant "Voltage Brownout" meanwhile watching the Axis Camera on the laptop running the DS. I didn't seem to lose camera or the router the entire time. Which seems like a good test of the power supply of the router and camera (each a VRM). Also ping times to the RIO seemed ok so I'm not sure this if the VRM has anything to due with your disconnects.

Plus Beta teams spent a lot of time beating up that aspect of the robot.

Can you explain the original problem a bit more?

Why would you guys label the maximum <peak> output current on the VRM instead of <continuous>? Seems a bit.. misleading? Why would you list peak current for one output (2A) and continuous (500mA) for the other? Very bad practice!

The original problem was that communication was dropping even though battery voltage did not fluctuate. Something as simple as enabling the robot would cause the DS to lose communication. I went through several steps to try and diagnose the issue (swapping radios, verifying connections, loading default code, etc.) and determined that the VRM was causing issues. We're continuing to use last year's regulators while we look into the issue with lab equipment.

Do you guys have any characterization data on these VRMs? Something a bit more concrete than anecdotal evidence? Also, why design the regulator's switching frequency to be in the audible range?

I'm about to head off to work and perform proper line/load regulation tests. I hope to post the results later today.

[MrRoboSteve]

Originally Posted by juchong

The original problem was that communication was dropping even though battery voltage did not fluctuate. Something as simple as enabling the robot would cause the DS to lose communication. I went through several steps to try and diagnose the issue (swapping radios, verifying connections, loading default code, etc.) and determined that the VRM was causing issues. We're continuing to use last year's regulators while we look into the issue with lab equipment.

This sounds a lot like the DS wifi driver issues that other teams have been reporting. Does the problem occur if you have a hardwired connection to the robot?

[juchong]

Everyone,

I finally had a chance to finish up the report. Take a look at it and let me know if you guys find any issues with it! LINK

TL;DR: The module did not perform well in any of the three load tests. There are high levels of noise present on the regulator's output and output ripple is also above 500 mV for most tests. I also make the conclusion that radio performance may be affected by the module's poor performance.

I wish the rules allowed a bit more flexibility with regards to power regulation. Maybe something to consider for next year?

If I can get my hands on some additional test hardware, I'm going to try and perform tests using a D-Link radio. Look for an update sometime this weekend.