Brownout behavior - alternative design goals

[jhersh]

Originally Posted by Joe Johnson

Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing.

That's the result of a poor behavior of the FMS. It should be improved next year to not cause auto to restart.

[Joe Johnson]

Originally Posted by Greg McKaskle

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

<snip>
Insults ... I've been called worse, but they aren't so productive.

<snip>
Greg McKaskle

Greg,

Was I being hyperbolic with the "criminal" comments? sure. Calling it "really dumb"? No, not at all. If anything I would use stronger language.

I am not a EE by training but I have years and years of robotics experience both inside and outside of FIRST, as a hobbyist and as a robotic professional. These years of experience have taught me that you just don't scrimp when it comes to protecting the power supplies that keep your processors/coprocessors, sensors, and communications link alive.

That is a general rule. When it comes to FIRST FRC control system, where we know there is a battery that can be easily overwhelmed in terms of amps it can source compared to the amps requested by the motors allowed, the designers should take even more precautions when it comes to protecting control system from the inevitable low battery conditions. FIRST can put as many "buyer beware" signs up as it wants. It doesn't absolve them from the responsibility to provide a control system worthy of the name.

Based on the results I have seen using the RoboRIO and the discussion about the trade offs people made during the design, it is obvious to me that FIRST (and NI) did not take power management as seriously as they should have when as they were designing the new control system.

All the defense of 7V Brownout is missing the point. As are discussions about what last year's system would have done. This was a chance to design a system from scratch.

Yes, once you decide that it is okay for the RoboRIO to reboot at 4.5V and that you are not going to keep the 5V and 3.3V rails alive that it is okay to power them off the 6V line, then yes, you eventually get to a Brownout Voltage of 7V. Sure, a brownout is better than a reboot, but can we all agree that few brownouts are better and no brownouts are best?

For all that, I really don't mind that the PWMs are disabled at a certain voltage. Do I wish it were a bit lower? Sure, but I bow to the reality that ultimately it is better to have the motors turned off than to lose more critical thinking and sensing functions. I said brownouts but I was really talking about the larger issue of preserving higher brain function down to a voltage sufficiently low that it never happens.

The original sin in all this is that FIRST didn't take power management on critical systems as seriously as it should (as evidenced by the initial choice to not to even protect the 5V and 3.3V rails which power sensors and co-processors).

Yes, I am calling FIRST (and NI) out here. This could have and should have been a complete non-issue. It was foreseeable and it was avoidable.

I have a question that I will pose to every person that defends the current status quo: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

Saving a few bucks by not providing this feature on a control system that costs literally 100s of dollars that 1000s of teams will use to control robots that each team will invest many 1000s of hours and many 1000s of dollars making is... well... I don't want to be called out for making insults again so I'll leave it to the readers to decide what they would call it once they understand how easily and inexpensively this feature could have been provided.

Sincerely,

Dr. Joe J.

*I am not 100% what the 6V supply powers. If it's just the servo power, we could I would be willing let the 6V go -- servos going dark are no worse than motors turning off.

[Greg McKaskle]

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

I'm sure my post came off more defensive than I intended, but then again, a rebuttal of almost any sort easily tends in that direction. I think your questions and goals make sense. It will be interesting to see what other's would have done. If there is information that would make the alternate design exercise easier or more factual, I'll do what I can to get the info to you. In that spirit, here are the links to the specs, user manual, etc.

Here is a page from team 358 that did some pretty extensive independent testing of the entire system and details what happens as the voltage does the limbo. They took the roboRIO to under 4V before it rebooted and the radio under 3.5. Not your 3.3V goal, but in the neighborhood. And as documented and designed, it is staged, which introduces the brownout condition. Some things start to disappear at about 6.8. During alpha I believe it was often rounded to 7V.
Beta team 358 Testing of Power Management

Here is a direct link to the documentation of the power management stages.
roboRIO User Manual and the details are on page 6 and 7.

Just to round out the power discussion, the other requirement, the one that was given a higher priority, was to allow for any pin to short to any other pin with no damage. We regularly rake a screwdriver across exposed pins to demo this. The power tab of the DS will show you which rail is shorted. So detection and protection against over voltage and shorts was deemed critical. The system will also tolerate being plugged into the older 24V PD without damage. It won't operate, but isn't damaged and indicates the issue via LEDs. By the way, having USB cables with exposed ground shield on a robot with high current 12V exposed connectors is also challenging.

Oh, and 6V is used to power servos.

Greg McKaskle

[Joe Johnson]

Originally Posted by Greg McKaskle

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

<snip>

Greg McKaskle

Greg,
As to the fact that the actual RoboRIO blackout voltage was show to be 3.5V in one case, while I want to say good things about this, I just can't. The Spec you link to says 4.5V, I suppose that is what it was designed to be. One particular example showing a lower actual drop out voltage doesn't change that. Specs are about what happens to the worst built RoboRIO on its worst day. Am I happy that one particular data point is suggestive that there is design margin in that 4.5V? Sure. Do I wish that we were talking about a designed blackout voltage of 3.3V (or lower -- recall that as an ME my opening ask was for the RoboRIO and other key equipment stay alive down to 1V and was only talked off the ledge by my EE friends) and were talking about actual blackout voltages in the sub 3V range? Yes, yes I do.

BUT the RoboRIO reboot voltage is only one piece of the puzzle. The control system is a SYSTEM. Its nice to have a RoboRIO not rebooting but that is not enough. I need to have radio communications alive and well. I need to have my incremental encoders not lose their positions. I need to have my MEMS IMU keep its position for a the whole match (I'm looking at you navX). I need to have my onboard Beaglebone coprocessor not reboot during a match. I need to have sensors doing their thing. ...

It was FIRST's task to think on a system level and provide a system that just works.

As to the other things that FIRST/NI did right, I am not disputing this. There are a ton of things that FIRST/NI did really well. Protection against reversed and misrouted power/ground IS important and, as far as I know, the RoboRIO does a good job of this. Nice job on that front.

But that doesn't free the FIRST from the responsibility to manage power effectively. On this front, FIRST has a lot to answer for.

And now I will finish with the question I am promised to ask until it gets answered: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

I have a pretty good idea what the answer is but I would like to hear those who made the decision not to provide give their estimates.

Dr. Joe J.

[Greg McKaskle]

OK. Time for me to make another post that will come off sounding more defensive that I intend for it to.

The link to the user manual and spec sheet of the roboRIO was intended to clarify the design and spec'ed behavior with low input power. The link to team 358's results were to intended to show how an alpha team tested the more complete system that included USB devices, rail power, VRM, etc. I'm not trying to sell you a used car or use a single team's testing to fool anyone. I'm giving the info I know of for the roboRIO and system elements that were delivered. That may help you or others on this alternate design quest.

As for the other background on short protection -- I described the additional protection mechanisms so alternate designs would be apples-to-apples.

As I described in my background, I don't have the chops to do the current or alternate power supply design. But I do volunteer as a team mentor and CSA, so that I can see first-hand how the elements are used and how they perform. From that perspective, as a customer and support engineer, I do not see the need for the 1V or even 3.3V input requirement.

Greg McKaskle

[tStano]

Originally Posted by Jon Stratis

I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

Can anyone explain why the backup battery went away? I've wasn't on a team in IFI days, but I maintain a robot with an IFI control system. To me, this seems like the best possible solution to the problem.



Also, I feel like if you set CAN low and PWM to zero and turn relays off, you kill all the motors, so unless your battery is literally useless, any other power you can draw(unless something's shorting) will be plenty to keep you above 4.5v. Even if you're using all the servos and sensors you possible can. I feel like it still would be a trivial software(unless its on ROM) change to not make the roborio stop doing 5v and 3.3v rail stuff down to at the very least 5v.

Another quite simple option would be allowing a team to supply a regulator(which fits some sort of inspection standard) to feed the roboRIO.

[Al Skierkiewicz]

Sorry I didn't weigh in on this earlier...
The original PD and cRio voltage dropout was lifted from automotive industry papers on electrical devices on vehicles. (One of those is a Daimler/Chrysler paper) In those recommendations, devices should be reliable down to 6 volts and are classed by importance based on the functionality of the device. (used only during cranking, used only during engine run, etc.)
The original control system wanted to be better than that knowing that FRC is a very harsh environment. The power supplies on the side car dropped out at around 5.5 volts. When that level was reached (if the input drop was long enough that the capacitors on the side car finally dropped below 5.5 volts) the sidecar would cease providing PWM output. In terms of a fully charged battery and ideal conditions, a robot load current of about 600 amps would drop enough voltage across the internal resistance of the battery to achieve that. (That is in keeping with the manufacturer's maximum current spec for our batteries.) If we add to the equation, the real world voltage drops across wiring, main breaker and typical wiring, that level can be achieved with a load current of approx. 500 amps (or less depending on robot design). The power supply for the PD ceased functioning at approx. 4.5 volts. It is pretty easy to see that the big CIM motors at 131 amps stall could easily approach the dropout specs with only a four motor drive. (In reality, with good wiring technique, a CIM motor in stall, driven by a motor controller, can draw between 116 and 120 amps.) It appears that the RoboRio spec and brownout behavior are consistent with those electrical considerations. Remember that long wire runs, poor crimp and termination techniques will degrade the available voltage at the RoboRio and other systems.
As to the backup battery on the old IFI controller, that was a design choice made by IFI to overcome the brownout conditions at the time. Remember that the controller was an old technology (compared to today) and would take a while to boot up and start running code. To keep the controller up and running, a circuit that switched the CPU (and attached circuitry) to battery supply during brownouts was a good choice.

[crake]

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

[slibert]

Originally Posted by crake

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

Thanks, Chris. As the founder of Kauai Labs, I have perhaps a unique perspective on this, as a result of developing/supporting the navX MXP IMU. Key goals included an easy-to-use, plug-n-play product useful to both upper-tier teams as well as beginning teams. Our focus is to keep improving things til we reach those goals.

The MXP connector was a great idea that really helped enable plug-n-play; however, this particular brownout issue negatively impacts these goals.

The primary reason is that the navX MXP (and certain other sensors, for sure) needs to manage state; in the case of the navX MXP there's some non-trivial run-time calibration state that's key. As noted by others, MXP products (Revduino) and other non-MXP products (e.g, the Beaglebone) are going to encounter similar issues.

In this case the specific issue is the 5 and 3.3V user rail supply rails on the MXP connector; voltage on these is not maintained in the case of a stage 2 brownout. The navX MXP has an onboard LDO that regulates the 5V MXP rail down to 3.3V to power a 32-bit ARM microcontroller; the board consumes a max of 250 mA.

Thanks to Joe H., who I met at CVR and pointed out the boost/buck reg attached to the USB port. Fortunately the navX MXP has a separate USB-based power supply on the navX MXP and the logic to cleanly switch between supplies w/out glitch. That's documented in our best practices now.

But that really isn't plug-n-play, to have this extra USB cable connecting the RoboRio to the navX MXP secondary power supply - just to avoid the brownout.

So the recommendation (OK, plea) is to maintain the power to the 5 and 3.3V MXP user rail during stage 2 brownout.

If that is not possible, we (MXP developers) need recommendations on how to best deal with this issue. Additionally, I think continuing the status quo on this will have the effect of diminishing the potential of the MXP interface. We are seeing tremendous growth in digital "smart" sensors/ICs w/onboard processing and state, and would like to help make them available to the FIRST community. I can confidently say that the robot of 3 years from now will have an even greater need than today.

[IKE]

Originally Posted by Al Skierkiewicz

Sorry I didn't weigh in on this earlier...
...snip.... Remember that long wire runs, poor crimp and termination techniques will degrade the available voltage at the RoboRio and other systems.
...snip...

Most of the brownouts I encountered in an LRI role highlighted non-conformance to one, but usually more than one of the practices Al outlined. The most common scenario was a KOP chassis with 4 CIMS, but 6" wheels with relatively poorly crimped heavy gauge wires to battery/PD/Mainbreaker or all 3. I also saw a few with damage on the AB-50 connectors due to charging with Aligaotr clips. All of these problems were in conjunction to only running a couple batteries all weekend (thus very little recharge time).

The brownouts were frustrating for the teams, but weren't terribly hard to diagnose. Only the loose terminals and charging were easy to fix. Talking teams out of 6" wheels are re-gearing the drivetrains was pretty tough.

For those teams, I am not really sure any back-up anything would really help much as it would just let them abuse a battery even more without us finding out they have a problem. I do think it would be worth discussing some back-up means for those that were just getting into some of the other scenarios that have been discussed. It was just not my experience at most competitions I supported.

[tkeil]

Hello. Power electronics engineer here. Somebody directed me to this thread and I thought I'd step in for a minute.

Several of the mentors as well as NI folks have already done a great job explaining this limitation and linking to helpful documentation for diagnosis and solutions. I don't have much to add on this front, but I would like to talk about what goes into selecting an appropriate input voltage from a technical perspective.

It's very easy for someone to be critical and say 'why doesn't it just work perfectly down to <X> volts?!' where <X> is some arbitrarily low voltage that some individual expects to never hit in their particular system implementation. The reason why is that this is a complex embedded controller and there are many trade-offs. In this case the main trade-off is thermal dissipation.

The roboRIO regulates 45W of power maximum. This is a lot for a small, plastic, convection cooled device with no open vents. The power supplies were a major concern for this design since with 45W input they account for a large percentage of the internally dissipated power.

Some of the best power supplies in the world can get >95% efficiency but these are nearly all power supplies with a fixed input voltage and a fixed output voltage. As soon as the power supply has to work over a wide input range, that efficiency falls because the power supply can no longer work at its most efficient operating point all the time. The roboRIO supports >2x input voltage variation. We can get better than 90% efficiency with this wide input range but the wider it is, the worse the losses get.

roboRIO is designed specifically for the FIRST competition, but it leverages many sections of circuits from other embedded controllers. It's designed by the same team that makes our professional cSeries controllers. You should think of roboRIO as a professional embedded controller with a protective cushion around it.

Fully around 1/3 of the circuity in the roboRIO is to protect the device from accidental damage and make it easy for the users to diagnose problems. Our highest priorities of course were preventing failures that permanently damage the device. I bring this up because some of these protections (for example reverse voltage protection) have a required minimum voltage and lie in the power path. These circuits are often leveraged from other designs and while they could be changed, it's costly to design and verify.

Some of these protection devices need to be in the direct current paths inside the device. This becomes a huge problem when your currents get large.

Let's look at a quick hypothetical where the input voltage minimum would be 4.5V:

The roboRIO needs to regulate 45W. If the input voltage drops, the current gets higher. Resistive losses are calculated as P=i^2*R. Consider that the current into the input connector of the roboRIO @ 7Vin is ~6.42A. If the minimum input voltage had been 4.5V that current would be ~10A. When you square those terms thats about a 2.5x increase in resistive losses through the whole input section of the power distribution path and switches in the power supplies.

Before you even get to the first power supply in the design, the power has to go through the input connector, the fuses, the common-mode choke, more noise filtering, reverse voltage and reverse current protection, and finally an input current sense shunt. The losses in all these devices would be 2.5x greater in this fictional example of allowing full power down to 4.5V.

Many (most?) of these devices would also have to be larger to support the current and thermal dissipation. For instance, 10A is far over the rating of the input voltage connectors. [For a laugh you can consider the device operating down to 1V in which case the current would be 45A and the input connector terminals would be around the size of an AA battery each.]

Anyway, when you combine the extra losses in the main input supplies with the extra losses in the resistive elements (and the increased component sizes) you will quickly lose your form factor and ability to use a plastic housing. If we had to go to a metal housing and possibly one with heat spreaders, you're talking about increasing the total device COGs by roughly 30% - 50% and increasing the form factor and weight.

Input power and voltage range is a big deal. Myself and others at NI think about it a lot and then over and over again for each new design. As you guys all know as good engineers and scientifically minded folks, everything's a trade off.

Thanks for your time volunteering with the students.

Tim

[Bryan Herbst]

Originally Posted by Joe Johnson

Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot.

We discovered this behavior with some teams at 10,000 lakes this year as well- when a robot is disabled and subsequently re-enabled during autonomous, autonomous is restarted.

In this scenario, you really only have two options code-wise: either restart auto or continue running the disabled loop until teleop starts. Picking up where the code left off in auto isn't a viable option unless your team explicitly programs a recovery mechanism.

Whether or not restarting or staying disabled is the right choice is a fair point for discussion, though my guess is that since recovery is a relatively viable option, the decision to restart auto was intentional. Perhaps this behavior could also be better publicized.

I'm also not sure why a brownout triggers disabled mode (I would think it could continue attempting to run during a brownout).

[Greg McKaskle]

Your intuition is correct. There is no need to disable for comms loss or brownout and once that is corrected, the double auto should be a thing of the past.

Greg McKaskle

[Alan Anderson]

Originally Posted by Tanis

I'm also not sure why a brownout triggers disabled mode (I would think it could continue attempting to run during a brownout).

The whole point of a brownout is to disable things so the load on the battery is reduced.

[Pault]

Originally Posted by Alan Anderson

The whole point of a brownout is to disable things so the load on the battery is reduced.

The issue that 246 and other teams experienced was not that the brownouts were disabling "things," but that in reaction to the brownout the FMS would command the robot to disable. This is much different, because the robot does not see it as just a standard brownout, but as if someone manually disabled and then re-enabled autonomous (or tele-op), and will therefore restart autonomous (or tele-op).

If teams have any fear of this happening at their offseason events, I would recommend putting a flag into your code that will only allow autonomous to initialize once. That is what we did, and it prevented this bug from breaking our robot a second time.