Brownout behavior - alternative design goals

[MrRoboSteve]

In another thread:

Originally Posted by Joe Johnson

*don't get me started about FIRST's decision to allow the NI folks to allow the RoboRIO loose its mind at just under 7V -- this is something close to criminal. Really... this was a really dumb design direction given the customer base and the intended use of the RoboRIO.

FWIW, this year I occasionally saw* robots with low battery enter brownout. These were always teams with bad/poorly charged batteries. For robots that rebooted, I think all of them had a wiring related root cause. Not to say that it didn't happen, but I don't think I ever saw it.

The purpose of this thread is to try to reverse engineer the requirements around the brownout feature, and discuss alternative requirements that might lead to a different outcome.

Here's what I think the requirements were for the current feature:

1. Provide as much telemetry from the robot as possible during low voltage states.
2. Avoid roboRIO reboots during matches.
   
3. Telemetry should show when bad robot behavior is caused by low voltage.

There are two sources of information on the brownout feature that have conflicting information.

http://wpilib.screenstepslive.com/s/...anual-id=24166

Page 7 of http://www.ni.com/pdf/manuals/374474a.pdf

* as FTAA at events with a total of 120 robots (4% sample with some overlap)

[AdamHeard]

Most teams aren't using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn't show up as symptom.

[Joe Johnson]

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

[Thad House]

Originally Posted by Joe Johnson

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

I would even say 5V is reasonable. And easily doable with the current system, because we know it doesn't reboot until about 4.5v. I would still be OK if they system shut of the PWMs at 7v, but left everything else on until 5v. This year wasn't as bad for brownouts because 6 CIM drives were not advantageous, but if next year comes around and 6 CIM drives are king again, teams are going to start learning real quick about that 7v drop out if it stays.

[Michael Hill]

While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It's basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget...maybe it's just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it's not needed, use thicker power wire, run the compressor at strategic times, etc.). I'm not saying I would like a lower brownout voltage, but I have a hard time thinking there's anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.

[Jon Stratis]

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

[Andrew Schreiber]

Originally Posted by Jon Stratis

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

The CRIO required 19-30V and the CRIO II (4 Slot) was 9-30V. The PDB would continue to provide 24V to the CRIO down to 4.5V per the spec sheet. (And in practice, we regularly saw drops to 5V in 2014 due to an intermittent short, anything less and it cut out logging)

The RoboRIO is 7-16V but instead of going through any sort of converter the PDP feeds VBat.

[Greg McKaskle]

Brain dead, close to criminal, and really dumb?

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

1. Why was this feature included?
2. Does it address the targeted issue(s)?
3. Does it introduce new issues?
4. Can it be improved?

---------

1. Why was this feature included?
The brownout feature is intended to cut power to some circuits in order to maintain power to others deemed more critical. There is a limited supply of current, and without this feature you don't have brownouts but instead have blackouts/reboots of devices based on their power supply details.

The critical components were deemed to be the roboRIO processor and the radio. High current consumers are motor controllers and compressor. The roboRIO's integrated 6V, 5.5V, and 3.3V were incorporated as well.

This is a new feature that wasn't present with the previous PD or cRIO. The cRIO voltage limits were listed above, and as described, the PD board boosted several supplies. Yet teams with a high current draw and a weak battery could still reboot the cRIO and/or radio. This primary goal of this feature was to avoid reboots of the roboRIO cpu and radio.

2. Does it address the targeted issue(s)?
The roboRIO alpha and beta testing included "push" tests where the robots were intentionally run with the same battery and encouraged to push walls. They were browned out repeatedly in order to observe and test this feature. Tom Line's observations were made and investigated during these test. In general, the results were positive, but there are side-effects, and there were discussions about how to improve the feature to minimize those.

During the 2015 season, I was CSA at three events and helped some at champs. I saw teams with dozens of brownout events and I remember seeing one with over 250 brownouts in a single match. The team had a weak battery, long robot, sticky wheels, and would have rebooted easily and repeatedly with the previous system. With it, they survived the match and contributed -- some -- to their alliance. They knew the battery was weak and new what to do to avoid that portion of the problem. So yeah, it seems to work for the intended problem.

But roboRIO and radio reboots still happen, and more than I'd like to see. I was able to investigate a few dozen of these at each regional and in 70% of the cases it was due to a loose connection at either the main breaker or the battery leads. A few were clearly due to cable insertion at the roboRIO or PD. Others were harder to pin down but could be due to fuse insertion on the PD. It may be possible for the brownout to go black before the motor controllers and compressor comply, but we need more usage to know how common this is.

3. Does it introduce new issues?
Yes.
To generalize, if teams implement more sophisticated feedback control and do not take brownouts into account, they will see new issues caused by the brownout feature. For example, if a PID is commanding a motor controller to 40%, but the motor output is actually at 0% due to brownout, integral error can accumulate and cause a bump when the motor operation is enabled again. If the sensor value droops, due to input voltage supply, this can compound the issue. If the sensor power supplied by the roboRIO goes into protection mode, then the sensor value isn't valid and this gibberish can cause even more severe control issues. Tom Line's comments above were caused by a combination of these effects. And as other beta teams pointed out, incremental sensors used for absolute measurement will now have a bias and/or need to be recalibrated.

There is another issue, and that is that the system now effectively limits the current usage in an additional way. The main breaker, individual breakers, and brownout features are all imposing limits on the available current. Teams familiar with the first two may feel like a third wasn't needed, and will once again have to discover the edge of the new envelope and how to push the system. Published specs and time with the system will address this.

4. Can it be improved?
Yes.
This is still underway. During beta, the FPGA stopped disabling the 5V and 3.3V sensor rail intentionally. It disables the 6V used by servos, the PWM values are set to zero or neutral, and the CAN devices are notified and should stop using large amounts of current. There are small tweaks that will attempt to shorten the brownout response of the high current devices but maintain solenoid states, etc. The PID functions and tutorials will include brownout awareness and can even incorporate sensor power loss awareness. Using incremental sensors for absolute measurements was already tricky, and some teams have chosen to supply their own sensor power via the VDM. All of these should improve the gap between brownout and sensor rail dropout and the side-effects of it if/when it happens.

As for pushing the envelope, the power API and PD features should make this much easier and more adaptive than it used to be.

The feedback that a robot is browning out also needs to be improved. The DS counts them, adds them to the log, puts a big red indicator behind the battery readout, and sends the into the field monitor. The indication on the field monitor is a momentary red blink on the voltage box. IMO there are many improvements to be made there.

Conclusion?
The team with 250 brownouts, the team with just one or two, they are able to finish the match instead of rebooting three or four times. They will have a better experience. The 1000's of teams who could use better mentoring still need to tighten the nuts on their main breaker, attach battery leads more robustly, and insert the fuses into the PD, but this feature seems to help them far more than it hurts. It has already been improved from the alpha/beta period, additional tweaks are underway, and we are open to questions and feedback. Insults ... I've been called worse, but they aren't so productive.

I like the OP's tone of this thread. So what are the specifics?
Greg McKaskle

[MrRoboSteve]

Originally Posted by Greg McKaskle

The indication on the field monitor is a momentary red blink on the voltage box.

I never noticed this -- will keep an eye out for it at Saturday's event.

[Joe Johnson]

Originally Posted by Greg McKaskle

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

<snip>
Insults ... I've been called worse, but they aren't so productive.

<snip>
Greg McKaskle

Greg,

Was I being hyperbolic with the "criminal" comments? sure. Calling it "really dumb"? No, not at all. If anything I would use stronger language.

I am not a EE by training but I have years and years of robotics experience both inside and outside of FIRST, as a hobbyist and as a robotic professional. These years of experience have taught me that you just don't scrimp when it comes to protecting the power supplies that keep your processors/coprocessors, sensors, and communications link alive.

That is a general rule. When it comes to FIRST FRC control system, where we know there is a battery that can be easily overwhelmed in terms of amps it can source compared to the amps requested by the motors allowed, the designers should take even more precautions when it comes to protecting control system from the inevitable low battery conditions. FIRST can put as many "buyer beware" signs up as it wants. It doesn't absolve them from the responsibility to provide a control system worthy of the name.

Based on the results I have seen using the RoboRIO and the discussion about the trade offs people made during the design, it is obvious to me that FIRST (and NI) did not take power management as seriously as they should have when as they were designing the new control system.

All the defense of 7V Brownout is missing the point. As are discussions about what last year's system would have done. This was a chance to design a system from scratch.

Yes, once you decide that it is okay for the RoboRIO to reboot at 4.5V and that you are not going to keep the 5V and 3.3V rails alive that it is okay to power them off the 6V line, then yes, you eventually get to a Brownout Voltage of 7V. Sure, a brownout is better than a reboot, but can we all agree that few brownouts are better and no brownouts are best?

For all that, I really don't mind that the PWMs are disabled at a certain voltage. Do I wish it were a bit lower? Sure, but I bow to the reality that ultimately it is better to have the motors turned off than to lose more critical thinking and sensing functions. I said brownouts but I was really talking about the larger issue of preserving higher brain function down to a voltage sufficiently low that it never happens.

The original sin in all this is that FIRST didn't take power management on critical systems as seriously as it should (as evidenced by the initial choice to not to even protect the 5V and 3.3V rails which power sensors and co-processors).

Yes, I am calling FIRST (and NI) out here. This could have and should have been a complete non-issue. It was foreseeable and it was avoidable.

I have a question that I will pose to every person that defends the current status quo: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

Saving a few bucks by not providing this feature on a control system that costs literally 100s of dollars that 1000s of teams will use to control robots that each team will invest many 1000s of hours and many 1000s of dollars making is... well... I don't want to be called out for making insults again so I'll leave it to the readers to decide what they would call it once they understand how easily and inexpensively this feature could have been provided.

Sincerely,

Dr. Joe J.

*I am not 100% what the 6V supply powers. If it's just the servo power, we could I would be willing let the 6V go -- servos going dark are no worse than motors turning off.

[Greg McKaskle]

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

I'm sure my post came off more defensive than I intended, but then again, a rebuttal of almost any sort easily tends in that direction. I think your questions and goals make sense. It will be interesting to see what other's would have done. If there is information that would make the alternate design exercise easier or more factual, I'll do what I can to get the info to you. In that spirit, here are the links to the specs, user manual, etc.

Here is a page from team 358 that did some pretty extensive independent testing of the entire system and details what happens as the voltage does the limbo. They took the roboRIO to under 4V before it rebooted and the radio under 3.5. Not your 3.3V goal, but in the neighborhood. And as documented and designed, it is staged, which introduces the brownout condition. Some things start to disappear at about 6.8. During alpha I believe it was often rounded to 7V.
Beta team 358 Testing of Power Management

Here is a direct link to the documentation of the power management stages.
roboRIO User Manual and the details are on page 6 and 7.

Just to round out the power discussion, the other requirement, the one that was given a higher priority, was to allow for any pin to short to any other pin with no damage. We regularly rake a screwdriver across exposed pins to demo this. The power tab of the DS will show you which rail is shorted. So detection and protection against over voltage and shorts was deemed critical. The system will also tolerate being plugged into the older 24V PD without damage. It won't operate, but isn't damaged and indicates the issue via LEDs. By the way, having USB cables with exposed ground shield on a robot with high current 12V exposed connectors is also challenging.

Oh, and 6V is used to power servos.

Greg McKaskle

[Joe Johnson]

Originally Posted by Greg McKaskle

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

<snip>

Greg McKaskle

Greg,
As to the fact that the actual RoboRIO blackout voltage was show to be 3.5V in one case, while I want to say good things about this, I just can't. The Spec you link to says 4.5V, I suppose that is what it was designed to be. One particular example showing a lower actual drop out voltage doesn't change that. Specs are about what happens to the worst built RoboRIO on its worst day. Am I happy that one particular data point is suggestive that there is design margin in that 4.5V? Sure. Do I wish that we were talking about a designed blackout voltage of 3.3V (or lower -- recall that as an ME my opening ask was for the RoboRIO and other key equipment stay alive down to 1V and was only talked off the ledge by my EE friends) and were talking about actual blackout voltages in the sub 3V range? Yes, yes I do.

BUT the RoboRIO reboot voltage is only one piece of the puzzle. The control system is a SYSTEM. Its nice to have a RoboRIO not rebooting but that is not enough. I need to have radio communications alive and well. I need to have my incremental encoders not lose their positions. I need to have my MEMS IMU keep its position for a the whole match (I'm looking at you navX). I need to have my onboard Beaglebone coprocessor not reboot during a match. I need to have sensors doing their thing. ...

It was FIRST's task to think on a system level and provide a system that just works.

As to the other things that FIRST/NI did right, I am not disputing this. There are a ton of things that FIRST/NI did really well. Protection against reversed and misrouted power/ground IS important and, as far as I know, the RoboRIO does a good job of this. Nice job on that front.

But that doesn't free the FIRST from the responsibility to manage power effectively. On this front, FIRST has a lot to answer for.

And now I will finish with the question I am promised to ask until it gets answered: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

I have a pretty good idea what the answer is but I would like to hear those who made the decision not to provide give their estimates.

Dr. Joe J.

[Greg McKaskle]

OK. Time for me to make another post that will come off sounding more defensive that I intend for it to.

The link to the user manual and spec sheet of the roboRIO was intended to clarify the design and spec'ed behavior with low input power. The link to team 358's results were to intended to show how an alpha team tested the more complete system that included USB devices, rail power, VRM, etc. I'm not trying to sell you a used car or use a single team's testing to fool anyone. I'm giving the info I know of for the roboRIO and system elements that were delivered. That may help you or others on this alternate design quest.

As for the other background on short protection -- I described the additional protection mechanisms so alternate designs would be apples-to-apples.

As I described in my background, I don't have the chops to do the current or alternate power supply design. But I do volunteer as a team mentor and CSA, so that I can see first-hand how the elements are used and how they perform. From that perspective, as a customer and support engineer, I do not see the need for the 1V or even 3.3V input requirement.

Greg McKaskle

[crake]

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

[slibert]

Originally Posted by crake

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

Thanks, Chris. As the founder of Kauai Labs, I have perhaps a unique perspective on this, as a result of developing/supporting the navX MXP IMU. Key goals included an easy-to-use, plug-n-play product useful to both upper-tier teams as well as beginning teams. Our focus is to keep improving things til we reach those goals.

The MXP connector was a great idea that really helped enable plug-n-play; however, this particular brownout issue negatively impacts these goals.

The primary reason is that the navX MXP (and certain other sensors, for sure) needs to manage state; in the case of the navX MXP there's some non-trivial run-time calibration state that's key. As noted by others, MXP products (Revduino) and other non-MXP products (e.g, the Beaglebone) are going to encounter similar issues.

In this case the specific issue is the 5 and 3.3V user rail supply rails on the MXP connector; voltage on these is not maintained in the case of a stage 2 brownout. The navX MXP has an onboard LDO that regulates the 5V MXP rail down to 3.3V to power a 32-bit ARM microcontroller; the board consumes a max of 250 mA.

Thanks to Joe H., who I met at CVR and pointed out the boost/buck reg attached to the USB port. Fortunately the navX MXP has a separate USB-based power supply on the navX MXP and the logic to cleanly switch between supplies w/out glitch. That's documented in our best practices now.

But that really isn't plug-n-play, to have this extra USB cable connecting the RoboRio to the navX MXP secondary power supply - just to avoid the brownout.

So the recommendation (OK, plea) is to maintain the power to the 5 and 3.3V MXP user rail during stage 2 brownout.

If that is not possible, we (MXP developers) need recommendations on how to best deal with this issue. Additionally, I think continuing the status quo on this will have the effect of diminishing the potential of the MXP interface. We are seeing tremendous growth in digital "smart" sensors/ICs w/onboard processing and state, and would like to help make them available to the FIRST community. I can confidently say that the robot of 3 years from now will have an even greater need than today.