Brownout behavior - alternative design goals

[Thad House]

Originally Posted by Andrew Schreiber

The second part is what I was suggesting as a solution to this issue. I'm well aware it's not legal, but a rule change could fix that fairly trivially.

I'd also disagree that it's a 100% software fix to lower voltage brownout, or, more accurately, that it should be fixed (And the 6.8-6.3 brownout on the 6V rails is more than likely hardware). To me the brownout is a feature meant to prevent controller shutdowns. While the 6.3V number may seem high, if you're drawing your battery down to 6.3V regularly it's not good and I'd rather have the control try to protect itself by disabling the large draws. Think of it as a soft failure. I'd rather have a soft failure than a dead robot.

To me, if I'm giving it voltage outside the input range then I'm not using the controller properly and any failures are my fault. The fact that the RoboRIO stays alive and tries to save itself as much as it does is nice.

So then why need an external LDO? The os and code stay running down to 4.5v anyway.

And the virtual brownouts should be changeable by the fpga. I'd be surprised id they were not.

[MrRoboSteve]

Some thoughts:

1. It will always be possible for the roboRIO to have less power than it needs to run, so load shedding features will continue to be useful for the great majority of teams.

2. Might be possible for the roboRIO to tolerate longer periods of low voltage. This would be a software change. That said, robots operating for extended periods at 7 volts are visibly sick.

3. regulating the input power would require alternate means to detect battery voltage. Analog sidecar anyone?

4. As FTAA, the only way I knew a robot was in brownout was to go stand behind the drivers and watch the display. We need to make brownout on the robot more visible, to avoid situations like the one Joe describes. Ideas:
a. FMS field monitor should show "brownout count" statistic
b. DS software should have a brownout indicator that stays lit if there's been a brownout
c. DS stack light should change state for a second or so when there's a brownout

[AllenGregoryIV]

Originally Posted by MrRoboSteve

3. regulating the input power would require alternate means to detect battery voltage. Analog sidecar anyone?

The PDP is already required to be wired up with CAN. Pretty sure the PDP with CAN reports battery voltage just fine.

[jhersh]

Originally Posted by Tom Line

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

Hi Tom,

I think this is a perfect example of where your robot has a better chance to operate reliably using the new functionality, but the control software needs to pay attention to the info available. Since you are not losing absolute position and the motor is disabled, the likely cause for the behavior you saw was a combination of integral windup and reading the sensor in a non-ratiometric way. Naturally as the voltage of the 5V supply drops the lower the resolution of the reading.

The roboRIO Power API will tell you what the voltage of the reference rail is so you can divide. There is also a new potentiometer API that does this for you. The Power API also tells you when the brown-out level has been crossed so you can pause your control loop until it has recovered. The Power API also tells you when the 5V supply has blacked out (though I did notice a bug in this in the LV API implementation that I'll fix for next season), so you can stop believing your sensor at that point. This will always be below the brown-out level, so your control loop should already be paused.

[jhersh]

Originally Posted by Thad House

And the virtual brownouts should be changeable by the fpga. I'd be surprised id they were not.

The level is actually set by a hardware comparator. However it is likely that a work around could be implemented in the FPGA if needed.

[Thad House]

Originally Posted by jhersh

The level is actually set by a hardware comparator. However it is likely that a work around could be implemented in the FPGA if needed.

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms. Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

[jhersh]

Originally Posted by MrRoboSteve

There are two sources of information on the brownout feature that have conflicting information.

http://wpilib.screenstepslive.com/s/...anual-id=24166

Page 7 of http://www.ni.com/pdf/manuals/374474a.pdf

The screen-steps page is accurate. I've filed a bug report about the User Manual. Thanks for pointing that out.

[Andrew Schreiber]

Originally Posted by Thad House

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms. Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

The PWM cutout, I'm actually happy about and hoping it will help end the stupid drivetrain wars we've been engaged in since 2005. The idea of "throw more power at it", while valid, gets annoying very quickly when 6CIM shifting drives smack into you from across the field.

[jhersh]

Originally Posted by Thad House

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms.

The point of having it "high" at 6.8 V is to have some breathing room to get those loads turned off in time for the battery to recover before the controller blacks out. Since we don't have control over the PD channels we can't simply turn off the high loads. We have to ask that the controller stop. That takes time. This transition happens relatively quickly in human-perceivable time, so if your load is very high but your battery is not dead, you will likely not even notice that the brown-out happened. That is the goal and others have reported this. If as you say it dips for a few milliseconds, then your motor controller will only be disabled for a single cycle (5 ms on most PWM motor controllers). If it is near the end of a match and your battery has been abused most of that time, it will not recover as quickly, but the goal is to avoid the controller rebooting even in that case. I think the level is appropriate.

Originally Posted by Thad House

Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

The three user supplies were originally considered a potentially "high load", around 22 W, (though practically that is not the case). As such those two were on the chopping block for brown out to help save the boost supply running the processor from faulting. During the design it was determined the 5 V and 3.3V supplies could not handle the upper end of the input voltage and still use the controller parts we selected, so it was decided that a fix for that would be to power them down-stream of the 6V supply to limit the input they see. All three supplies were also designed as buck supplies only. Since at that time they were all set to be disabled at 6.8V anyway, this was an appropriate solution.

Later (alpha 2) when we decided that 5 V and 3.3 V supplies should not disable until black out, they were still down stream of the 6 V supply. At this point the hardware was already being manufactured and the power supply topology could not be changed with acceptable risk and schedule impact. Being down stream of the 6 V supply meant that the input they see goes away completely when the 6 V supply controller enters a fault state due to insufficient input voltage. This happens at about 6.25 V. This means that if the motor controllers respond (starting at 6.8 V) before the input drops to 6.25 V, the sensor supplies will not black out.

[MrRoboSteve]

Updated:

1. It will always be possible for the roboRIO to have less power than it needs to run, so load shedding features will continue to be useful for the great majority of teams.

2. Might be possible for the roboRIO to tolerate longer periods of low voltage. That said, robots operating for extended periods at 7 volts are visibly sick.

3. As FTAA, the only way I knew a robot was in brownout was to go stand behind the drivers and watch the display. We need to make brownout on the robot more visible, to avoid situations like the one Joe describes. Ideas:
a. FMS field monitor should show "brownout count" statistic, separately showing stage 1 and 2
b. DS software should have a brownout indicator that stays lit if there's been a brownout
c. DS stack light should change state for a second or so when there's a brownout

4. It's difficult to change the voltage of the stage 2 brownout, because it's a function of the power supply architecture.

5. It would be interesting to collect DS logs of machines in brownout, to see if at the point of outage the voltage has a steep slope. This would help assess the utility of changes to how stage 1/2 brownouts work.

[Thad House]

Originally Posted by jhersh

The point of having it "high" at 6.8 V is to have some breathing room to get those loads turned off in time for the battery to recover before the controller blacks out. Since we don't have control over the PD channels we can't simply turn off the high loads. We have to ask that the controller stop. That takes time. This transition happens relatively quickly in human-perceivable time, so if your load is very high but your battery is not dead, you will likely not even notice that the brown-out happened. That is the goal and others have reported this. If as you say it dips for a few milliseconds, then your motor controller will only be disabled for a single cycle (5 ms on most PWM motor controllers). If it is near the end of a match and your battery has been abused most of that time, it will not recover as quickly, but the goal is to avoid the controller rebooting even in that case. I think the level is appropriate.



The three user supplies were originally considered a potentially "high load", around 22 W, (though practically that is not the case). As such those two were on the chopping block for brown out to help save the boost supply running the processor from faulting. During the design it was determined the 5 V and 3.3V supplies could not handle the upper end of the input voltage and still use the controller parts we selected, so it was decided that a fix for that would be to power them down-stream of the 6V supply to limit the input they see. All three supplies were also designed as buck supplies only. Since at that time they were all set to be disabled at 6.8V anyway, this was an appropriate solution.

Later (alpha 2) when we decided that 5 V and 3.3 V supplies should not disable until black out, they were still down stream of the 6 V supply. At this point the hardware was already being manufactured and the power supply topology could not be changed with acceptable risk and schedule impact. Being down stream of the 6 V supply meant that the input they see goes away completely when the 6 V supply controller enters a fault state due to insufficient input voltage. This happens at about 6.25 V. This means that if the motor controllers respond (starting at 6.8 V) before the input drops to 6.25 V, the sensor supplies will not black out.

If these drop happens, do the internal pullups still drop? Or are they ran off a seperate voltage regulator?

[jhersh]

Originally Posted by Thad House

If these drop happens, do the internal pullups still drop? Or are they ran off a seperate voltage regulator?

They do not drop. Separate regulator. The internal pulls on all DIO / I2C / SPI lines are connected to the internal 3.3 V supply that is fed by the boost supply in roboRIO.

[Greg McKaskle]

Brain dead, close to criminal, and really dumb?

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

1. Why was this feature included?
2. Does it address the targeted issue(s)?
3. Does it introduce new issues?
4. Can it be improved?

---------

1. Why was this feature included?
The brownout feature is intended to cut power to some circuits in order to maintain power to others deemed more critical. There is a limited supply of current, and without this feature you don't have brownouts but instead have blackouts/reboots of devices based on their power supply details.

The critical components were deemed to be the roboRIO processor and the radio. High current consumers are motor controllers and compressor. The roboRIO's integrated 6V, 5.5V, and 3.3V were incorporated as well.

This is a new feature that wasn't present with the previous PD or cRIO. The cRIO voltage limits were listed above, and as described, the PD board boosted several supplies. Yet teams with a high current draw and a weak battery could still reboot the cRIO and/or radio. This primary goal of this feature was to avoid reboots of the roboRIO cpu and radio.

2. Does it address the targeted issue(s)?
The roboRIO alpha and beta testing included "push" tests where the robots were intentionally run with the same battery and encouraged to push walls. They were browned out repeatedly in order to observe and test this feature. Tom Line's observations were made and investigated during these test. In general, the results were positive, but there are side-effects, and there were discussions about how to improve the feature to minimize those.

During the 2015 season, I was CSA at three events and helped some at champs. I saw teams with dozens of brownout events and I remember seeing one with over 250 brownouts in a single match. The team had a weak battery, long robot, sticky wheels, and would have rebooted easily and repeatedly with the previous system. With it, they survived the match and contributed -- some -- to their alliance. They knew the battery was weak and new what to do to avoid that portion of the problem. So yeah, it seems to work for the intended problem.

But roboRIO and radio reboots still happen, and more than I'd like to see. I was able to investigate a few dozen of these at each regional and in 70% of the cases it was due to a loose connection at either the main breaker or the battery leads. A few were clearly due to cable insertion at the roboRIO or PD. Others were harder to pin down but could be due to fuse insertion on the PD. It may be possible for the brownout to go black before the motor controllers and compressor comply, but we need more usage to know how common this is.

3. Does it introduce new issues?
Yes.
To generalize, if teams implement more sophisticated feedback control and do not take brownouts into account, they will see new issues caused by the brownout feature. For example, if a PID is commanding a motor controller to 40%, but the motor output is actually at 0% due to brownout, integral error can accumulate and cause a bump when the motor operation is enabled again. If the sensor value droops, due to input voltage supply, this can compound the issue. If the sensor power supplied by the roboRIO goes into protection mode, then the sensor value isn't valid and this gibberish can cause even more severe control issues. Tom Line's comments above were caused by a combination of these effects. And as other beta teams pointed out, incremental sensors used for absolute measurement will now have a bias and/or need to be recalibrated.

There is another issue, and that is that the system now effectively limits the current usage in an additional way. The main breaker, individual breakers, and brownout features are all imposing limits on the available current. Teams familiar with the first two may feel like a third wasn't needed, and will once again have to discover the edge of the new envelope and how to push the system. Published specs and time with the system will address this.

4. Can it be improved?
Yes.
This is still underway. During beta, the FPGA stopped disabling the 5V and 3.3V sensor rail intentionally. It disables the 6V used by servos, the PWM values are set to zero or neutral, and the CAN devices are notified and should stop using large amounts of current. There are small tweaks that will attempt to shorten the brownout response of the high current devices but maintain solenoid states, etc. The PID functions and tutorials will include brownout awareness and can even incorporate sensor power loss awareness. Using incremental sensors for absolute measurements was already tricky, and some teams have chosen to supply their own sensor power via the VDM. All of these should improve the gap between brownout and sensor rail dropout and the side-effects of it if/when it happens.

As for pushing the envelope, the power API and PD features should make this much easier and more adaptive than it used to be.

The feedback that a robot is browning out also needs to be improved. The DS counts them, adds them to the log, puts a big red indicator behind the battery readout, and sends the into the field monitor. The indication on the field monitor is a momentary red blink on the voltage box. IMO there are many improvements to be made there.

Conclusion?
The team with 250 brownouts, the team with just one or two, they are able to finish the match instead of rebooting three or four times. They will have a better experience. The 1000's of teams who could use better mentoring still need to tighten the nuts on their main breaker, attach battery leads more robustly, and insert the fuses into the PD, but this feature seems to help them far more than it hurts. It has already been improved from the alpha/beta period, additional tweaks are underway, and we are open to questions and feedback. Insults ... I've been called worse, but they aren't so productive.

I like the OP's tone of this thread. So what are the specifics?
Greg McKaskle

[Tom Line]

Originally Posted by jhersh

Hi Tom,

I think this is a perfect example of where your robot has a better chance to operate reliably using the new functionality, but the control software needs to pay attention to the info available. Since you are not losing absolute position and the motor is disabled, the likely cause for the behavior you saw was a combination of integral windup and reading the sensor in a non-ratiometric way. Naturally as the voltage of the 5V supply drops the lower the resolution of the reading.

The roboRIO Power API will tell you what the voltage of the reference rail is so you can divide. There is also a new potentiometer API that does this for you. The Power API also tells you when the brown-out level has been crossed so you can pause your control loop until it has recovered. The Power API also tells you when the 5V supply has blacked out (though I did notice a bug in this in the LV API implementation that I'll fix for next season), so you can stop believing your sensor at that point. This will always be below the brown-out level, so your control loop should already be paused.

I just emailed myself this at home so I can implement it over the summer. We never had a problem with it during competition but having the failover built into VI's will make it pretty easy to implement. Thanks Joe.

[MrRoboSteve]

Originally Posted by Greg McKaskle

The indication on the field monitor is a momentary red blink on the voltage box.

I never noticed this -- will keep an eye out for it at Saturday's event.