Brownout behavior - alternative design goals

[crake]

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

[slibert]

Originally Posted by crake

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

Thanks, Chris. As the founder of Kauai Labs, I have perhaps a unique perspective on this, as a result of developing/supporting the navX MXP IMU. Key goals included an easy-to-use, plug-n-play product useful to both upper-tier teams as well as beginning teams. Our focus is to keep improving things til we reach those goals.

The MXP connector was a great idea that really helped enable plug-n-play; however, this particular brownout issue negatively impacts these goals.

The primary reason is that the navX MXP (and certain other sensors, for sure) needs to manage state; in the case of the navX MXP there's some non-trivial run-time calibration state that's key. As noted by others, MXP products (Revduino) and other non-MXP products (e.g, the Beaglebone) are going to encounter similar issues.

In this case the specific issue is the 5 and 3.3V user rail supply rails on the MXP connector; voltage on these is not maintained in the case of a stage 2 brownout. The navX MXP has an onboard LDO that regulates the 5V MXP rail down to 3.3V to power a 32-bit ARM microcontroller; the board consumes a max of 250 mA.

Thanks to Joe H., who I met at CVR and pointed out the boost/buck reg attached to the USB port. Fortunately the navX MXP has a separate USB-based power supply on the navX MXP and the logic to cleanly switch between supplies w/out glitch. That's documented in our best practices now.

But that really isn't plug-n-play, to have this extra USB cable connecting the RoboRio to the navX MXP secondary power supply - just to avoid the brownout.

So the recommendation (OK, plea) is to maintain the power to the 5 and 3.3V MXP user rail during stage 2 brownout.

If that is not possible, we (MXP developers) need recommendations on how to best deal with this issue. Additionally, I think continuing the status quo on this will have the effect of diminishing the potential of the MXP interface. We are seeing tremendous growth in digital "smart" sensors/ICs w/onboard processing and state, and would like to help make them available to the FIRST community. I can confidently say that the robot of 3 years from now will have an even greater need than today.

[tkeil]

Hello. Power electronics engineer here. Somebody directed me to this thread and I thought I'd step in for a minute.

Several of the mentors as well as NI folks have already done a great job explaining this limitation and linking to helpful documentation for diagnosis and solutions. I don't have much to add on this front, but I would like to talk about what goes into selecting an appropriate input voltage from a technical perspective.

It's very easy for someone to be critical and say 'why doesn't it just work perfectly down to <X> volts?!' where <X> is some arbitrarily low voltage that some individual expects to never hit in their particular system implementation. The reason why is that this is a complex embedded controller and there are many trade-offs. In this case the main trade-off is thermal dissipation.

The roboRIO regulates 45W of power maximum. This is a lot for a small, plastic, convection cooled device with no open vents. The power supplies were a major concern for this design since with 45W input they account for a large percentage of the internally dissipated power.

Some of the best power supplies in the world can get >95% efficiency but these are nearly all power supplies with a fixed input voltage and a fixed output voltage. As soon as the power supply has to work over a wide input range, that efficiency falls because the power supply can no longer work at its most efficient operating point all the time. The roboRIO supports >2x input voltage variation. We can get better than 90% efficiency with this wide input range but the wider it is, the worse the losses get.

roboRIO is designed specifically for the FIRST competition, but it leverages many sections of circuits from other embedded controllers. It's designed by the same team that makes our professional cSeries controllers. You should think of roboRIO as a professional embedded controller with a protective cushion around it.

Fully around 1/3 of the circuity in the roboRIO is to protect the device from accidental damage and make it easy for the users to diagnose problems. Our highest priorities of course were preventing failures that permanently damage the device. I bring this up because some of these protections (for example reverse voltage protection) have a required minimum voltage and lie in the power path. These circuits are often leveraged from other designs and while they could be changed, it's costly to design and verify.

Some of these protection devices need to be in the direct current paths inside the device. This becomes a huge problem when your currents get large.

Let's look at a quick hypothetical where the input voltage minimum would be 4.5V:

The roboRIO needs to regulate 45W. If the input voltage drops, the current gets higher. Resistive losses are calculated as P=i^2*R. Consider that the current into the input connector of the roboRIO @ 7Vin is ~6.42A. If the minimum input voltage had been 4.5V that current would be ~10A. When you square those terms thats about a 2.5x increase in resistive losses through the whole input section of the power distribution path and switches in the power supplies.

Before you even get to the first power supply in the design, the power has to go through the input connector, the fuses, the common-mode choke, more noise filtering, reverse voltage and reverse current protection, and finally an input current sense shunt. The losses in all these devices would be 2.5x greater in this fictional example of allowing full power down to 4.5V.

Many (most?) of these devices would also have to be larger to support the current and thermal dissipation. For instance, 10A is far over the rating of the input voltage connectors. [For a laugh you can consider the device operating down to 1V in which case the current would be 45A and the input connector terminals would be around the size of an AA battery each.]

Anyway, when you combine the extra losses in the main input supplies with the extra losses in the resistive elements (and the increased component sizes) you will quickly lose your form factor and ability to use a plastic housing. If we had to go to a metal housing and possibly one with heat spreaders, you're talking about increasing the total device COGs by roughly 30% - 50% and increasing the form factor and weight.

Input power and voltage range is a big deal. Myself and others at NI think about it a lot and then over and over again for each new design. As you guys all know as good engineers and scientifically minded folks, everything's a trade off.

Thanks for your time volunteering with the students.

Tim

[Joe Johnson]

Originally Posted by tkeil

Hello. Power electronics engineer here. Somebody directed me to this thread and I thought I'd step in for a minute.

Several of the mentors as well as NI folks have already done a great job explaining this limitation and linking to helpful documentation for diagnosis and solutions. I don't have much to add on this front, but I would like to talk about what goes into selecting an appropriate input voltage from a technical perspective.

It's very easy for someone to be critical and say 'why doesn't it just work perfectly down to <X> volts?!' where <X> is some arbitrarily low voltage that some individual expects to never hit in their particular system implementation. The reason why is that this is a complex embedded controller and there are many trade-offs. In this case the main trade-off is thermal dissipation.

The roboRIO regulates 45W of power maximum. This is a lot for a small, plastic, convection cooled device with no open vents. The power supplies were a major concern for this design since with 45W input they account for a large percentage of the internally dissipated power.

Some of the best power supplies in the world can get >95% efficiency but these are nearly all power supplies with a fixed input voltage and a fixed output voltage. As soon as the power supply has to work over a wide input range, that efficiency falls because the power supply can no longer work at its most efficient operating point all the time. The roboRIO supports >2x input voltage variation. We can get better than 90% efficiency with this wide input range but the wider it is, the worse the losses get.

roboRIO is designed specifically for the FIRST competition, but it leverages many sections of circuits from other embedded controllers. It's designed by the same team that makes our professional cSeries controllers. You should think of roboRIO as a professional embedded controller with a protective cushion around it.

Fully around 1/3 of the circuity in the roboRIO is to protect the device from accidental damage and make it easy for the users to diagnose problems. Our highest priorities of course were preventing failures that permanently damage the device. I bring this up because some of these protections (for example reverse voltage protection) have a required minimum voltage and lie in the power path. These circuits are often leveraged from other designs and while they could be changed, it's costly to design and verify.

Some of these protection devices need to be in the direct current paths inside the device. This becomes a huge problem when your currents get large.

Let's look at a quick hypothetical where the input voltage minimum would be 4.5V:

The roboRIO needs to regulate 45W. If the input voltage drops, the current gets higher. Resistive losses are calculated as P=i^2*R. Consider that the current into the input connector of the roboRIO @ 7Vin is ~6.42A. If the minimum input voltage had been 4.5V that current would be ~10A. When you square those terms thats about a 2.5x increase in resistive losses through the whole input section of the power distribution path and switches in the power supplies.

Before you even get to the first power supply in the design, the power has to go through the input connector, the fuses, the common-mode choke, more noise filtering, reverse voltage and reverse current protection, and finally an input current sense shunt. The losses in all these devices would be 2.5x greater in this fictional example of allowing full power down to 4.5V.

Many (most?) of these devices would also have to be larger to support the current and thermal dissipation. For instance, 10A is far over the rating of the input voltage connectors. [For a laugh you can consider the device operating down to 1V in which case the current would be 45A and the input connector terminals would be around the size of an AA battery each.]

Anyway, when you combine the extra losses in the main input supplies with the extra losses in the resistive elements (and the increased component sizes) you will quickly lose your form factor and ability to use a plastic housing. If we had to go to a metal housing and possibly one with heat spreaders, you're talking about increasing the total device COGs by roughly 30% - 50% and increasing the form factor and weight.

Input power and voltage range is a big deal. Myself and others at NI think about it a lot and then over and over again for each new design. As you guys all know as good engineers and scientifically minded folks, everything's a trade off.

Thanks for your time volunteering with the students.

Tim

Tim,
I appreciate that there are a lot of constraints and trade offs in every design but at the end of the day it has to be suitable for the intended use.

I have circled back with a number of very experienced FIRST folks from teams with many years of FIRST experience. I'm confirmed in my view that given the FRC battery and the motors that FIRST allows that a 4.5V reboot of the CPU was a poor design choice that was made worse by allowing the 5V and 3.3V rails to away at above 6V making sensors and coprocessors go dark.

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

Dr. Joe J.

[Andrew Schreiber]

Originally Posted by Joe Johnson

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

Dr. Joe J.

254 probably has more data than I do because they run closer to the edge, but I can tell you that in a season like 2014, when we DIDN'T have a short in our wiring, 125 rarely went below 6V despite some very aggressive gearing in both our drivetrain and our launching system. The only times (I recall) we had serious current draw* issues were when a vastly over-inflated ball jammed in our intake, the choo-choo was loading, and we were driving aggressively.

That being said, wasn't it you who coined the term "drivetrain wars" all those years back when we first started getting CIMs? I just view this as a good way to end those. EDIT: My memory DIDN'T fail me. http://www.chiefdelphi.com/forums/sh...79&postcount=1



* This ignores that we spent the first half of the season with a short that was causing us to die on the field regularly.

[Jared Russell]

Originally Posted by Andrew Schreiber

254 probably has more data than I do because they run closer to the edge, but I can tell you that in a season like 2014, when we DIDN'T have a short in our wiring, 125 rarely went below 6V despite some very aggressive gearing in both our drivetrain and our launching system. The only times (I recall) we had serious current draw* issues were when a vastly over-inflated ball jammed in our intake, the choo-choo was loading, and we were driving aggressively.

That being said, wasn't it you who coined the term "drivetrain wars" all those years back when we first started getting CIMs? I just view this as a good way to end those. EDIT: My memory DIDN'T fail me. http://www.chiefdelphi.com/forums/sh...79&postcount=1



* This ignores that we spent the first half of the season with a short that was causing us to die on the field regularly.

I'd much rather see the drivetrain wars moderated by game design (one of the only things I liked about Recycle Rush, though 2012 and 2013 did a great job of this as well).

Failing that, I favor limiting the number of drive motors, a speed limit, wheel restriction, etc., rather than brownouts. Brownouts can be insidious to deal with (if you have any control loops), and be next to impossible to completely avoid (the dynamics of any FRC robot frame mean that instantaneously, your wheels may be loaded really unfavorably when driving around the field). An FRC robot that is designed (mechanically) to be incapable of dropping below 6.8V would be a fairly uninspiring robot to watch.

Forgive my ignorance on such things, but is there any way to default the motor brownout to 6.8V, but allow teams to do something else "if they (think) they know what they are doing"? For example, I could see doing prioritized load shedding (ex. first drive, then intake, but position-controlled elevator last) as a viable way to deal with this, but there isn't really any headroom above 6.8V with which to implement such a scheme.

[Greg McKaskle]

Forgive my ignorance on such things, but is there any way to ...

Not yet, but it has been discussed and is technically feasible. It would be a way for teams to prioritize current supply, but also a way to bypass the protection and to blackout some portion of the robot. So, it is not a magic bullet.

Greg McKaskle

[Joe Johnson]

Originally Posted by Jared Russell

I'd much rather see the drivetrain wars moderated by game design (one of the only things I liked about Recycle Rush, though 2012 and 2013 did a great job of this as well).

I am with you as long as by game design you are not talking about making the refs enforce more and more critical pinning rules. It makes me cringe whenever I hear the announcer saying things like, "Ok. Good match. Now let's see if penalties will change the outcome..."

[MrRoboSteve]

Originally Posted by Joe Johnson

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

We're in agreement that the verdict won't be in until we have a game that demands higher current. This year wasn't that game.

[Tom Line]

Ok. So like good engineers, let's solve the potential problem. is it possible to design a circuit such that it can be put in line with the crio power wiring, plug into a battery pack, and protect the crio voltage supply?