Brownout behavior - alternative design goals

[MrRoboSteve]

In another thread:


*don't get me started about FIRST's decision to allow the NI folks to allow the RoboRIO loose its mind at just under 7V -- this is something close to criminal. Really... this was a really dumb design direction given the customer base and the intended use of the RoboRIO.

FWIW, this year I occasionally saw* robots with low battery enter brownout. These were always teams with bad/poorly charged batteries. For robots that rebooted, I think all of them had a wiring related root cause. Not to say that it didn't happen, but I don't think I ever saw it.

The purpose of this thread is to try to reverse engineer the requirements around the brownout feature, and discuss alternative requirements that might lead to a different outcome.

Here's what I think the requirements were for the current feature:


Provide as much telemetry from the robot as possible during low voltage states.
Avoid roboRIO reboots during matches.

Telemetry should show when bad robot behavior is caused by low voltage.

There are two sources of information on the brownout feature that have conflicting information.

http://wpilib.screenstepslive.com/s/4485/m/24166/l/289498?data-resolve-url=true&data-manual-id=24166

Page 7 of http://www.ni.com/pdf/manuals/374474a.pdf

* as FTAA at events with a total of 120 robots (4% sample with some overlap)

[AdamHeard]

Most teams aren't using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn't show up as symptom.

[Joe Johnson]

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

[Tom Line]

Most teams aren't using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn't show up as symptom.

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

[Thad House]

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

I would even say 5V is reasonable. And easily doable with the current system, because we know it doesn't reboot until about 4.5v. I would still be OK if they system shut of the PWMs at 7v, but left everything else on until 5v. This year wasn't as bad for brownouts because 6 CIM drives were not advantageous, but if next year comes around and 6 CIM drives are king again, teams are going to start learning real quick about that 7v drop out if it stays.

[Michael Hill]

While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It's basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget...maybe it's just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it's not needed, use thicker power wire, run the compressor at strategic times, etc.). I'm not saying I would like a lower brownout voltage, but I have a hard time thinking there's anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.

[Jon Stratis]

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

[Andrew Schreiber]

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

The CRIO required 19-30V and the CRIO II (4 Slot) was 9-30V. The PDB would continue to provide 24V to the CRIO down to 4.5V per the spec sheet. (And in practice, we regularly saw drops to 5V in 2014 due to an intermittent short, anything less and it cut out logging)

The RoboRIO is 7-16V but instead of going through any sort of converter the PDP feeds VBat.

[Joe Johnson]

While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It's basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget...maybe it's just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it's not needed, use thicker power wire, run the compressor at strategic times, etc.). I'm not saying I would like a lower brownout voltage, but I have a hard time thinking there's anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.

If this were the "real world," I would say, sure, one more constraint, add it to the list. BUT it isn't. This is a system who's job is to control a robot that plays a FIRST FRC game. FIRST was painted into a corner with the cRIO. They had insane time pressures and the had to pretty much take what they could get.

But the RoboRIO? It was designed SPECIFICALLY for FIRST FRC. It was a huge oversight to have anything important loose power, ever.

Another problem is that FIRST's system is SO COMPLEX that it is basically impossible for teams to duplicate what happens out on the field in their pit or even on the practice field.

I am not saying that it would have made that much of a difference but this whole brown out thing cost us in St. Louis. Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing. Half of our Qualifying Matches were gone before we got a reasonable answer as to what the problem was and had a work around. It seems to be something related to our robot pulling the voltage below 7V for a small fraction of a second at one point in our auton.

Again, so, yes, shame on us for not realizing that we were close to the edge for 4 prior tournaments where we never saw this problem. But also shame on FIRST for having an unnecessary edge for us to fall off to begin with and for not providing a person with the time and the knowledge to diagnose tricky problems such as ours (had it not been for Mike Copioli and one of his minions from CTRE, we'd still be in the dark about it).

So, that is my team's experience. If you're not on our team, it is easy to say, "Who cares?" Except, I am sure that there were many other teams that ran into problems as well.

Bottom line, I think this is the bad fruit that grew from a bad seed. FIRST should have made this a non-issue with the new controller.

Dr. Joe J.

[Andrew Schreiber]

If this were the "real world," I would say, sure, one more constraint, add it to the list. BUT it isn't. This is a system who's job is to control a robot that plays a FIRST FRC game. FIRST was painted into a corner with the cRIO. They had insane time pressures and the had to pretty much take what they could get.

But the RoboRIO? It was designed SPECIFICALLY for FIRST FRC. It was a huge oversight to have anything important loose power, ever.

Another problem is that FIRST's system is SO COMPLEX that it is basically impossible for teams to duplicate what happens out on the field in their pit or even on the practice field.

I am not saying that it would have made that much of a difference but this whole brown out thing cost us in St. Louis. Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing. Half of our Qualifying Matches were gone before we got a reasonable answer as to what the problem was and had a work around. It seems to be something related to our robot pulling the voltage below 7V for a small fraction of a second at one point in our auton.

Again, so, yes, shame on us for not realizing that we were close to the edge for 4 prior tournaments where we never saw this problem. But also shame on FIRST for having an unnecessary edge for us to fall off to begin with and for not providing a person with the time and the knowledge to diagnose tricky problems such as ours (had it not been for Mike Copioli and one of his minions from CTRE, we'd still be in the dark about it).

So, that is my team's experience. If you're not on our team, it is easy to say, "Who cares?" Except, I am sure that there were many other teams that ran into problems as well.

Bottom line, I think this is the bad fruit that grew from a bad seed. FIRST should have made this a non-issue with the new controller.

Dr. Joe J.

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

[Thad House]

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

The RoboRIO internally does have a Buck/Boost internal to it. It stays powered down to about 4.5v. However, when it dips below 7v, the FPGA disables everything. So it would be a 100% software fix to lower the brownout voltage.

On our practice bot, we actually fed the RoboRIO straight from the regulated 12v. This is because it gets the voltage that is uses from Vin. So if you feed regulated voltage into the RoboRIO it never goes into brownout state. However, this is not FRC legal, and it is required by the rules to be fed straight from Vbatt.

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.
The cRIO didnt have a specified brownout voltage. The 24v regulator I believe dropped at about 4.5v. But since the cRIO didnt control the power rails on the sidecar, the power from the sidecar would stay on until about 4v, when the sidecar would shut off.

We should start calling the 7v RoboRIO brownout the Virtual Brownout, because it is entirely controlled in software. The hardware brownout doesnt occur until 4.5v, just like the cRIO did.

[AdamHeard]

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

Yes, but in the long run you don't lose absolute position.

People see the mechanism drop/move and assume it must be from the battery voltage drop... but when it gets back up the control loop brings it back to where it wants to be and the drivers keep driving without issue. So most teams won't really complain about this.

[Andrew Schreiber]

The RoboRIO internally does have a Buck/Boost internal to it. It stays powered down to about 4.5v. However, when it dips below 7v, the FPGA disables everything. So it would be a 100% software fix to lower the brownout voltage.

On our practice bot, we actually fed the RoboRIO straight from the regulated 12v. This is because it gets the voltage that is uses from Vin. So if you feed regulated voltage into the RoboRIO it never goes into brownout state. However, this is not FRC legal, and it is required by the rules to be fed straight from Vbatt.

The second part is what I was suggesting as a solution to this issue. I'm well aware it's not legal, but a rule change could fix that fairly trivially.

I'd also disagree that it's a 100% software fix to lower voltage brownout, or, more accurately, that it should be fixed (And the 6.8-6.3 brownout on the 6V rails is more than likely hardware). To me the brownout is a feature meant to prevent controller shutdowns. While the 6.3V number may seem high, if you're drawing your battery down to 6.3V regularly it's not good and I'd rather have the control try to protect itself by disabling the large draws. Think of it as a soft failure. I'd rather have a soft failure than a dead robot.

To me, if I'm giving it voltage outside the input range then I'm not using the controller properly and any failures are my fault. The fact that the RoboRIO stays alive and tries to save itself as much as it does is nice.

[AdamHeard]

It is legal to replace the 5V rail going to sensors though. 971 implemented/championed this fix and a few teams ran it this year.

[Michael Hill]

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

$30 for a buck/boost regulator? Yeah, they'll put that in when they make the next version out of gold...</aloof> Quite simply, I'm sure they just stuck in an LDO and said "we're done with it". Don't forget that since they have an FPGA on board, they likely need several more voltage regulators. Yet If they decided to put in a decent switcher, it's going to be expensive and require expensive magnetics, which will take up more board space. Not to mention the fact that the packaging is low profile, so they may not even have enough height for a decent inductor. Switchers are going to be noisy (electrically) as well. If they were going to do more regulation, I would rather have it done on the PDP, since it's about half the price of a new RoboRio.

[Thad House]

The second part is what I was suggesting as a solution to this issue. I'm well aware it's not legal, but a rule change could fix that fairly trivially.

I'd also disagree that it's a 100% software fix to lower voltage brownout, or, more accurately, that it should be fixed (And the 6.8-6.3 brownout on the 6V rails is more than likely hardware). To me the brownout is a feature meant to prevent controller shutdowns. While the 6.3V number may seem high, if you're drawing your battery down to 6.3V regularly it's not good and I'd rather have the control try to protect itself by disabling the large draws. Think of it as a soft failure. I'd rather have a soft failure than a dead robot.

To me, if I'm giving it voltage outside the input range then I'm not using the controller properly and any failures are my fault. The fact that the RoboRIO stays alive and tries to save itself as much as it does is nice.

So then why need an external LDO? The os and code stay running down to 4.5v anyway.

And the virtual brownouts should be changeable by the fpga. I'd be surprised id they were not.

[MrRoboSteve]

Some thoughts:

1. It will always be possible for the roboRIO to have less power than it needs to run, so load shedding features will continue to be useful for the great majority of teams.

2. Might be possible for the roboRIO to tolerate longer periods of low voltage. This would be a software change. That said, robots operating for extended periods at 7 volts are visibly sick.

3. regulating the input power would require alternate means to detect battery voltage. Analog sidecar anyone?

4. As FTAA, the only way I knew a robot was in brownout was to go stand behind the drivers and watch the display. We need to make brownout on the robot more visible, to avoid situations like the one Joe describes. Ideas:
a. FMS field monitor should show "brownout count" statistic
b. DS software should have a brownout indicator that stays lit if there's been a brownout
c. DS stack light should change state for a second or so when there's a brownout

[AllenGregoryIV]

3. regulating the input power would require alternate means to detect battery voltage. Analog sidecar anyone?


The PDP is already required to be wired up with CAN. Pretty sure the PDP with CAN reports battery voltage just fine.

[jhersh]

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

Hi Tom,

I think this is a perfect example of where your robot has a better chance to operate reliably using the new functionality, but the control software needs to pay attention to the info available. Since you are not losing absolute position and the motor is disabled, the likely cause for the behavior you saw was a combination of integral windup and reading the sensor in a non-ratiometric way. Naturally as the voltage of the 5V supply drops the lower the resolution of the reading.

The roboRIO Power API will tell you what the voltage of the reference rail is so you can divide. There is also a new potentiometer API that does this for you. The Power API also tells you when the brown-out level has been crossed so you can pause your control loop until it has recovered. The Power API also tells you when the 5V supply has blacked out (though I did notice a bug in this in the LV API implementation that I'll fix for next season), so you can stop believing your sensor at that point. This will always be below the brown-out level, so your control loop should already be paused.

[jhersh]

And the virtual brownouts should be changeable by the fpga. I'd be surprised id they were not.

The level is actually set by a hardware comparator. However it is likely that a work around could be implemented in the FPGA if needed.

[Thad House]

The level is actually set by a hardware comparator. However it is likely that a work around could be implemented in the FPGA if needed.

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms. Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

[jhersh]

There are two sources of information on the brownout feature that have conflicting information.

http://wpilib.screenstepslive.com/s/4485/m/24166/l/289498?data-resolve-url=true&data-manual-id=24166

Page 7 of http://www.ni.com/pdf/manuals/374474a.pdf

The screen-steps page is accurate. I've filed a bug report about the User Manual. Thanks for pointing that out.

[Andrew Schreiber]

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms. Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

The PWM cutout, I'm actually happy about and hoping it will help end the stupid drivetrain wars we've been engaged in since 2005. The idea of "throw more power at it", while valid, gets annoying very quickly when 6CIM shifting drives smack into you from across the field.

[jhersh]

Really? So the next question is who made the decision to put the PWM shutoff so high? With things we saw in 2013 and 2014, 7V is a number pretty easily hit by a 6 CIM drive, even if it is just for a few ms.

The point of having it "high" at 6.8 V is to have some breathing room to get those loads turned off in time for the battery to recover before the controller blacks out. Since we don't have control over the PD channels we can't simply turn off the high loads. We have to ask that the controller stop. That takes time. This transition happens relatively quickly in human-perceivable time, so if your load is very high but your battery is not dead, you will likely not even notice that the brown-out happened. That is the goal and others have reported this. If as you say it dips for a few milliseconds, then your motor controller will only be disabled for a single cycle (5 ms on most PWM motor controllers). If it is near the end of a match and your battery has been abused most of that time, it will not recover as quickly, but the goal is to avoid the controller rebooting even in that case. I think the level is appropriate.

Also why ever drop the 5v and 3.3v rails? It seems like disabling those causes more problems for teams, and troubleshooting sensor drops are harder then pwm drops.

The three user supplies were originally considered a potentially "high load", around 22 W, (though practically that is not the case). As such those two were on the chopping block for brown out to help save the boost supply running the processor from faulting. During the design it was determined the 5 V and 3.3V supplies could not handle the upper end of the input voltage and still use the controller parts we selected, so it was decided that a fix for that would be to power them down-stream of the 6V supply to limit the input they see. All three supplies were also designed as buck supplies only. Since at that time they were all set to be disabled at 6.8V anyway, this was an appropriate solution.

Later (alpha 2) when we decided that 5 V and 3.3 V supplies should not disable until black out, they were still down stream of the 6 V supply. At this point the hardware was already being manufactured and the power supply topology could not be changed with acceptable risk and schedule impact. Being down stream of the 6 V supply meant that the input they see goes away completely when the 6 V supply controller enters a fault state due to insufficient input voltage. This happens at about 6.25 V. This means that if the motor controllers respond (starting at 6.8 V) before the input drops to 6.25 V, the sensor supplies will not black out.

[MrRoboSteve]

Updated:

1. It will always be possible for the roboRIO to have less power than it needs to run, so load shedding features will continue to be useful for the great majority of teams.

2. Might be possible for the roboRIO to tolerate longer periods of low voltage. That said, robots operating for extended periods at 7 volts are visibly sick.

3. As FTAA, the only way I knew a robot was in brownout was to go stand behind the drivers and watch the display. We need to make brownout on the robot more visible, to avoid situations like the one Joe describes. Ideas:
a. FMS field monitor should show "brownout count" statistic, separately showing stage 1 and 2
b. DS software should have a brownout indicator that stays lit if there's been a brownout
c. DS stack light should change state for a second or so when there's a brownout

4. It's difficult to change the voltage of the stage 2 brownout, because it's a function of the power supply architecture.

5. It would be interesting to collect DS logs of machines in brownout, to see if at the point of outage the voltage has a steep slope. This would help assess the utility of changes to how stage 1/2 brownouts work.

[Thad House]

The point of having it "high" at 6.8 V is to have some breathing room to get those loads turned off in time for the battery to recover before the controller blacks out. Since we don't have control over the PD channels we can't simply turn off the high loads. We have to ask that the controller stop. That takes time. This transition happens relatively quickly in human-perceivable time, so if your load is very high but your battery is not dead, you will likely not even notice that the brown-out happened. That is the goal and others have reported this. If as you say it dips for a few milliseconds, then your motor controller will only be disabled for a single cycle (5 ms on most PWM motor controllers). If it is near the end of a match and your battery has been abused most of that time, it will not recover as quickly, but the goal is to avoid the controller rebooting even in that case. I think the level is appropriate.



The three user supplies were originally considered a potentially "high load", around 22 W, (though practically that is not the case). As such those two were on the chopping block for brown out to help save the boost supply running the processor from faulting. During the design it was determined the 5 V and 3.3V supplies could not handle the upper end of the input voltage and still use the controller parts we selected, so it was decided that a fix for that would be to power them down-stream of the 6V supply to limit the input they see. All three supplies were also designed as buck supplies only. Since at that time they were all set to be disabled at 6.8V anyway, this was an appropriate solution.

Later (alpha 2) when we decided that 5 V and 3.3 V supplies should not disable until black out, they were still down stream of the 6 V supply. At this point the hardware was already being manufactured and the power supply topology could not be changed with acceptable risk and schedule impact. Being down stream of the 6 V supply meant that the input they see goes away completely when the 6 V supply controller enters a fault state due to insufficient input voltage. This happens at about 6.25 V. This means that if the motor controllers respond (starting at 6.8 V) before the input drops to 6.25 V, the sensor supplies will not black out.

If these drop happens, do the internal pullups still drop? Or are they ran off a seperate voltage regulator?

[jhersh]

If these drop happens, do the internal pullups still drop? Or are they ran off a seperate voltage regulator?

They do not drop. Separate regulator. The internal pulls on all DIO / I2C / SPI lines are connected to the internal 3.3 V supply that is fed by the boost supply in roboRIO.

[Greg McKaskle]

Brain dead, close to criminal, and really dumb?

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

1. Why was this feature included?
2. Does it address the targeted issue(s)?
3. Does it introduce new issues?
4. Can it be improved?

---------

1. Why was this feature included?
The brownout feature is intended to cut power to some circuits in order to maintain power to others deemed more critical. There is a limited supply of current, and without this feature you don't have brownouts but instead have blackouts/reboots of devices based on their power supply details.

The critical components were deemed to be the roboRIO processor and the radio. High current consumers are motor controllers and compressor. The roboRIO's integrated 6V, 5.5V, and 3.3V were incorporated as well.

This is a new feature that wasn't present with the previous PD or cRIO. The cRIO voltage limits were listed above, and as described, the PD board boosted several supplies. Yet teams with a high current draw and a weak battery could still reboot the cRIO and/or radio. This primary goal of this feature was to avoid reboots of the roboRIO cpu and radio.

2. Does it address the targeted issue(s)?
The roboRIO alpha and beta testing included "push" tests where the robots were intentionally run with the same battery and encouraged to push walls. They were browned out repeatedly in order to observe and test this feature. Tom Line's observations were made and investigated during these test. In general, the results were positive, but there are side-effects, and there were discussions about how to improve the feature to minimize those.

During the 2015 season, I was CSA at three events and helped some at champs. I saw teams with dozens of brownout events and I remember seeing one with over 250 brownouts in a single match. The team had a weak battery, long robot, sticky wheels, and would have rebooted easily and repeatedly with the previous system. With it, they survived the match and contributed -- some -- to their alliance. They knew the battery was weak and new what to do to avoid that portion of the problem. So yeah, it seems to work for the intended problem.

But roboRIO and radio reboots still happen, and more than I'd like to see. I was able to investigate a few dozen of these at each regional and in 70% of the cases it was due to a loose connection at either the main breaker or the battery leads. A few were clearly due to cable insertion at the roboRIO or PD. Others were harder to pin down but could be due to fuse insertion on the PD. It may be possible for the brownout to go black before the motor controllers and compressor comply, but we need more usage to know how common this is.

3. Does it introduce new issues?
Yes.
To generalize, if teams implement more sophisticated feedback control and do not take brownouts into account, they will see new issues caused by the brownout feature. For example, if a PID is commanding a motor controller to 40%, but the motor output is actually at 0% due to brownout, integral error can accumulate and cause a bump when the motor operation is enabled again. If the sensor value droops, due to input voltage supply, this can compound the issue. If the sensor power supplied by the roboRIO goes into protection mode, then the sensor value isn't valid and this gibberish can cause even more severe control issues. Tom Line's comments above were caused by a combination of these effects. And as other beta teams pointed out, incremental sensors used for absolute measurement will now have a bias and/or need to be recalibrated.

There is another issue, and that is that the system now effectively limits the current usage in an additional way. The main breaker, individual breakers, and brownout features are all imposing limits on the available current. Teams familiar with the first two may feel like a third wasn't needed, and will once again have to discover the edge of the new envelope and how to push the system. Published specs and time with the system will address this.

4. Can it be improved?
Yes.
This is still underway. During beta, the FPGA stopped disabling the 5V and 3.3V sensor rail intentionally. It disables the 6V used by servos, the PWM values are set to zero or neutral, and the CAN devices are notified and should stop using large amounts of current. There are small tweaks that will attempt to shorten the brownout response of the high current devices but maintain solenoid states, etc. The PID functions and tutorials will include brownout awareness and can even incorporate sensor power loss awareness. Using incremental sensors for absolute measurements was already tricky, and some teams have chosen to supply their own sensor power via the VDM. All of these should improve the gap between brownout and sensor rail dropout and the side-effects of it if/when it happens.

As for pushing the envelope, the power API and PD features should make this much easier and more adaptive than it used to be.

The feedback that a robot is browning out also needs to be improved. The DS counts them, adds them to the log, puts a big red indicator behind the battery readout, and sends the into the field monitor. The indication on the field monitor is a momentary red blink on the voltage box. IMO there are many improvements to be made there.

Conclusion?
The team with 250 brownouts, the team with just one or two, they are able to finish the match instead of rebooting three or four times. They will have a better experience. The 1000's of teams who could use better mentoring still need to tighten the nuts on their main breaker, attach battery leads more robustly, and insert the fuses into the PD, but this feature seems to help them far more than it hurts. It has already been improved from the alpha/beta period, additional tweaks are underway, and we are open to questions and feedback. Insults ... I've been called worse, but they aren't so productive.

I like the OP's tone of this thread. So what are the specifics?
Greg McKaskle

[Tom Line]

Hi Tom,

I think this is a perfect example of where your robot has a better chance to operate reliably using the new functionality, but the control software needs to pay attention to the info available. Since you are not losing absolute position and the motor is disabled, the likely cause for the behavior you saw was a combination of integral windup and reading the sensor in a non-ratiometric way. Naturally as the voltage of the 5V supply drops the lower the resolution of the reading.

The roboRIO Power API will tell you what the voltage of the reference rail is so you can divide. There is also a new potentiometer API that does this for you. The Power API also tells you when the brown-out level has been crossed so you can pause your control loop until it has recovered. The Power API also tells you when the 5V supply has blacked out (though I did notice a bug in this in the LV API implementation that I'll fix for next season), so you can stop believing your sensor at that point. This will always be below the brown-out level, so your control loop should already be paused.

I just emailed myself this at home so I can implement it over the summer. We never had a problem with it during competition but having the failover built into VI's will make it pretty easy to implement. Thanks Joe.

[MrRoboSteve]

The indication on the field monitor is a momentary red blink on the voltage box.

I never noticed this -- will keep an eye out for it at Saturday's event.

[jhersh]

Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing.

That's the result of a poor behavior of the FMS. It should be improved next year to not cause auto to restart.

[Joe Johnson]

Well now that we have the emotional venting out of the way, let's see if we can look at this in a more productive way.

<snip>
Insults ... I've been called worse, but they aren't so productive.

<snip>
Greg McKaskle


Greg,

Was I being hyperbolic with the "criminal" comments? sure. Calling it "really dumb"? No, not at all. If anything I would use stronger language.

I am not a EE by training but I have years and years of robotics experience both inside and outside of FIRST, as a hobbyist and as a robotic professional. These years of experience have taught me that you just don't scrimp when it comes to protecting the power supplies that keep your processors/coprocessors, sensors, and communications link alive.

That is a general rule. When it comes to FIRST FRC control system, where we know there is a battery that can be easily overwhelmed in terms of amps it can source compared to the amps requested by the motors allowed, the designers should take even more precautions when it comes to protecting control system from the inevitable low battery conditions. FIRST can put as many "buyer beware" signs up as it wants. It doesn't absolve them from the responsibility to provide a control system worthy of the name.

Based on the results I have seen using the RoboRIO and the discussion about the trade offs people made during the design, it is obvious to me that FIRST (and NI) did not take power management as seriously as they should have when as they were designing the new control system.

All the defense of 7V Brownout is missing the point. As are discussions about what last year's system would have done. This was a chance to design a system from scratch.

Yes, once you decide that it is okay for the RoboRIO to reboot at 4.5V and that you are not going to keep the 5V and 3.3V rails alive that it is okay to power them off the 6V line, then yes, you eventually get to a Brownout Voltage of 7V. Sure, a brownout is better than a reboot, but can we all agree that few brownouts are better and no brownouts are best?

For all that, I really don't mind that the PWMs are disabled at a certain voltage. Do I wish it were a bit lower? Sure, but I bow to the reality that ultimately it is better to have the motors turned off than to lose more critical thinking and sensing functions. I said brownouts but I was really talking about the larger issue of preserving higher brain function down to a voltage sufficiently low that it never happens.

The original sin in all this is that FIRST didn't take power management on critical systems as seriously as it should (as evidenced by the initial choice to not to even protect the 5V and 3.3V rails which power sensors and co-processors).

Yes, I am calling FIRST (and NI) out here. This could have and should have been a complete non-issue. It was foreseeable and it was avoidable.

I have a question that I will pose to every person that defends the current status quo: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

Saving a few bucks by not providing this feature on a control system that costs literally 100s of dollars that 1000s of teams will use to control robots that each team will invest many 1000s of hours and many 1000s of dollars making is... well... I don't want to be called out for making insults again so I'll leave it to the readers to decide what they would call it once they understand how easily and inexpensively this feature could have been provided.

Sincerely,

Dr. Joe J.

*I am not 100% what the 6V supply powers. If it's just the servo power, we could I would be willing let the 6V go -- servos going dark are no worse than motors turning off.

[Greg McKaskle]

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

I'm sure my post came off more defensive than I intended, but then again, a rebuttal of almost any sort easily tends in that direction. I think your questions and goals make sense. It will be interesting to see what other's would have done. If there is information that would make the alternate design exercise easier or more factual, I'll do what I can to get the info to you. In that spirit, here are the links (https://decibel.ni.com/content/docs/DOC-30419) to the specs, user manual, etc.

Here is a page from team 358 that did some pretty extensive independent testing of the entire system and details what happens as the voltage does the limbo. They took the roboRIO to under 4V before it rebooted and the radio under 3.5. Not your 3.3V goal, but in the neighborhood. And as documented and designed, it is staged, which introduces the brownout condition. Some things start to disappear at about 6.8. During alpha I believe it was often rounded to 7V.
Beta team 358 Testing of Power Management (http://team358.org/files/programming/ControlSystem2015-2019/BetaResults.php#powermgt)

Here is a direct link to the documentation of the power management stages.
roboRIO User Manual (http://www.ni.com/pdf/manuals/374474a.pdf) and the details are on page 6 and 7.

Just to round out the power discussion, the other requirement, the one that was given a higher priority, was to allow for any pin to short to any other pin with no damage. We regularly rake a screwdriver across exposed pins to demo this. The power tab of the DS will show you which rail is shorted. So detection and protection against over voltage and shorts was deemed critical. The system will also tolerate being plugged into the older 24V PD without damage. It won't operate, but isn't damaged and indicates the issue via LEDs. By the way, having USB cables with exposed ground shield on a robot with high current 12V exposed connectors is also challenging.

Oh, and 6V is used to power servos.

Greg McKaskle

[Joe Johnson]

Full disclosure -- also not an EE, that was only my minor. Also not a HW designer much less a power supply guru.

<snip>

Greg McKaskle

Greg,
As to the fact that the actual RoboRIO blackout voltage was show to be 3.5V in one case, while I want to say good things about this, I just can't. The Spec you link to says 4.5V, I suppose that is what it was designed to be. One particular example showing a lower actual drop out voltage doesn't change that. Specs are about what happens to the worst built RoboRIO on its worst day. Am I happy that one particular data point is suggestive that there is design margin in that 4.5V? Sure. Do I wish that we were talking about a designed blackout voltage of 3.3V (or lower -- recall that as an ME my opening ask was for the RoboRIO and other key equipment stay alive down to 1V and was only talked off the ledge by my EE friends) and were talking about actual blackout voltages in the sub 3V range? Yes, yes I do.

BUT the RoboRIO reboot voltage is only one piece of the puzzle. The control system is a SYSTEM. Its nice to have a RoboRIO not rebooting but that is not enough. I need to have radio communications alive and well. I need to have my incremental encoders not lose their positions. I need to have my MEMS IMU keep its position for a the whole match (I'm looking at you navX). I need to have my onboard Beaglebone coprocessor not reboot during a match. I need to have sensors doing their thing. ...

It was FIRST's task to think on a system level and provide a system that just works.

As to the other things that FIRST/NI did right, I am not disputing this. There are a ton of things that FIRST/NI did really well. Protection against reversed and misrouted power/ground IS important and, as far as I know, the RoboRIO does a good job of this. Nice job on that front.

But that doesn't free the FIRST from the responsibility to manage power effectively. On this front, FIRST has a lot to answer for.

And now I will finish with the question I am promised to ask until it gets answered: How much would it have cost if FROM THE START OF A CLEAN SHEET DESIGN to have the RoboRIO, the Radio, 6V*, 5V & 3.3V rails all stay alive down to a Battery voltage of 3.3V?

I have a pretty good idea what the answer is but I would like to hear those who made the decision not to provide give their estimates.

Dr. Joe J.

[Greg McKaskle]

OK. Time for me to make another post that will come off sounding more defensive that I intend for it to.

The link to the user manual and spec sheet of the roboRIO was intended to clarify the design and spec'ed behavior with low input power. The link to team 358's results were to intended to show how an alpha team tested the more complete system that included USB devices, rail power, VRM, etc. I'm not trying to sell you a used car or use a single team's testing to fool anyone. I'm giving the info I know of for the roboRIO and system elements that were delivered. That may help you or others on this alternate design quest.

As for the other background on short protection -- I described the additional protection mechanisms so alternate designs would be apples-to-apples.

As I described in my background, I don't have the chops to do the current or alternate power supply design. But I do volunteer as a team mentor and CSA, so that I can see first-hand how the elements are used and how they perform. From that perspective, as a customer and support engineer, I do not see the need for the 1V or even 3.3V input requirement.

Greg McKaskle

[tStano]

I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

Can anyone explain why the backup battery went away? I've wasn't on a team in IFI days, but I maintain a robot with an IFI control system. To me, this seems like the best possible solution to the problem.



Also, I feel like if you set CAN low and PWM to zero and turn relays off, you kill all the motors, so unless your battery is literally useless, any other power you can draw(unless something's shorting) will be plenty to keep you above 4.5v. Even if you're using all the servos and sensors you possible can. I feel like it still would be a trivial software(unless its on ROM) change to not make the roborio stop doing 5v and 3.3v rail stuff down to at the very least 5v.

Another quite simple option would be allowing a team to supply a regulator(which fits some sort of inspection standard) to feed the roboRIO.

[Al Skierkiewicz]

Sorry I didn't weigh in on this earlier...
The original PD and cRio voltage dropout was lifted from automotive industry papers on electrical devices on vehicles. (One of those is a Daimler/Chrysler paper) In those recommendations, devices should be reliable down to 6 volts and are classed by importance based on the functionality of the device. (used only during cranking, used only during engine run, etc.)
The original control system wanted to be better than that knowing that FRC is a very harsh environment. The power supplies on the side car dropped out at around 5.5 volts. When that level was reached (if the input drop was long enough that the capacitors on the side car finally dropped below 5.5 volts) the sidecar would cease providing PWM output. In terms of a fully charged battery and ideal conditions, a robot load current of about 600 amps would drop enough voltage across the internal resistance of the battery to achieve that. (That is in keeping with the manufacturer's maximum current spec for our batteries.) If we add to the equation, the real world voltage drops across wiring, main breaker and typical wiring, that level can be achieved with a load current of approx. 500 amps (or less depending on robot design). The power supply for the PD ceased functioning at approx. 4.5 volts. It is pretty easy to see that the big CIM motors at 131 amps stall could easily approach the dropout specs with only a four motor drive. (In reality, with good wiring technique, a CIM motor in stall, driven by a motor controller, can draw between 116 and 120 amps.) It appears that the RoboRio spec and brownout behavior are consistent with those electrical considerations. Remember that long wire runs, poor crimp and termination techniques will degrade the available voltage at the RoboRio and other systems.
As to the backup battery on the old IFI controller, that was a design choice made by IFI to overcome the brownout conditions at the time. Remember that the controller was an old technology (compared to today) and would take a while to boot up and start running code. To keep the controller up and running, a circuit that switched the CPU (and attached circuitry) to battery supply during brownouts was a good choice.

[crake]

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

[slibert]

I'm putting on my moderator hat for a second.

This is be to a productive forum with constructive dialog. Please do not deviate from that.

Ok... now for my control system hat...

I'd like to say a few things. First off I wrote the brown out spec. It was reviewed by control system members and it was tested against real-world data before it was accepted and implemented. Is it perfect? No - there are things that could have been better. For instance, user rails probably don't need to be shut down as aggressively as they are. Also even though teams enter/exit brownout without even knowing (see Greg's post on this - the system does actually work), the interactions with the FMS can sometimes make it worse. Some of these things (FMS) can be adjusted, while others (user rail behavior) can not.

Greg, Joe, myself, and others are heavily committed to providing a reliable control system that enables team success. We are continuously looking at field and robot performance as well as community feedback to determine ways to make improvements.

Thanks, Chris. As the founder of Kauai Labs, I have perhaps a unique perspective on this, as a result of developing/supporting the navX MXP IMU. Key goals included an easy-to-use, plug-n-play product useful to both upper-tier teams as well as beginning teams. Our focus is to keep improving things til we reach those goals.

The MXP connector was a great idea that really helped enable plug-n-play; however, this particular brownout issue negatively impacts these goals.

The primary reason is that the navX MXP (and certain other sensors, for sure) needs to manage state; in the case of the navX MXP there's some non-trivial run-time calibration state that's key. As noted by others, MXP products (Revduino) and other non-MXP products (e.g, the Beaglebone) are going to encounter similar issues.

In this case the specific issue is the 5 and 3.3V user rail supply rails on the MXP connector; voltage on these is not maintained in the case of a stage 2 brownout. The navX MXP has an onboard LDO that regulates the 5V MXP rail down to 3.3V to power a 32-bit ARM microcontroller; the board consumes a max of 250 mA.

Thanks to Joe H., who I met at CVR and pointed out the boost/buck reg attached to the USB port. Fortunately the navX MXP has a separate USB-based power supply on the navX MXP and the logic to cleanly switch between supplies w/out glitch. That's documented in our best practices (http://www.pdocs.kauailabs.com/navx-mxp/guidance/best-practices/) now.

But that really isn't plug-n-play, to have this extra USB cable connecting the RoboRio to the navX MXP secondary power supply - just to avoid the brownout.

So the recommendation (OK, plea) is to maintain the power to the 5 and 3.3V MXP user rail during stage 2 brownout.

If that is not possible, we (MXP developers) need recommendations on how to best deal with this issue. Additionally, I think continuing the status quo on this will have the effect of diminishing the potential of the MXP interface. We are seeing tremendous growth in digital "smart" sensors/ICs w/onboard processing and state, and would like to help make them available to the FIRST community. I can confidently say that the robot of 3 years from now will have an even greater need than today.

[IKE]

Sorry I didn't weigh in on this earlier...
...snip.... Remember that long wire runs, poor crimp and termination techniques will degrade the available voltage at the RoboRio and other systems.
...snip...

Most of the brownouts I encountered in an LRI role highlighted non-conformance to one, but usually more than one of the practices Al outlined. The most common scenario was a KOP chassis with 4 CIMS, but 6" wheels with relatively poorly crimped heavy gauge wires to battery/PD/Mainbreaker or all 3. I also saw a few with damage on the AB-50 connectors due to charging with Aligaotr clips. All of these problems were in conjunction to only running a couple batteries all weekend (thus very little recharge time).

The brownouts were frustrating for the teams, but weren't terribly hard to diagnose. Only the loose terminals and charging were easy to fix. Talking teams out of 6" wheels are re-gearing the drivetrains was pretty tough.

For those teams, I am not really sure any back-up anything would really help much as it would just let them abuse a battery even more without us finding out they have a problem. I do think it would be worth discussing some back-up means for those that were just getting into some of the other scenarios that have been discussed. It was just not my experience at most competitions I supported.

[tkeil]

Hello. Power electronics engineer here. Somebody directed me to this thread and I thought I'd step in for a minute.

Several of the mentors as well as NI folks have already done a great job explaining this limitation and linking to helpful documentation for diagnosis and solutions. I don't have much to add on this front, but I would like to talk about what goes into selecting an appropriate input voltage from a technical perspective.

It's very easy for someone to be critical and say 'why doesn't it just work perfectly down to <X> volts?!' where <X> is some arbitrarily low voltage that some individual expects to never hit in their particular system implementation. The reason why is that this is a complex embedded controller and there are many trade-offs. In this case the main trade-off is thermal dissipation.

The roboRIO regulates 45W of power maximum. This is a lot for a small, plastic, convection cooled device with no open vents. The power supplies were a major concern for this design since with 45W input they account for a large percentage of the internally dissipated power.

Some of the best power supplies in the world can get >95% efficiency but these are nearly all power supplies with a fixed input voltage and a fixed output voltage. As soon as the power supply has to work over a wide input range, that efficiency falls because the power supply can no longer work at its most efficient operating point all the time. The roboRIO supports >2x input voltage variation. We can get better than 90% efficiency with this wide input range but the wider it is, the worse the losses get.

roboRIO is designed specifically for the FIRST competition, but it leverages many sections of circuits from other embedded controllers. It's designed by the same team that makes our professional cSeries controllers. You should think of roboRIO as a professional embedded controller with a protective cushion around it.

Fully around 1/3 of the circuity in the roboRIO is to protect the device from accidental damage and make it easy for the users to diagnose problems. Our highest priorities of course were preventing failures that permanently damage the device. I bring this up because some of these protections (for example reverse voltage protection) have a required minimum voltage and lie in the power path. These circuits are often leveraged from other designs and while they could be changed, it's costly to design and verify.

Some of these protection devices need to be in the direct current paths inside the device. This becomes a huge problem when your currents get large.

Let's look at a quick hypothetical where the input voltage minimum would be 4.5V:

The roboRIO needs to regulate 45W. If the input voltage drops, the current gets higher. Resistive losses are calculated as P=i^2*R. Consider that the current into the input connector of the roboRIO @ 7Vin is ~6.42A. If the minimum input voltage had been 4.5V that current would be ~10A. When you square those terms thats about a 2.5x increase in resistive losses through the whole input section of the power distribution path and switches in the power supplies.

Before you even get to the first power supply in the design, the power has to go through the input connector, the fuses, the common-mode choke, more noise filtering, reverse voltage and reverse current protection, and finally an input current sense shunt. The losses in all these devices would be 2.5x greater in this fictional example of allowing full power down to 4.5V.

Many (most?) of these devices would also have to be larger to support the current and thermal dissipation. For instance, 10A is far over the rating of the input voltage connectors. [For a laugh you can consider the device operating down to 1V in which case the current would be 45A and the input connector terminals would be around the size of an AA battery each.]

Anyway, when you combine the extra losses in the main input supplies with the extra losses in the resistive elements (and the increased component sizes) you will quickly lose your form factor and ability to use a plastic housing. If we had to go to a metal housing and possibly one with heat spreaders, you're talking about increasing the total device COGs by roughly 30% - 50% and increasing the form factor and weight.

Input power and voltage range is a big deal. Myself and others at NI think about it a lot and then over and over again for each new design. As you guys all know as good engineers and scientifically minded folks, everything's a trade off.

Thanks for your time volunteering with the students.

Tim

[Bryan Herbst]

Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot.

We discovered this behavior with some teams at 10,000 lakes this year as well- when a robot is disabled and subsequently re-enabled during autonomous, autonomous is restarted.

In this scenario, you really only have two options code-wise: either restart auto or continue running the disabled loop until teleop starts. Picking up where the code left off in auto isn't a viable option unless your team explicitly programs a recovery mechanism.

Whether or not restarting or staying disabled is the right choice is a fair point for discussion, though my guess is that since recovery is a relatively viable option, the decision to restart auto was intentional. Perhaps this behavior could also be better publicized.

I'm also not sure why a brownout triggers disabled mode (I would think it could continue attempting to run during a brownout).

[Greg McKaskle]

Your intuition is correct. There is no need to disable for comms loss or brownout and once that is corrected, the double auto should be a thing of the past.

Greg McKaskle

[Alan Anderson]

I'm also not sure why a brownout triggers disabled mode (I would think it could continue attempting to run during a brownout).

The whole point of a brownout is to disable things so the load on the battery is reduced.

[Pault]

The whole point of a brownout is to disable things so the load on the battery is reduced.

The issue that 246 and other teams experienced was not that the brownouts were disabling "things," but that in reaction to the brownout the FMS would command the robot to disable. This is much different, because the robot does not see it as just a standard brownout, but as if someone manually disabled and then re-enabled autonomous (or tele-op), and will therefore restart autonomous (or tele-op).

If teams have any fear of this happening at their offseason events, I would recommend putting a flag into your code that will only allow autonomous to initialize once. That is what we did, and it prevented this bug from breaking our robot a second time.

[jhersh]

The whole point of a brownout is to disable things so the load on the battery is reduced.

Disabling motors is a totally different thing than being commanded to disable by the field. The FMS is wrong and its behavior will be changed.

[Alan Anderson]

The issue that 246 and other teams experienced was not that the brownouts were disabling "things," but that in reaction to the brownout the FMS would command the robot to disable.

Disabling motors is a totally different thing than being commanded to disable by the field. The FMS is wrong and its behavior will be changed.

Whoops. I missed the fact that Tanis was talking about a software-commanded "disabled mode" instead of the local disabling of actuator outputs. I was not aware that the FMS was even in the loop for brownout behavior.

I'm also not sure why a brownout triggers disabled mode (I would think it could continue attempting to run during a brownout).

I'm still not fully understanding the reference to "continue attempting to run", though.

[Greg McKaskle]

The FMS is aware of brownout in order to display to the field display. I took Tanis's statement to mean, don't run disabled code, run the code based on the match transitions. And the ability for the robots to actuate is ANDed with the brownout and coms status.

Greg McKaskle

[Bryan Herbst]

The FMS is aware of brownout in order to display to the field display. I took Tanis's statement to mean, don't run disabled code, run the code based on the match transitions. And the ability for the robots to actuate is ANDed with the brownout and coms status.

Greg McKaskle

Bingo. If the control system is in a brownout condition, the code should continue running as though nothing had ever happened. Some of the commands sent to certain devices might not actually cause anything to happen (e.g. if the speed controllers have been disabled), but the code should continue to run.

As for comms loss causing the robots to be disabled- I am in favor of that. Without comms, you can't e-stop your robot. That means that if you lose comms and your robot becomes a danger to itself or to people, you can't stop it. Disabling the robot when it can no longer communicate with the driver station sounds like an appropriate action to take.

[jhersh]

As for comms loss causing the robots to be disabled- I am in favor of that. Without comms, you can't e-stop your robot. That means that if you lose comms and your robot becomes a danger to itself or to people, you can't stop it. Disabling the robot when it can no longer communicate with the driver station sounds like an appropriate action to take.

That's contradictory and not how it will work. And anyway, if you have no comms how does FMS disabling do any good. You have no comms.

No, we will continue to have the designed behavior that the robot will override (locally) the enable signal when a situation deems it (no comms and brown out) and the DS and FMS should only use that as status for humans (if they even have comms to receive that status).

The FMS broke this year and it will be fixed.

[Joe Johnson]

Hello. Power electronics engineer here. Somebody directed me to this thread and I thought I'd step in for a minute.

Several of the mentors as well as NI folks have already done a great job explaining this limitation and linking to helpful documentation for diagnosis and solutions. I don't have much to add on this front, but I would like to talk about what goes into selecting an appropriate input voltage from a technical perspective.

It's very easy for someone to be critical and say 'why doesn't it just work perfectly down to <X> volts?!' where <X> is some arbitrarily low voltage that some individual expects to never hit in their particular system implementation. The reason why is that this is a complex embedded controller and there are many trade-offs. In this case the main trade-off is thermal dissipation.

The roboRIO regulates 45W of power maximum. This is a lot for a small, plastic, convection cooled device with no open vents. The power supplies were a major concern for this design since with 45W input they account for a large percentage of the internally dissipated power.

Some of the best power supplies in the world can get >95% efficiency but these are nearly all power supplies with a fixed input voltage and a fixed output voltage. As soon as the power supply has to work over a wide input range, that efficiency falls because the power supply can no longer work at its most efficient operating point all the time. The roboRIO supports >2x input voltage variation. We can get better than 90% efficiency with this wide input range but the wider it is, the worse the losses get.

roboRIO is designed specifically for the FIRST competition, but it leverages many sections of circuits from other embedded controllers. It's designed by the same team that makes our professional cSeries controllers. You should think of roboRIO as a professional embedded controller with a protective cushion around it.

Fully around 1/3 of the circuity in the roboRIO is to protect the device from accidental damage and make it easy for the users to diagnose problems. Our highest priorities of course were preventing failures that permanently damage the device. I bring this up because some of these protections (for example reverse voltage protection) have a required minimum voltage and lie in the power path. These circuits are often leveraged from other designs and while they could be changed, it's costly to design and verify.

Some of these protection devices need to be in the direct current paths inside the device. This becomes a huge problem when your currents get large.

Let's look at a quick hypothetical where the input voltage minimum would be 4.5V:

The roboRIO needs to regulate 45W. If the input voltage drops, the current gets higher. Resistive losses are calculated as P=i^2*R. Consider that the current into the input connector of the roboRIO @ 7Vin is ~6.42A. If the minimum input voltage had been 4.5V that current would be ~10A. When you square those terms thats about a 2.5x increase in resistive losses through the whole input section of the power distribution path and switches in the power supplies.

Before you even get to the first power supply in the design, the power has to go through the input connector, the fuses, the common-mode choke, more noise filtering, reverse voltage and reverse current protection, and finally an input current sense shunt. The losses in all these devices would be 2.5x greater in this fictional example of allowing full power down to 4.5V.

Many (most?) of these devices would also have to be larger to support the current and thermal dissipation. For instance, 10A is far over the rating of the input voltage connectors. [For a laugh you can consider the device operating down to 1V in which case the current would be 45A and the input connector terminals would be around the size of an AA battery each.]

Anyway, when you combine the extra losses in the main input supplies with the extra losses in the resistive elements (and the increased component sizes) you will quickly lose your form factor and ability to use a plastic housing. If we had to go to a metal housing and possibly one with heat spreaders, you're talking about increasing the total device COGs by roughly 30% - 50% and increasing the form factor and weight.

Input power and voltage range is a big deal. Myself and others at NI think about it a lot and then over and over again for each new design. As you guys all know as good engineers and scientifically minded folks, everything's a trade off.

Thanks for your time volunteering with the students.

Tim

Tim,
I appreciate that there are a lot of constraints and trade offs in every design but at the end of the day it has to be suitable for the intended use.

I have circled back with a number of very experienced FIRST folks from teams with many years of FIRST experience. I'm confirmed in my view that given the FRC battery and the motors that FIRST allows that a 4.5V reboot of the CPU was a poor design choice that was made worse by allowing the 5V and 3.3V rails to away at above 6V making sensors and coprocessors go dark.

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

Dr. Joe J.

[Andrew Schreiber]

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

Dr. Joe J.

254 probably has more data than I do because they run closer to the edge, but I can tell you that in a season like 2014, when we DIDN'T have a short in our wiring, 125 rarely went below 6V despite some very aggressive gearing in both our drivetrain and our launching system. The only times (I recall) we had serious current draw* issues were when a vastly over-inflated ball jammed in our intake, the choo-choo was loading, and we were driving aggressively.

That being said, wasn't it you who coined the term "drivetrain wars" all those years back when we first started getting CIMs? I just view this as a good way to end those. EDIT: My memory DIDN'T fail me. http://www.chiefdelphi.com/forums/showpost.php?p=581779&postcount=1



* This ignores that we spent the first half of the season with a short that was causing us to die on the field regularly.

[MrRoboSteve]

On the other hand, a lot of people are making claims that the system basically works based on the 2015 experience. I predict that when we go back to a more traditional FIRST FRC season, there will be a ton of folks who are unhappy with these design decisions. In the mean time, I don't need to argue about it. The system is what it is.

We're in agreement that the verdict won't be in until we have a game that demands higher current. This year wasn't that game.

[Tom Line]

Ok. So like good engineers, let's solve the potential problem. is it possible to design a circuit such that it can be put in line with the crio power wiring, plug into a battery pack, and protect the crio voltage supply?

[Jared Russell]

254 probably has more data than I do because they run closer to the edge, but I can tell you that in a season like 2014, when we DIDN'T have a short in our wiring, 125 rarely went below 6V despite some very aggressive gearing in both our drivetrain and our launching system. The only times (I recall) we had serious current draw* issues were when a vastly over-inflated ball jammed in our intake, the choo-choo was loading, and we were driving aggressively.

That being said, wasn't it you who coined the term "drivetrain wars" all those years back when we first started getting CIMs? I just view this as a good way to end those. EDIT: My memory DIDN'T fail me. http://www.chiefdelphi.com/forums/showpost.php?p=581779&postcount=1



* This ignores that we spent the first half of the season with a short that was causing us to die on the field regularly.

I'd much rather see the drivetrain wars moderated by game design (one of the only things I liked about Recycle Rush, though 2012 and 2013 did a great job of this as well).

Failing that, I favor limiting the number of drive motors, a speed limit, wheel restriction, etc., rather than brownouts. Brownouts can be insidious to deal with (if you have any control loops), and be next to impossible to completely avoid (the dynamics of any FRC robot frame mean that instantaneously, your wheels may be loaded really unfavorably when driving around the field). An FRC robot that is designed (mechanically) to be incapable of dropping below 6.8V would be a fairly uninspiring robot to watch.

Forgive my ignorance on such things, but is there any way to default the motor brownout to 6.8V, but allow teams to do something else "if they (think) they know what they are doing"? For example, I could see doing prioritized load shedding (ex. first drive, then intake, but position-controlled elevator last) as a viable way to deal with this, but there isn't really any headroom above 6.8V with which to implement such a scheme.

[Greg McKaskle]

Forgive my ignorance on such things, but is there any way to ...

Not yet, but it has been discussed and is technically feasible. It would be a way for teams to prioritize current supply, but also a way to bypass the protection and to blackout some portion of the robot. So, it is not a magic bullet.

Greg McKaskle

[Joe Johnson]

I'd much rather see the drivetrain wars moderated by game design (one of the only things I liked about Recycle Rush, though 2012 and 2013 did a great job of this as well).



I am with you as long as by game design you are not talking about making the refs enforce more and more critical pinning rules. It makes me cringe whenever I hear the announcer saying things like, "Ok. Good match. Now let's see if penalties will change the outcome..."