Brownout behavior - alternative design goals

[MrRoboSteve]

In another thread:

Originally Posted by Joe Johnson

*don't get me started about FIRST's decision to allow the NI folks to allow the RoboRIO loose its mind at just under 7V -- this is something close to criminal. Really... this was a really dumb design direction given the customer base and the intended use of the RoboRIO.

FWIW, this year I occasionally saw* robots with low battery enter brownout. These were always teams with bad/poorly charged batteries. For robots that rebooted, I think all of them had a wiring related root cause. Not to say that it didn't happen, but I don't think I ever saw it.

The purpose of this thread is to try to reverse engineer the requirements around the brownout feature, and discuss alternative requirements that might lead to a different outcome.

Here's what I think the requirements were for the current feature:

1. Provide as much telemetry from the robot as possible during low voltage states.
2. Avoid roboRIO reboots during matches.
   
3. Telemetry should show when bad robot behavior is caused by low voltage.

There are two sources of information on the brownout feature that have conflicting information.

http://wpilib.screenstepslive.com/s/...anual-id=24166

Page 7 of http://www.ni.com/pdf/manuals/374474a.pdf

* as FTAA at events with a total of 120 robots (4% sample with some overlap)

[AdamHeard]

Most teams aren't using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn't show up as symptom.

[Joe Johnson]

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

[Tom Line]

Originally Posted by AdamHeard

Most teams aren't using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn't show up as symptom.

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

[Thad House]

Originally Posted by Joe Johnson

FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.

Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.

More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, ...) could stay alive to something ridiculously low making this problem disappear for good.

How low? I would go for 1V (which is possible but could get expensive) but I've heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.

Dr. Joe J.

I would even say 5V is reasonable. And easily doable with the current system, because we know it doesn't reboot until about 4.5v. I would still be OK if they system shut of the PWMs at 7v, but left everything else on until 5v. This year wasn't as bad for brownouts because 6 CIM drives were not advantageous, but if next year comes around and 6 CIM drives are king again, teams are going to start learning real quick about that 7v drop out if it stays.

[Michael Hill]

While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It's basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget...maybe it's just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it's not needed, use thicker power wire, run the compressor at strategic times, etc.). I'm not saying I would like a lower brownout voltage, but I have a hard time thinking there's anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.

[Jon Stratis]

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

[Andrew Schreiber]

Originally Posted by Jon Stratis

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

The CRIO required 19-30V and the CRIO II (4 Slot) was 9-30V. The PDB would continue to provide 24V to the CRIO down to 4.5V per the spec sheet. (And in practice, we regularly saw drops to 5V in 2014 due to an intermittent short, anything less and it cut out logging)

The RoboRIO is 7-16V but instead of going through any sort of converter the PDP feeds VBat.

[Joe Johnson]

Originally Posted by Michael Hill

While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It's basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget...maybe it's just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it's not needed, use thicker power wire, run the compressor at strategic times, etc.). I'm not saying I would like a lower brownout voltage, but I have a hard time thinking there's anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.

If this were the "real world," I would say, sure, one more constraint, add it to the list. BUT it isn't. This is a system who's job is to control a robot that plays a FIRST FRC game. FIRST was painted into a corner with the cRIO. They had insane time pressures and the had to pretty much take what they could get.

But the RoboRIO? It was designed SPECIFICALLY for FIRST FRC. It was a huge oversight to have anything important loose power, ever.

Another problem is that FIRST's system is SO COMPLEX that it is basically impossible for teams to duplicate what happens out on the field in their pit or even on the practice field.

I am not saying that it would have made that much of a difference but this whole brown out thing cost us in St. Louis. Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing. Half of our Qualifying Matches were gone before we got a reasonable answer as to what the problem was and had a work around. It seems to be something related to our robot pulling the voltage below 7V for a small fraction of a second at one point in our auton.

Again, so, yes, shame on us for not realizing that we were close to the edge for 4 prior tournaments where we never saw this problem. But also shame on FIRST for having an unnecessary edge for us to fall off to begin with and for not providing a person with the time and the knowledge to diagnose tricky problems such as ours (had it not been for Mike Copioli and one of his minions from CTRE, we'd still be in the dark about it).

So, that is my team's experience. If you're not on our team, it is easy to say, "Who cares?" Except, I am sure that there were many other teams that ran into problems as well.

Bottom line, I think this is the bad fruit that grew from a bad seed. FIRST should have made this a non-issue with the new controller.

Dr. Joe J.

[Andrew Schreiber]

Originally Posted by Joe Johnson

If this were the "real world," I would say, sure, one more constraint, add it to the list. BUT it isn't. This is a system who's job is to control a robot that plays a FIRST FRC game. FIRST was painted into a corner with the cRIO. They had insane time pressures and the had to pretty much take what they could get.

But the RoboRIO? It was designed SPECIFICALLY for FIRST FRC. It was a huge oversight to have anything important loose power, ever.

Another problem is that FIRST's system is SO COMPLEX that it is basically impossible for teams to duplicate what happens out on the field in their pit or even on the practice field.

I am not saying that it would have made that much of a difference but this whole brown out thing cost us in St. Louis. Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing. Half of our Qualifying Matches were gone before we got a reasonable answer as to what the problem was and had a work around. It seems to be something related to our robot pulling the voltage below 7V for a small fraction of a second at one point in our auton.

Again, so, yes, shame on us for not realizing that we were close to the edge for 4 prior tournaments where we never saw this problem. But also shame on FIRST for having an unnecessary edge for us to fall off to begin with and for not providing a person with the time and the knowledge to diagnose tricky problems such as ours (had it not been for Mike Copioli and one of his minions from CTRE, we'd still be in the dark about it).

So, that is my team's experience. If you're not on our team, it is easy to say, "Who cares?" Except, I am sure that there were many other teams that ran into problems as well.

Bottom line, I think this is the bad fruit that grew from a bad seed. FIRST should have made this a non-issue with the new controller.

Dr. Joe J.

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

[Thad House]

Originally Posted by Andrew Schreiber

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

The RoboRIO internally does have a Buck/Boost internal to it. It stays powered down to about 4.5v. However, when it dips below 7v, the FPGA disables everything. So it would be a 100% software fix to lower the brownout voltage.

On our practice bot, we actually fed the RoboRIO straight from the regulated 12v. This is because it gets the voltage that is uses from Vin. So if you feed regulated voltage into the RoboRIO it never goes into brownout state. However, this is not FRC legal, and it is required by the rules to be fed straight from Vbatt.

Originally Posted by Jon Stratis

For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it's behavior was completely different.

The cRIO didnt have a specified brownout voltage. The 24v regulator I believe dropped at about 4.5v. But since the cRIO didnt control the power rails on the sidecar, the power from the sidecar would stay on until about 4v, when the sidecar would shut off.

We should start calling the 7v RoboRIO brownout the Virtual Brownout, because it is entirely controlled in software. The hardware brownout doesnt occur until 4.5v, just like the cRIO did.

[AdamHeard]

Originally Posted by Tom Line

If I understand what you're saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.

Yes, but in the long run you don't lose absolute position.

People see the mechanism drop/move and assume it must be from the battery voltage drop... but when it gets back up the control loop brings it back to where it wants to be and the drivers keep driving without issue. So most teams won't really complain about this.

[Andrew Schreiber]

Originally Posted by Thad House

The RoboRIO internally does have a Buck/Boost internal to it. It stays powered down to about 4.5v. However, when it dips below 7v, the FPGA disables everything. So it would be a 100% software fix to lower the brownout voltage.

On our practice bot, we actually fed the RoboRIO straight from the regulated 12v. This is because it gets the voltage that is uses from Vin. So if you feed regulated voltage into the RoboRIO it never goes into brownout state. However, this is not FRC legal, and it is required by the rules to be fed straight from Vbatt.

The second part is what I was suggesting as a solution to this issue. I'm well aware it's not legal, but a rule change could fix that fairly trivially.

I'd also disagree that it's a 100% software fix to lower voltage brownout, or, more accurately, that it should be fixed (And the 6.8-6.3 brownout on the 6V rails is more than likely hardware). To me the brownout is a feature meant to prevent controller shutdowns. While the 6.3V number may seem high, if you're drawing your battery down to 6.3V regularly it's not good and I'd rather have the control try to protect itself by disabling the large draws. Think of it as a soft failure. I'd rather have a soft failure than a dead robot.

To me, if I'm giving it voltage outside the input range then I'm not using the controller properly and any failures are my fault. The fact that the RoboRIO stays alive and tries to save itself as much as it does is nice.

[AdamHeard]

It is legal to replace the 5V rail going to sensors though. 971 implemented/championed this fix and a few teams ran it this year.

[Michael Hill]

Originally Posted by Andrew Schreiber

Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I'd bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.

Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.



*I remembered having conversations with Brando about the point this happened and what we'd have to change in how we designed robots well prior to build when spec sheets first started coming out.

$30 for a buck/boost regulator? Yeah, they'll put that in when they make the next version out of gold...</aloof> Quite simply, I'm sure they just stuck in an LDO and said "we're done with it". Don't forget that since they have an FPGA on board, they likely need several more voltage regulators. Yet If they decided to put in a decent switcher, it's going to be expensive and require expensive magnetics, which will take up more board space. Not to mention the fact that the packaging is low profile, so they may not even have enough height for a decent inductor. Switchers are going to be noisy (electrically) as well. If they were going to do more regulation, I would rather have it done on the PDP, since it's about half the price of a new RoboRio.