FWIW, this year I occasionally saw* robots with low battery enter brownout. These were always teams with bad/poorly charged batteries. For robots that rebooted, I think all of them had a wiring related root cause. Not to say that it didn’t happen, but I don’t think I ever saw it.
The purpose of this thread is to try to reverse engineer the requirements around the brownout feature, and discuss alternative requirements that might lead to a different outcome.
Here’s what I think the requirements were for the current feature:
Provide as much telemetry from the robot as possible during low voltage states.
Avoid roboRIO reboots during matches.
Telemetry should show when bad robot behavior is caused by low voltage.
There are two sources of information on the brownout feature that have conflicting information.
Most teams aren’t using incremental sensors for position systems, and as such intermittent brownout of the 5v rail for sensors wouldn’t show up as symptom.
FIRST has a battery of capacity X and allows enough motors and such to draw 10X that current.
Having the RoboRIO go brain dead at just under 7V seem to me to be a poor decision. Perhaps there was data that this was not a problem. But I call that data into question.
More to the point, FIRST has 1000s of teams many with no technical mentors to speak of. For a some extra expense it seems to me that that the critical systems (RoboRIO, radio, USB ports, Sensor Voltages, PDP, …) could stay alive to something ridiculously low making this problem disappear for good.
How low? I would go for 1V (which is possible but could get expensive) but I’ve heard reasonable folk say 3V or 3.3V would get is 99.9% of the way to where I want to go and is much less technically challenging and also less costly.
If I understand what you’re saying I have to disagree. We found this issue during alpha in our 2013 hilo. We use a string pot for positioning, and the drop out would cause the pot to report a low position. That sent our hilo up. So non Incrementing sensors still create problems when they brown out.
I would even say 5V is reasonable. And easily doable with the current system, because we know it doesn’t reboot until about 4.5v. I would still be OK if they system shut of the PWMs at 7v, but left everything else on until 5v. This year wasn’t as bad for brownouts because 6 CIM drives were not advantageous, but if next year comes around and 6 CIM drives are king again, teams are going to start learning real quick about that 7v drop out if it stays.
While having brownout at such a high voltage sucks, it should just be treated as another design constraint. It’s basically a built-in current limiter, something that exists in A LOT of real devices. You go over [X] current, you blow your power budget…maybe it’s just the Aerospace Engineer in me. Successful teams will find ways of managing their power budget (disengage a CIM when it’s not needed, use thicker power wire, run the compressor at strategic times, etc.). I’m not saying I would like a lower brownout voltage, but I have a hard time thinking there’s anything that will be done on the RoboRio to fix the issue. So for now, we just have to treat it as another constraint.
For reference in this thread, does anyone know the brownout voltage for the cRIO? I would ask about the old IFI system, but that had a backup battery for the control system, so it’s behavior was completely different.
The CRIO required 19-30V and the CRIO II (4 Slot) was 9-30V. The PDB would continue to provide 24V to the CRIO down to 4.5V per the spec sheet. (And in practice, we regularly saw drops to 5V in 2014 due to an intermittent short, anything less and it cut out logging)
The RoboRIO is 7-16V but instead of going through any sort of converter the PDP feeds VBat.
If this were the “real world,” I would say, sure, one more constraint, add it to the list. BUT it isn’t. This is a system who’s job is to control a robot that plays a FIRST FRC game. FIRST was painted into a corner with the cRIO. They had insane time pressures and the had to pretty much take what they could get.
But the RoboRIO? It was designed SPECIFICALLY for FIRST FRC. It was a huge oversight to have anything important loose power, ever.
Another problem is that FIRST’s system is SO COMPLEX that it is basically impossible for teams to duplicate what happens out on the field in their pit or even on the practice field.
I am not saying that it would have made that much of a difference but this whole brown out thing cost us in St. Louis. Our first competition match on Carson, something disabled us during auton, then re-enabled us. Shame on us but our code just restarted our auton and ran it again - which broke the robot. Here is the thing. Half of our Qualifying Matches were gone before we got a reasonable answer as to what the problem was and had a work around. It seems to be something related to our robot pulling the voltage below 7V for a small fraction of a second at one point in our auton.
Again, so, yes, shame on us for not realizing that we were close to the edge for 4 prior tournaments where we never saw this problem. But also shame on FIRST for having an unnecessary edge for us to fall off to begin with and for not providing a person with the time and the knowledge to diagnose tricky problems such as ours (had it not been for Mike Copioli and one of his minions from CTRE, we’d still be in the dark about it).
So, that is my team’s experience. If you’re not on our team, it is easy to say, “Who cares?” Except, I am sure that there were many other teams that ran into problems as well.
Bottom line, I think this is the bad fruit that grew from a bad seed. FIRST should have made this a non-issue with the new controller.
Honestly, the issue can probably be solved fairly easily with a buck/boost converter to the roboRio. A quick search on Amazon showed a handful that should work (Max power consumption of the roboRio is 45W according to NI Specs) and they cost under $30. The RoboRio actually has quite a wide input voltage range but feeding it from VBAT just seems like a bad idea. And I’d bet someone smarter than me with electronics could do it even cheaper than what I can find in 10 minutes on Amazon.
Proposed solutions aside - this was a well documented failure mode* of the RoboRIO and I find it extremely concerning that the neither FTAs nor CSAs at CMP were able to identify the cause.
*I remembered having conversations with Brando about the point this happened and what we’d have to change in how we designed robots well prior to build when spec sheets first started coming out.
The RoboRIO internally does have a Buck/Boost internal to it. It stays powered down to about 4.5v. However, when it dips below 7v, the FPGA disables everything. So it would be a 100% software fix to lower the brownout voltage.
On our practice bot, we actually fed the RoboRIO straight from the regulated 12v. This is because it gets the voltage that is uses from Vin. So if you feed regulated voltage into the RoboRIO it never goes into brownout state. However, this is not FRC legal, and it is required by the rules to be fed straight from Vbatt.
The cRIO didnt have a specified brownout voltage. The 24v regulator I believe dropped at about 4.5v. But since the cRIO didnt control the power rails on the sidecar, the power from the sidecar would stay on until about 4v, when the sidecar would shut off.
We should start calling the 7v RoboRIO brownout the Virtual Brownout, because it is entirely controlled in software. The hardware brownout doesnt occur until 4.5v, just like the cRIO did.
Yes, but in the long run you don’t lose absolute position.
People see the mechanism drop/move and assume it must be from the battery voltage drop… but when it gets back up the control loop brings it back to where it wants to be and the drivers keep driving without issue. So most teams won’t really complain about this.
The second part is what I was suggesting as a solution to this issue. I’m well aware it’s not legal, but a rule change could fix that fairly trivially.
I’d also disagree that it’s a 100% software fix to lower voltage brownout, or, more accurately, that it should be fixed (And the 6.8-6.3 brownout on the 6V rails is more than likely hardware). To me the brownout is a feature meant to prevent controller shutdowns. While the 6.3V number may seem high, if you’re drawing your battery down to 6.3V regularly it’s not good and I’d rather have the control try to protect itself by disabling the large draws. Think of it as a soft failure. I’d rather have a soft failure than a dead robot.
To me, if I’m giving it voltage outside the input range then I’m not using the controller properly and any failures are my fault. The fact that the RoboRIO stays alive and tries to save itself as much as it does is nice.
$30 for a buck/boost regulator? Yeah, they’ll put that in when they make the next version out of gold…</aloof> Quite simply, I’m sure they just stuck in an LDO and said “we’re done with it”. Don’t forget that since they have an FPGA on board, they likely need several more voltage regulators. Yet If they decided to put in a decent switcher, it’s going to be expensive and require expensive magnetics, which will take up more board space. Not to mention the fact that the packaging is low profile, so they may not even have enough height for a decent inductor. Switchers are going to be noisy (electrically) as well. If they were going to do more regulation, I would rather have it done on the PDP, since it’s about half the price of a new RoboRio.
It will always be possible for the roboRIO to have less power than it needs to run, so load shedding features will continue to be useful for the great majority of teams.
Might be possible for the roboRIO to tolerate longer periods of low voltage. This would be a software change. That said, robots operating for extended periods at 7 volts are visibly sick.
regulating the input power would require alternate means to detect battery voltage. Analog sidecar anyone?
As FTAA, the only way I knew a robot was in brownout was to go stand behind the drivers and watch the display. We need to make brownout on the robot more visible, to avoid situations like the one Joe describes. Ideas:
a. FMS field monitor should show “brownout count” statistic
b. DS software should have a brownout indicator that stays lit if there’s been a brownout
c. DS stack light should change state for a second or so when there’s a brownout
I think this is a perfect example of where your robot has a better chance to operate reliably using the new functionality, but the control software needs to pay attention to the info available. Since you are not losing absolute position and the motor is disabled, the likely cause for the behavior you saw was a combination of integral windup and reading the sensor in a non-ratiometric way. Naturally as the voltage of the 5V supply drops the lower the resolution of the reading.
The roboRIO Power API will tell you what the voltage of the reference rail is so you can divide. There is also a new potentiometer API that does this for you. The Power API also tells you when the brown-out level has been crossed so you can pause your control loop until it has recovered. The Power API also tells you when the 5V supply has blacked out (though I did notice a bug in this in the LV API implementation that I’ll fix for next season), so you can stop believing your sensor at that point. This will always be below the brown-out level, so your control loop should already be paused.