View Full Version : paper: New Control Functions - Drive System Testing
Chris Fultz
17-12-2014, 19:54
Thread created automatically to discuss a document in CD-Media.
New Control Functions - Drive System Testing (http://www.chiefdelphi.com/media/papers/3067?) by Chris Fultz
Chris Fultz
17-12-2014, 19:59
As part of our beta testing, we used the roboRIO for some drive system evaluation testing, capturing data not available through the 2014 control. We were testing 2, 4 and 6 CIM and 4 CIM+2mini-CIM drives and capturing data, including amperage and voltage.
Through this testing, we identified an operational effect of one of the control features that we felt would be good to share.
The new control disables output to the PWMs when the system voltage drops below 7V. This is a good feature as it helps keep the control and radio on-line.
However, operationally, this can cause the PWMs to cycle off and on as voltage recovers and then reduces if no change is made by the operator. Depending on the situation - pushing, lifting, etc. - the effect on the robot may be a non-event or may create a response that needs to be managed with software or by operator actions.
The attached paper describes the specific testing we were doing, and the results. It includes 2 charts that show graphically what was occurring.
As teams prepare to use the new control for 2015, we thought this was an interesting and different control response that would be worth sharing.
notmattlythgoe
18-12-2014, 07:56
Good information to know. Thanks.
Thank you for sharing this. I noticed that the paper only refers to the two CIM configuration. What were your results when testing the various other combinations you mentioned above?
Michael Hill
18-12-2014, 11:24
Thanks a lot for sharing this. This is pretty interesting, and could potentially cause a lot of headaches for teams that run 6-cim drivetrains in high gear. I only have theoretical calculations, but it appears that if you have a fully loaded robot (weight-wise), and run in high (fast) gear, you could potentially not even start. I'm using my drivetrain calculator (available as a white paper), and looking at a gear ratio of 4.63:1, 4 inch wheels, and a battery voltage of 12.8V (typical of a mid-match voltage) (Note: this is the default setting for my drivetrain simulator). From a stop, it looks like the battery voltage could drop down to 6.67V if the robot were commanded to go full forward. Again, this is all just a simulation that has absolutely no guarantee of correctness, but if possible, I'd love to see if this causes problems in reality.
Chris Fultz
18-12-2014, 11:42
We only did this specific test with the 4 CIM configuration because with 6 CIMs or 4+2 we just spun the wheels.
The full paper has data on 2, 4, 6 and 4+2 configurations for basic acceleration and for pushing another robot (130 pounds). We are still a few days away from publishing that.
We only did this specific test with the 4 CIM configuration because with 6 CIMs or 4+2 we just spun the wheels.
The full paper has data on 2, 4, 6 and 4+2 configurations for basic acceleration and for pushing another robot (130 pounds). We are still a few days away from publishing that.
Out of curiosity, does this affect CAN controlled controllers? Or just PWM?
MrRoboSteve
18-12-2014, 12:03
My understanding from last night's GameSense is that the load shedding that occurs at 7v impacts all speed controllers, no matter how interfaced.
Michael Hill
18-12-2014, 12:40
It would also be interesting to see if SPI cut out as well.
Thad House
18-12-2014, 13:03
This worries me. I know during hard acceleration at IRI with our 6 CIM Drive we got below 7 volts. I'll have to look at our logs and see how often we actually dropped blow.
Alan Anderson
18-12-2014, 13:07
SPI should function until the battery voltage drops too low for the roboRIO's 5v supply to be maintained. The brownout "load shedding" is a software function that explicitly turns off pneumatic solenoid valves and motor speed controllers when the battery voltage falls below a certain threshhold. It is done in order to try to prevent the voltage from dropping further and causing a complete loss of the control system.
My understanding from last night's GameSense is that the load shedding that occurs at 7v impacts all speed controllers, no matter how interfaced.
How would that be implemented with a CAN connection without affecting other non-motor CAN devices?
(I could guess, but if anyone knows where this is documented would you please post a link)
Alan Anderson
18-12-2014, 13:10
This worries me. I know during hard acceleration at IRI with our 6 CIM Drive we got below 7 volts.
Consider this an opportunity to do dynamic power management. You can monitor battery voltage and motor currents, and you should be able to come up with a way to keep the voltage from sagging low enough to trigger the automatic brownout.
Thad House
18-12-2014, 13:20
Consider this an opportunity to do dynamic power management. You can monitor battery voltage and motor currents, and you should be able to come up with a way to keep the voltage from sagging low enough to trigger the automatic brownout.
Yeah were probably going to look into doing something like this. I'm glad they added current monitoring this year.
Ty Tremblay
18-12-2014, 14:18
Here's the graphic from last night's Behind The Lines showing the key battery voltage points and what happens when they're reached.
NOTE: This is different from what was shown during the episode, and is more accurate.
http://i.imgur.com/rJcp9fg.png (http://i.imgur.com/rJcp9fg.png)
The BTL episode can be viewed here: https://www.youtube.com/watch?v=uUYlS2Vkyuo
Toward the end of our match, we get the battery voltage down to 6.5V easily during normal driving. This will affect us.
Perhaps there could be an option to disable this "feature". I understand that it's there for a reason, but I'm comfortable cutting it a little closer.
As a related question (I was never electrically closely involved) - could you regulate the 12V power being fed to the roborio?
How would that be implemented with a CAN connection without affecting other non-motor CAN devices?
(I could guess, but if anyone knows where this is documented would you please post a link)
CAN Speed Controllers need an enable 'flag' to be sent out over CAN before they output anything. That enable 'flag' (or lack thereof) presumably wouldn't affect devices that it doesn't need to.
As a related question (I was never electrically closely involved) - could you regulate the 12V power being fed to the roborio?
The VRM does have enough capacity to run the RoboRIO, but that configuration will most likely not be legal for competition.
Thad House
18-12-2014, 17:56
Consider this an opportunity to do dynamic power management. You can monitor battery voltage and motor currents, and you should be able to come up with a way to keep the voltage from sagging low enough to trigger the automatic brownout.
Is there a way to calculate how much current it would take in total to drop battery voltage to a certain level? It seems like there should be, but I don't know where to start.
AdamHeard
18-12-2014, 18:14
Is there a way to calculate how much current it would take in total to drop battery voltage to a certain level? It seems like there should be, but I don't know where to start.
You'd have to know the batteries current nominal voltage (no load), the batteries resistance (which will change some over the match), and the current current draw.
I imagine you could BS this enough to get a decent cutoff that is somewhat conservative.
CAN Speed Controllers need an enable 'flag' to be sent out over CAN before they output anything.
so connect the dots. how does that relate to the original question.
and where is it documented.
Mark McLeod
18-12-2014, 20:53
Staged Brown Out when the battery voltage drops too low proceeds through the following stages:
Stage 1: 6.5-7.3v -PWM/CAN motor controllers/Relay outputs are cutoff, 6v power rail drops out
- Driver Station displays “Voltage Brownout”
- Servo power lost
- Disable pulse sent to CAN motor controllers, turning them off.
- Can result in robot stutter as motors pull the voltage low, get cut off, voltage rebounds, motors immediately pull voltage low again
Stage 2: 4.5-6.4v - , GPI outputs go High, 5v/3.3v power rails drop out
- Sensor brownout occurs, e.g., encoders!
- Stage 2 persists until battery voltage recovers to 7.5v
Stage 3:<4.5v - roboRIO drops out
- <3.5v the VRM/DLink drops out
That slide posted earlier from Behind The Lines was inadvertently incorrect as it was taken from an early sketch of the notes we were developing for the show.
Here are the latest notes:
Staged Brown Out when the battery voltage drops too low proceeds through the following stages:
Stage 1: 6.5-7.3v -PWM/CAN motor controllers/Relay outputs are cutoff, 6v power rail drops out
- Driver Station displays “Voltage Brownout”
- Servo power lost
- Disable pulse sent to PWM and CAN motor controllers, turning them off.
- Can result in robot stutter as motors pull the voltage low, get cut off, voltage rebounds, motors immediately pull voltage low again
Stage 2: 4.5-6.4v - , GPI outputs go High, 5v/3.3v power rails drop out
- Sensor brownout occurs, e.g., encoders!
- Stage 2 persists until battery voltage recovers to 7.5v
Stage 3:<4.5v - roboRIO drops out
- <3.5v the VRM/DLink drops out
I'm really not certain why this was done; we regularly pull our batteries below 7 volts in normal operation. Why is there no regulated supply to the roborio? Is there a technical hurdle to accomplish this?
AustinSchuh
18-12-2014, 21:14
so connect the dots. how does that relate to the original question.
and where is it documented.
Not sure where it is documented (Q/A on the beta forums is the closest that I know of), but there is an explicit pulse/message sent out over PWM and CAN to disable all controllers as soon as the system enters the brownout state. The goal of this (whether you agree with it or not) is to save the rest of the system from browning out.
..there is an explicit pulse/message sent out over ... CAN to disable all controllers as soon as the system enters the brownout state.
that's the opposite of what cgmv123 posted:
CAN Speed Controllers need an enable 'flag' to be sent out over CAN before they output anything.
AustinSchuh
19-12-2014, 01:48
that's the opposite of what cgmv123 posted:
I'm quoting what I was told by the engineer who implemented the brownout procedure when I asked for the gory details. Not sure where cgmv123 got his info. There should be some sort of heart beat on the bus in addition, but I don't have any hard information about how that works.
RufflesRidge
19-12-2014, 08:20
It doesn't describe the implementation to the detail Ether is asking about, but page 7 of the roboRIO user manual describes the brownout: https://decibel.ni.com/content/servlet/JiveServlet/download/30419-60-90614/roboRIO%20User%20Manual.pdf
Alan Anderson
19-12-2014, 09:46
I'm really not certain why this was done; we regularly pull our batteries below 7 volts in normal operation. Why is there no regulated supply to the roborio? Is there a technical hurdle to accomplish this?
There is a regulated roboRIO supply. It's built in to the roboRIO. It functions properly until battery voltage drops below 4.5 volts.
MrForbes
19-12-2014, 10:13
The new "low battery" indicator....a jerking robot. Wonder how long it will take teams (who don't read Technical Paper threads on CD) to figure this out? :o
Monochron
19-12-2014, 11:12
The new "low battery" indicator....a jerking robot. Wonder how long it will take teams (who don't read Technical Paper threads on CD) to figure this out? :o
Hopefully they will read teh RoboRio documentation and learn about it . . . but we'll see if they do. Though even that seems to not match what teams are reporting. Especially with CAN brownout as far as I can tell.
It doesn't describe the implementation to the detail Ether is asking about
Actually, it just says the motor controllers on CAN are disabled. It doesn't give any details at all how that is implemented.
Thad House
19-12-2014, 12:09
Actually, it just says the motor controllers on CAN are disabled. It doesn't give any details at all how that is implemented.
I know that in the old system, in the Firmware for the jaguars, if it did not recieve a new packet for 100ms, it would shut itself off.
Also, I have heard that the PWM's don't just turn off, they actually all get set to 0 by the FPGA. I would suspect something similar happens on the CAN bus as well. It probably just sends a 0 to all motors on the bus. Easy to implement and makes sense.
I know that in the old system...
To be clear: having designed and implemented many fail-safe systems in my former life, I can imagine ways to accomplish this. My question is not how it might be done, but rather how it is actually implemented in the 2015 system... and where such details are documented.
Rob Stehlik
19-12-2014, 13:17
This is great information, thanks for sharing. 4 CIM drivetrain anyone?
AustinSchuh
19-12-2014, 15:49
This is great information, thanks for sharing. 4 CIM drivetrain anyone?
We will be running a 4 CIM drivetrain.
sdcantrell56
19-12-2014, 16:09
We will be running a 4 CIM drivetrain.
Us as well
MrForbes
19-12-2014, 16:51
Don't know yet. Last time we played Lunacy, we had a 2 cim drivetrain
AllenGregoryIV
19-12-2014, 16:55
Have any BETA teams been running 6+ CIMs and experienced problems accelerating?
Has anyone managed to pop the main breaker using the new control system?
Chris Fultz
19-12-2014, 22:04
When we were accelerating just our robot (140 pounds), with 4 CIM, 6 CIM or 4+2 CIM, we were above 100 amps for about .4 seconds. There is not much difference in the peak for 6 or 4+2 (about 250 amps). The peak for the 4 CIM was just over 200 amps. We could run about 11 FPS, with the 4+2 approaching 12 (mini-CIMs have a higher output speed).
When we were pushing another robot (Victors in braking mode, 130 pounds but robot not active), the 4 CIM was above 100 amps for about 4 seconds, the 6 CIM for about 1.2 seconds and the 4+2 CIM was above for about 2 seconds. The recorded peaks were all in the 240 amp range. We could run about 7 - 8 FPS when pushing, so a significant loss in performance.
This is with our specific set up for the test - ~8:1 reduction and 4" wheels, so a theoretical top speed of ~12 FPS.
We never popped the main breaker, in any of our testing.
MrRoboSteve
21-12-2014, 12:25
Beta teams: are periods of brownout plotted in the driver station log viewer? Maybe a screen shot?
Mark McLeod
21-12-2014, 13:30
For Voltage Brownout the log viewer will show orange "Disconnected" dots or bars at the top, right under the Disable/Auto/Tele plot lines.
http://team358.org/files/programming/ControlSystem2015-2019/images/VoltageBrownout-LogViewer-med.jpg (http://team358.org/files/programming/ControlSystem2015-2019/images/VoltageBrownout-LogViewer.jpg)
Here's a plot showing how the Disconnects for PWM stutter would look like. Basically a series of Disconnect dots two seconds apart-a second on/a second off/a second on/etc.
http://team358.org/files/programming/ControlSystem2015-2019/images/VoltageBrownout-LogViewer-stutter-med.jpg (http://team358.org/files/programming/ControlSystem2015-2019/images/VoltageBrownout-LogViewer-stutter.jpg)
MrRoboSteve
21-12-2014, 13:39
Hmmm. So is it possible to distinguish from actual connectivity issues? I'm thinking about the troubleshooting procedure here:
http://wpilib.screenstepslive.com/s/3120/m/8559/l/97119-driver-station-log-file-viewer
Would hope that it's not bundled with one of the other "major events" (key 10 in first image)
Mark McLeod
21-12-2014, 13:46
Hmmm. So is it possible to distinguish from actual connectivity issues? I'm thinking about the troubleshooting procedure here:
http://wpilib.screenstepslive.com/s/3120/m/8559/l/97119-driver-station-log-file-viewer
Would hope that it's not bundled with one of the other "major events" (key 10 in first image)
It can be distinguished from other Disconnect events in that data is still being received from the robot. There are no vertical orange bars showing where data from the robot is totally missing.
Other Disconnect events, e.g., loss of robot power, will also show a blank for all the other robot readings (battery voltage, robot CPU %, etc.).
Don't know yet. Last time we played Lunacy, we had a 2 cim drivetrain
We had a 0-CIM drive train in lunacy.
Thad House
22-12-2014, 17:01
So I went and found our logs from IRI. This was a 100lb robot geared for 15.9 FPS.
http://i.imgur.com/QHBIWIV.jpg
Sorry about the giant picture im trying to figure out how to shrink it.
MrForbes
22-12-2014, 17:12
pretty easy to shrink an image, if you have Windows and Office on your computer...just open it in Paint and Resize, and Save As...
http://selectric.org/nerds/graph.jpg
If you're using a phone, it's more challenging. If you don't have Office, but have windows, you might try downloading a program such as IrFanView, which is good for stuff like this. If you have another type of operating system, you will need to find another consultant.
Thad House
22-12-2014, 17:14
pretty easy to shrink an image, if you have Windows and Office on your computer...just open it in Paint and Resize, and Save As...
If you're using a phone, it's more challenging. If you don't have Office, but have windows, you might try downloading a program such as IrFanView, which is good for stuff like this. If you have another type of operating system, you will need to find another consultant.
I meant try and do it in forum code. Ill resize it in a bit when I have time.
MrForbes
22-12-2014, 17:50
More modern forum software automatically sizes externally hosted images to fit the window...but not this neat old software!
you might try downloading a program such as IrFanView, which is good for stuff like this
+1 for IrfanView. Been using it for years. Freeware. Works great for resampling.
The full paper has data on 2, 4, 6 and 4+2 configurations for basic acceleration and for pushing another robot (130 pounds). We are still a few days away from publishing that.
Do you guys have an ETA of when you plan on releasing the full paper?
Chris Fultz
30-12-2014, 09:18
Do you guys have an ETA of when you plan on releasing the full paper?
We are doing a final review at the team meeting tonight, and the full report should be posted later tonight or early Wednesday AM.
Joe Johnson
30-12-2014, 10:04
I'm not your typical beat up on FIRST type guy but this seems like a really poorly designed system feature.
In the first place it seems to me that a small amount of money together with some clever EE thinking could have allowed all critical brain functions of the FRC control system to remain alive well down to battery voltages as low as we'd like (e.g. 1-2 volts).
In the second place, once you've decided to go down the path of load shedding, this seems like a not very clean implementation. By going with "all off" rather than scaling you pretty much guarrantee crazy and unwanted behavior. A simple scaling reduction scheme would make these brown out events almost unnoticeable for the vast majority of teams.
In the third place, a prioritzed and scaled load shedding seems like the obvious right thing to do. Teams should be given a way to let the system shed load in ways that make sense for that robot at that time (e.g. run this function that provides a list of motors to turn off, motors to scale, and motors to leave at full power or to scale as a last resort...).
From the outside looking in, it seems like FIRST missed an opportunity to do the right thing here.
Dr. Joe J.
Jared Russell
30-12-2014, 10:19
...snip...
I think that some teams may go ahead and implement their own version of prioritized load shedding or scaling on top of the built-in all-or-nothing implementation. With the built-in current monitoring of the new control system, it is certainly doable (although not as quick to react or as simple as it could have been if implemented deeper down the stack).
I'm not your typical beat up on FIRST type guy but this seems like a really poorly designed system feature.
In the first place it seems to me that a small amount of money together with some clever EE thinking could have allowed all critical brain functions of the FRC control system to remain alive well down to battery voltages as low as we'd like (e.g. 1-2 volts).
In the second place, once you've decided to go down the path of load shedding, this seems like a not very clean implementation. By going with "all off" rather than scaling you pretty much guarrantee crazy and unwanted behavior. A simple scaling reduction scheme would make these brown out events almost unnoticeable for the vast majority of teams.
In the third place, a prioritzed and scaled load shedding seems like the obvious right thing to do. Teams should be given a way to let the system shed load in ways that make sense for that robot at that time (e.g. run this function that provides a list of motors to turn off, motors to scale, and motors to leave at full power or to scale as a last resort...).
From the outside looking in, it seems like FIRST missed an opportunity to do the right thing here.
Dr. Joe J.
I see where FIRST is coming from with this decision. Last year at the championships, I saw tons of robots trip their main breaker, reset their main breaker, or suffer from having a very low battery voltage. The load shedding is likely an attempt to fix some of these problems.
That said, I've review our teams logs from the championship and one district event. We had matches where we would have been affected by this 7 volt limit. Toward the end of our matches, after running around in high gear, with the compressor running constantly, we saw the battery voltage hit 6v and below, and we definitely below 7 volts for a decent amount of time. This match was played with an older battery, which we had to use because of the number of replays/elimination rounds that went to three.
My solution would be to have the load shedding functionality be optional/adjustable. Teams who don't want to deal with running the system on the very edge of breaker tripping/resets can leave it on, but teams are welcome to (at their own risk) disable the brownout protection and deal with it themselves.
This is speculation, but the 7v limit may be set to prevent transients below 7v that the system is not fast enough to capture. I don't know how fast they're sampling for the protection or how fast the protection can react, but it may not be fast enough to react to the really sudden voltage drops that can occur when a motor suddenly starts. The cRIO could stay alive during fast spikes, but I have yet to see tests of the roborio with transients.
Joe Johnson
30-12-2014, 11:48
As is often the case, upon reflection, I find I may have been a bit harsher than I should have.
The problem is a little more complex than I painted it. The speed controllers that have internal PID loops make for strange power shedding situations. How do you tell a control loop not to exceed certain current outputs? Of course, you can, but it isn't simple and it adds to the complexity of the already complicated interplay between the RoboRio and the Talon SRX (or ... shutter... Jaguar).
Dr. Joe J.
Chris Fultz
30-12-2014, 11:49
There is no reason capable teams cannot implement their own logic to manage this beyond "off/on", but this will really save a lot of teams without that capability. I could also see many teams turning off the protection instead of determing why it was tripping and how to manage it.
For 2014, we knew if our compressor was on, our winch was really slow to retract our launcher. In software, we switched off the compressor when the winch motors were turned on. Using the new control capability, we could make that decision based on battery voltage and possibly make better decisions.
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.