12-17-2014 06:59 PM

Chris Fultz

Re: paper: New Control Functions - Drive System Testing

As part of our beta testing, we used the roboRIO for some drive system evaluation testing, capturing data not available through the 2014 control. We were testing 2, 4 and 6 CIM and 4 CIM+2mini-CIM drives and capturing data, including amperage and voltage.

Through this testing, we identified an operational effect of one of the control features that we felt would be good to share.

The new control disables output to the PWMs when the system voltage drops below 7V. This is a good feature as it helps keep the control and radio on-line.

However, operationally, this can cause the PWMs to cycle off and on as voltage recovers and then reduces if no change is made by the operator. Depending on the situation - pushing, lifting, etc. - the effect on the robot may be a non-event or may create a response that needs to be managed with software or by operator actions.

The attached paper describes the specific testing we were doing, and the results. It includes 2 charts that show graphically what was occurring.

As teams prepare to use the new control for 2015, we thought this was an interesting and different control response that would be worth sharing.

12-18-2014 06:56 AM

notmattlythgoe

Re: paper: New Control Functions - Drive System Testing

Good information to know. Thanks.

12-18-2014 07:03 AM

mrnoble

Re: paper: New Control Functions - Drive System Testing

Thank you for sharing this. I noticed that the paper only refers to the two CIM configuration. What were your results when testing the various other combinations you mentioned above?

12-18-2014 10:24 AM

Michael Hill

Re: paper: New Control Functions - Drive System Testing

Thanks a lot for sharing this. This is pretty interesting, and could potentially cause a lot of headaches for teams that run 6-cim drivetrains in high gear. I only have theoretical calculations, but it appears that if you have a fully loaded robot (weight-wise), and run in high (fast) gear, you could potentially not even start. I'm using my drivetrain calculator (available as a white paper), and looking at a gear ratio of 4.63:1, 4 inch wheels, and a battery voltage of 12.8V (typical of a mid-match voltage) (Note: this is the default setting for my drivetrain simulator). From a stop, it looks like the battery voltage could drop down to 6.67V if the robot were commanded to go full forward. Again, this is all just a simulation that has absolutely no guarantee of correctness, but if possible, I'd love to see if this causes problems in reality.

12-18-2014 10:42 AM

Chris Fultz

Re: paper: New Control Functions - Drive System Testing

We only did this specific test with the 4 CIM configuration because with 6 CIMs or 4+2 we just spun the wheels.

The full paper has data on 2, 4, 6 and 4+2 configurations for basic acceleration and for pushing another robot (130 pounds). We are still a few days away from publishing that.

12-18-2014 10:48 AM

Arpan

Re: paper: New Control Functions - Drive System Testing

Out of curiosity, does this affect CAN controlled controllers? Or just PWM?

12-18-2014 11:03 AM

MrRoboSteve

Re: paper: New Control Functions - Drive System Testing

My understanding from last night's GameSense is that the load shedding that occurs at 7v impacts all speed controllers, no matter how interfaced.

12-18-2014 11:40 AM

Michael Hill

Re: paper: New Control Functions - Drive System Testing

It would also be interesting to see if SPI cut out as well.

12-18-2014 12:03 PM

Thad House

Re: paper: New Control Functions - Drive System Testing

This worries me. I know during hard acceleration at IRI with our 6 CIM Drive we got below 7 volts. I'll have to look at our logs and see how often we actually dropped blow.

12-18-2014 12:07 PM

Alan Anderson

Re: paper: New Control Functions - Drive System Testing

SPI should function until the battery voltage drops too low for the roboRIO's 5v supply to be maintained. The brownout "load shedding" is a software function that explicitly turns off pneumatic solenoid valves and motor speed controllers when the battery voltage falls below a certain threshhold. It is done in order to try to prevent the voltage from dropping further and causing a complete loss of the control system.

12-18-2014 12:09 PM

Ether

Re: paper: New Control Functions - Drive System Testing

How would that be implemented with a CAN connection without affecting other non-motor CAN devices?

(I could guess, but if anyone knows where this is documented would you please post a link)

12-18-2014 12:10 PM

Alan Anderson

Re: paper: New Control Functions - Drive System Testing

Consider this an opportunity to do dynamic power management. You can monitor battery voltage and motor currents, and you should be able to come up with a way to keep the voltage from sagging low enough to trigger the automatic brownout.

12-18-2014 12:20 PM

Thad House

Re: paper: New Control Functions - Drive System Testing

Yeah were probably going to look into doing something like this. I'm glad they added current monitoring this year.

12-18-2014 01:18 PM

Ty Tremblay

Re: paper: New Control Functions - Drive System Testing

Here's the graphic from last night's Behind The Lines showing the key battery voltage points and what happens when they're reached.

NOTE: This is different from what was shown during the episode, and is more accurate.



The BTL episode can be viewed here: https://www.youtube.com/watch?v=uUYlS2Vkyuo

12-18-2014 02:18 PM

magnets

Re: paper: New Control Functions - Drive System Testing

Toward the end of our match, we get the battery voltage down to 6.5V easily during normal driving. This will affect us.

Perhaps there could be an option to disable this "feature". I understand that it's there for a reason, but I'm comfortable cutting it a little closer.

12-18-2014 02:24 PM

Arpan

Re: paper: New Control Functions - Drive System Testing

As a related question (I was never electrically closely involved) - could you regulate the 12V power being fed to the roborio?

12-18-2014 02:29 PM

cgmv123

Re: paper: New Control Functions - Drive System Testing

CAN Speed Controllers need an enable 'flag' to be sent out over CAN before they output anything. That enable 'flag' (or lack thereof) presumably wouldn't affect devices that it doesn't need to.

The VRM does have enough capacity to run the RoboRIO, but that configuration will most likely not be legal for competition.

12-18-2014 04:56 PM

Thad House

Re: paper: New Control Functions - Drive System Testing

Is there a way to calculate how much current it would take in total to drop battery voltage to a certain level? It seems like there should be, but I don't know where to start.

12-18-2014 05:14 PM

AdamHeard

Re: paper: New Control Functions - Drive System Testing

You'd have to know the batteries current nominal voltage (no load), the batteries resistance (which will change some over the match), and the current current draw.

I imagine you could BS this enough to get a decent cutoff that is somewhat conservative.

12-18-2014 06:41 PM

Ether

Re: paper: New Control Functions - Drive System Testing

so connect the dots. how does that relate to the original question.

and where is it documented.

12-18-2014 07:53 PM

Mark McLeod

Re: paper: New Control Functions - Drive System Testing

Staged Brown Out when the battery voltage drops too low proceeds through the following stages:

Stage 1: 6.5-7.3v -PWM/CAN motor controllers/Relay outputs are cutoff, 6v power rail drops out
- Driver Station displays “Voltage Brownout”
- Servo power lost
- Disable pulse sent to CAN motor controllers, turning them off.
- Can result in robot stutter as motors pull the voltage low, get cut off, voltage rebounds, motors immediately pull voltage low again

Stage 2: 4.5-6.4v - , GPI outputs go High, 5v/3.3v power rails drop out
- Sensor brownout occurs, e.g., encoders!
- Stage 2 persists until battery voltage recovers to 7.5v

Stage 3:<4.5v - roboRIO drops out
- <3.5v the VRM/DLink drops out

12-18-2014 08:01 PM

Arpan

Re: paper: New Control Functions - Drive System Testing

I'm really not certain why this was done; we regularly pull our batteries below 7 volts in normal operation. Why is there no regulated supply to the roborio? Is there a technical hurdle to accomplish this?

12-18-2014 08:14 PM

AustinSchuh

Re: paper: New Control Functions - Drive System Testing

Not sure where it is documented (Q/A on the beta forums is the closest that I know of), but there is an explicit pulse/message sent out over PWM and CAN to disable all controllers as soon as the system enters the brownout state. The goal of this (whether you agree with it or not) is to save the rest of the system from browning out.

12-18-2014 11:02 PM

Ether

Re: paper: New Control Functions - Drive System Testing

that's the opposite of what cgmv123 posted:

12-19-2014 12:48 AM

AustinSchuh

Re: paper: New Control Functions - Drive System Testing

I'm quoting what I was told by the engineer who implemented the brownout procedure when I asked for the gory details. Not sure where cgmv123 got his info. There should be some sort of heart beat on the bus in addition, but I don't have any hard information about how that works.

12-19-2014 07:20 AM

RufflesRidge

Re: paper: New Control Functions - Drive System Testing

It doesn't describe the implementation to the detail Ether is asking about, but page 7 of the roboRIO user manual describes the brownout: https://decibel.ni.com/content/servl...r%20Manual.pdf

12-19-2014 08:46 AM

Alan Anderson

Re: paper: New Control Functions - Drive System Testing

There is a regulated roboRIO supply. It's built in to the roboRIO. It functions properly until battery voltage drops below 4.5 volts.

12-19-2014 09:13 AM

MrForbes

Re: paper: New Control Functions - Drive System Testing

The new "low battery" indicator....a jerking robot. Wonder how long it will take teams (who don't read Technical Paper threads on CD) to figure this out?

12-19-2014 10:12 AM

Monochron

Re: paper: New Control Functions - Drive System Testing

Hopefully they will read teh RoboRio documentation and learn about it . . . but we'll see if they do. Though even that seems to not match what teams are reporting. Especially with CAN brownout as far as I can tell.

12-19-2014 10:28 AM

Ether

Re: paper: New Control Functions - Drive System Testing

Actually, it just says the motor controllers on CAN are disabled. It doesn't give any details at all how that is implemented.

12-19-2014 11:09 AM

Thad House

Re: paper: New Control Functions - Drive System Testing

I know that in the old system, in the Firmware for the jaguars, if it did not recieve a new packet for 100ms, it would shut itself off.

Also, I have heard that the PWM's don't just turn off, they actually all get set to 0 by the FPGA. I would suspect something similar happens on the CAN bus as well. It probably just sends a 0 to all motors on the bus. Easy to implement and makes sense.

12-19-2014 11:26 AM

Ether

Re: paper: New Control Functions - Drive System Testing

To be clear: having designed and implemented many fail-safe systems in my former life, I can imagine ways to accomplish this. My question is not how it might be done, but rather how it is actually implemented in the 2015 system... and where such details are documented.

12-19-2014 12:17 PM

Rob Stehlik

Re: paper: New Control Functions - Drive System Testing

This is great information, thanks for sharing. 4 CIM drivetrain anyone?

12-19-2014 02:49 PM

AustinSchuh

Re: paper: New Control Functions - Drive System Testing

We will be running a 4 CIM drivetrain.

12-19-2014 03:09 PM

sdcantrell56

Re: paper: New Control Functions - Drive System Testing

Us as well

12-19-2014 03:51 PM

MrForbes

Re: paper: New Control Functions - Drive System Testing

Don't know yet. Last time we played Lunacy, we had a 2 cim drivetrain

12-19-2014 03:55 PM

AllenGregoryIV

Re: paper: New Control Functions - Drive System Testing

Have any BETA teams been running 6+ CIMs and experienced problems accelerating?

Has anyone managed to pop the main breaker using the new control system?

12-19-2014 09:04 PM

Chris Fultz

Re: paper: New Control Functions - Drive System Testing

When we were accelerating just our robot (140 pounds), with 4 CIM, 6 CIM or 4+2 CIM, we were above 100 amps for about .4 seconds. There is not much difference in the peak for 6 or 4+2 (about 250 amps). The peak for the 4 CIM was just over 200 amps. We could run about 11 FPS, with the 4+2 approaching 12 (mini-CIMs have a higher output speed).

When we were pushing another robot (Victors in braking mode, 130 pounds but robot not active), the 4 CIM was above 100 amps for about 4 seconds, the 6 CIM for about 1.2 seconds and the 4+2 CIM was above for about 2 seconds. The recorded peaks were all in the 240 amp range. We could run about 7 - 8 FPS when pushing, so a significant loss in performance.

This is with our specific set up for the test - ~8:1 reduction and 4" wheels, so a theoretical top speed of ~12 FPS.

We never popped the main breaker, in any of our testing.

12-21-2014 11:25 AM

MrRoboSteve

Re: paper: New Control Functions - Drive System Testing

Beta teams: are periods of brownout plotted in the driver station log viewer? Maybe a screen shot?

12-21-2014 12:30 PM

Mark McLeod

Re: paper: New Control Functions - Drive System Testing

For Voltage Brownout the log viewer will show orange "Disconnected" dots or bars at the top, right under the Disable/Auto/Tele plot lines.



Here's a plot showing how the Disconnects for PWM stutter would look like. Basically a series of Disconnect dots two seconds apart-a second on/a second off/a second on/etc.

12-21-2014 12:39 PM

MrRoboSteve

Re: paper: New Control Functions - Drive System Testing

Hmmm. So is it possible to distinguish from actual connectivity issues? I'm thinking about the troubleshooting procedure here:

http://wpilib.screenstepslive.com/s/...og-file-viewer

Would hope that it's not bundled with one of the other "major events" (key 10 in first image)

12-21-2014 12:46 PM

Mark McLeod

Re: paper: New Control Functions - Drive System Testing

It can be distinguished from other Disconnect events in that data is still being received from the robot. There are no vertical orange bars showing where data from the robot is totally missing.
Other Disconnect events, e.g., loss of robot power, will also show a blank for all the other robot readings (battery voltage, robot CPU %, etc.).

12-21-2014 01:25 PM

Oblarg

Re: paper: New Control Functions - Drive System Testing

We had a 0-CIM drive train in lunacy.

12-22-2014 04:01 PM

Thad House

Re: paper: New Control Functions - Drive System Testing

So I went and found our logs from IRI. This was a 100lb robot geared for 15.9 FPS.




Sorry about the giant picture im trying to figure out how to shrink it.

12-22-2014 04:12 PM

MrForbes

Re: paper: New Control Functions - Drive System Testing

pretty easy to shrink an image, if you have Windows and Office on your computer...just open it in Paint and Resize, and Save As...



If you're using a phone, it's more challenging. If you don't have Office, but have windows, you might try downloading a program such as IrFanView, which is good for stuff like this. If you have another type of operating system, you will need to find another consultant.

12-22-2014 04:14 PM

Thad House

Re: paper: New Control Functions - Drive System Testing

I meant try and do it in forum code. Ill resize it in a bit when I have time.

12-22-2014 04:50 PM

MrForbes

Re: paper: New Control Functions - Drive System Testing

More modern forum software automatically sizes externally hosted images to fit the window...but not this neat old software!

12-22-2014 04:56 PM

Ether

Re: paper: New Control Functions - Drive System Testing

+1 for IrfanView. Been using it for years. Freeware. Works great for resampling.

12-26-2014 04:39 AM

Knufire

Re: paper: New Control Functions - Drive System Testing

Do you guys have an ETA of when you plan on releasing the full paper?

12-30-2014 08:18 AM

Chris Fultz

Re: paper: New Control Functions - Drive System Testing

We are doing a final review at the team meeting tonight, and the full report should be posted later tonight or early Wednesday AM.

12-30-2014 09:04 AM

Joe Johnson

Re: paper: New Control Functions - Drive System Testing

I'm not your typical beat up on FIRST type guy but this seems like a really poorly designed system feature.

In the first place it seems to me that a small amount of money together with some clever EE thinking could have allowed all critical brain functions of the FRC control system to remain alive well down to battery voltages as low as we'd like (e.g. 1-2 volts).

In the second place, once you've decided to go down the path of load shedding, this seems like a not very clean implementation. By going with "all off" rather than scaling you pretty much guarrantee crazy and unwanted behavior. A simple scaling reduction scheme would make these brown out events almost unnoticeable for the vast majority of teams.

In the third place, a prioritzed and scaled load shedding seems like the obvious right thing to do. Teams should be given a way to let the system shed load in ways that make sense for that robot at that time (e.g. run this function that provides a list of motors to turn off, motors to scale, and motors to leave at full power or to scale as a last resort...).

From the outside looking in, it seems like FIRST missed an opportunity to do the right thing here.

Dr. Joe J.

12-30-2014 09:19 AM

Jared Russell

Re: paper: New Control Functions - Drive System Testing

I think that some teams may go ahead and implement their own version of prioritized load shedding or scaling on top of the built-in all-or-nothing implementation. With the built-in current monitoring of the new control system, it is certainly doable (although not as quick to react or as simple as it could have been if implemented deeper down the stack).

12-30-2014 09:30 AM

Jared

Re: paper: New Control Functions - Drive System Testing

I see where FIRST is coming from with this decision. Last year at the championships, I saw tons of robots trip their main breaker, reset their main breaker, or suffer from having a very low battery voltage. The load shedding is likely an attempt to fix some of these problems.

That said, I've review our teams logs from the championship and one district event. We had matches where we would have been affected by this 7 volt limit. Toward the end of our matches, after running around in high gear, with the compressor running constantly, we saw the battery voltage hit 6v and below, and we definitely below 7 volts for a decent amount of time. This match was played with an older battery, which we had to use because of the number of replays/elimination rounds that went to three.

My solution would be to have the load shedding functionality be optional/adjustable. Teams who don't want to deal with running the system on the very edge of breaker tripping/resets can leave it on, but teams are welcome to (at their own risk) disable the brownout protection and deal with it themselves.

This is speculation, but the 7v limit may be set to prevent transients below 7v that the system is not fast enough to capture. I don't know how fast they're sampling for the protection or how fast the protection can react, but it may not be fast enough to react to the really sudden voltage drops that can occur when a motor suddenly starts. The cRIO could stay alive during fast spikes, but I have yet to see tests of the roborio with transients.

12-30-2014 10:48 AM

Joe Johnson

Re: paper: New Control Functions - Drive System Testing

As is often the case, upon reflection, I find I may have been a bit harsher than I should have.

The problem is a little more complex than I painted it. The speed controllers that have internal PID loops make for strange power shedding situations. How do you tell a control loop not to exceed certain current outputs? Of course, you can, but it isn't simple and it adds to the complexity of the already complicated interplay between the RoboRio and the Talon SRX (or ... shutter... Jaguar).

Dr. Joe J.

12-30-2014 10:49 AM

Chris Fultz

Re: paper: New Control Functions - Drive System Testing

There is no reason capable teams cannot implement their own logic to manage this beyond "off/on", but this will really save a lot of teams without that capability. I could also see many teams turning off the protection instead of determing why it was tripping and how to manage it.

For 2014, we knew if our compressor was on, our winch was really slow to retract our launcher. In software, we switched off the compressor when the winch motors were turned on. Using the new control capability, we could make that decision based on battery voltage and possibly make better decisions.