971's Control System

[billbo911]

In March I had the pleasure of being a Robot Inspector at the Sacramento Regional. One of the Robots I had the pleasure of inspecting was 971's Mammoth!

Let's just say, they do things a bit differently than your standard robot when it comes to their control of the robot. Sure, they have the requisite cRIO, DSC, PDB, radio etc. but that's where it ends. They also have custom built circuit boards, custom sensors that you can now buy, and I believe offboard processor(s) just to name a few.

I'm inviting Austin Schuh, and anyone else associated with 971, to "open the Kimono" a little and let the rest of CD learn from their innovative approach to controlling a robot.

[billbo911]

First question:

Austin mentioned the use of the SPI bus.

What data did it carry. Where was it sourced and where did it end up?


Second question:

Mammoth's shooter system had two moving assemblies: Upper jaw with roller, Lower jaw with rotating "tusks".

Were there controlled independently and synchronised, or did one move with respect to the other?

[Mark Sheridan]

My question is: What is the purpose of the beaglebone cape? I am guessing that its to raise the voltage of the DIO from 3.3 to 5 volts? Did I see that correctly, there is a gyro on that board too? Lastly, for the record, what was the final count of sensors on mammoth?


Looks like I am gonna by a bunch of hall effect sensors, very cool.

[BrianSilverman]

Here’s a bunch of details about our control system. It’s kind of a lot, but I’ve read enough CD threads to know that somebody’s going to want each detail (and it’s fun to share), so here’s my attempt to write it all down at once:

The cape handles reading all of our sensors (except the pressure switch because of the rules ) and providing power for the BeagleBone Black. It gets regulated 12V from an external boost/buck converter. We have an MCU on the cape (an STM32F2XX) for counting encoder ticks and capturing encoder values on the edges of digital input signals. It also handles analog inputs and offloads integrating the gyro to provide us with a more robust integration. The MCU uses SPI to talk to the ADC and gyro (an ADXRS453). It sends all of the data to the BeagleBone Black over TTL serial ~500 times a second. We upgraded to serial with a custom resyncing protocol this year from USB so that any missed data will not cause long dropouts like we were seeing with USB and there’s no USB cable to come unplugged or break. We used the STM32F2XX primarily because it can do a lot of the encoders in hardware which lets us handle more counts than interrupts would. We handle the rest of the encoders and edge captures with interrupts. We had reliability issues with our custom sensor-reading board last year, and the BBB is electrically fragile, so we designed in a lot of protections against electrical noise. We have differential line receivers (AM26LV32EIPWR) on all of the digital inputs and op-amp buffers on all of the analog ones. Also, we used a separate ADC (vs the ones in the MCU), and all of the logic exposed to the robot (line receivers, ADC, and the power pins on the inputs) is powered by a separate buck 5V regulator from the MCU, gyro, and BBB. This means that noise injected on the power or signal lines for any sensor doesn’t affect the BBB power, and any shorts with the sensors don’t take the BBB or MCU down.

Overall sensor count: 5 encoders, 10 digital hall effects, 1 sonar, 1 PWM pulse width from the side car, 4 analog hall effects, 1 potentiometer, and 1 battery voltage monitor. The 5 encoders are all quadrature Grayhill 63Rnnn’s (2 drivetrain, 2 claws, and 1 shooter). The 10 digital hall effects are designed by 971 and sold by WCP (3 for each claw and 4 on the shooter). The sonar was a clever idea for catching that didn’t pan out and was removed. We have one of the PWM outputs from the digital sidecar attached to our custom electronics so we can detect when the cRIO stops sending motors outputs and can be used to update observers and prevent our shooter from timing out. We have found that the cRIO disables outputs mostly due to network problems, but this happens fairly often for short enough periods that the driver’s don’t notice. This is a problem because the software can’t tell if something like our shooter is stuck and needs to be killed or the motors just aren’t on. That was an especially big issue on the shooter where we have a timeout for loading to avoid burning up the motors if it gets stuck. The 4 analog hall effects are on the drivetrain gearboxes so we can see where the dogs are so we can automate our shifting nicely (2 each because the range of 1 wasn’t large enough). The battery voltage measurer is also used for that so we can open-loop speed match both sides of the gearboxes accurately, and correlate problems with the robot with the battery voltage. The potentiometer is for switching between auto modes.

The claws are driven completely independently. Each one has one CIM, and encoder, and 3 hall effect sensors for zeroing. The 3 sensors are at both limits and another one in the middle so we can accurately zero quickly. If you watch videos, both claws twitch right after auto as the robot rezeros. We choose to rezero at the beginning of teleop so that any manual zeroing problems from auto mode won’t last through teleop. When the robot is being set up, we look for edges on the hall effect sensors to calibrate the robot. Austin always moves the claws past one of the edges while setting the robot up before the match. There’s a lot of cool math and code behind making them quickly go where we want and not drop the ball. James is going to do a post explaining more of that.

[James Kuszmaul]

Brian covered the more electrical side of things; because of the question about our control loops, I’ll explain a few of the general ideas here, but I have to go off to a concert soon, so I don’t have much time to get into the details.
Basically, for all of our control loops, we use a state-space representation of our various systems (since motors are so very conveniently linear and time-invariant) using feedback from our encoders. This relies on an equation:
dx/dt = Ax + Bu
Where x is the current state of the motor/system (angle and angular velocity), u is the voltage(s) applied to the motor (we have multiple motors in the drivetrain and claw). A and B are constants which have to do with the physical characteristics of the system. This entire system can be discretized into:
x[k + 1] = A_discrete * x[k] + B_discrete * u[k]
Using this state-space representation and using the a controller where u = K(R - x) where R is the goal state and K is a constant matrix which we can adjust to tune the poles of the system. There is also an observer for taking in sensor feedback.

In the claw, by carefully choosing a u such that the two inputs control separation of the two claw halves and position of the bottom half separately (so u isn’t actually the two voltages applied to the motors... it is just a simple transform away ), we can then place certain constraints on K such that certain values in the matrix must be zero. This allows us to control separation of the claws and position of the bottom half of the claw independently.

In order to deal with the caps on voltage (we only have plus or minus 12 volts available to us in a match; realistically, less), we create constraints in a 2-dimensional space where the two dimensions are the top and bottom claw voltages. We then transform these (linear) constraints into a space where we have separation and position of the claw as the two dimensions. If our controller is asking for voltages outside of the realistic range, we then find the best point in this space to prioritize reducing separation error such that, if the claws open or close a bit, they quickly go back to normal and don’t drop the ball. This is also used in our drivetrain to ensure constant-radius turns. More details will be forthcoming. That was the 15 minutes of writing explanation.
Edit: I wasn't very clear, but everything in here is a matrix.

[Jared Russell]

What are your typical control loop rates, and what states are you generally tracking in your formulation (position/velocity/acceleration?). In the case of the higher order derivatives, are you filtering the measurements at all (in your observers or otherwise)?

I understand state space control, but have found that getting a smooth velocity/acceleration signal to be one of the "dark arts" when I have played with it in the past.

[AustinSchuh]

Originally Posted by Jared Russell

What are your typical control loop rates, and what states are you generally tracking in your formulation (position/velocity/acceleration?). In the case of the higher order derivatives, are you filtering the measurements at all (in your observers or otherwise)?

I understand state space control, but have found that getting a smooth velocity/acceleration signal to be one of the "dark arts" when I have played with it in the past.

We run our control loops at 100 hz , and our data comes in at 500 hz. I was going to run that through a kalman filter at that rate, but nothing really needs anything that overkill, and I never got around to implementing it.

Our drivetrain is 1 control loop. Our states are [left position; left velocity; right position; right velocity].
Our shooter is 3 states. [observed voltage, position, velocity]. Our shooter knows if the spring is attached to the pusher or not, and gain schedules the shooter model and gains accordingly. The origin of the shooter is internally set to the virtual 0 length position of the spring. I really dislike implementing integral control and dealing with integral windup, so I like to implement things using what my professor called Delta U control. The trick is to add an integrator to the plant (so that you model it as taking in changes in power), and then have the observer estimate the actual power being applied. This means that when the system is responding like it is supposed to, the integral term doesn't wind up.
Our claw is 4 states and 1 control loop. [bottom position, bottom velocity, separation position, separation velocity]. This lets us control what we actually care about.

I have found that to get clean velocity estimates, you need to slow down the observer and trust the model more. An observer with really slow poles decays error in the estimate very slowly. An observer with really fast poles decays error in the estimates very quickly. The end result is that noise in the input signal gets amplified when passed through an observer with fast poles, and filtered when passed through an observer with slow poles. If you formulate the problem with a kalman filter instead and look at the gain matrix that the kalman filter is using, you can see how increasing the noise in the input signals will slow down the poles.

[NotInControl]

Just off the bat, immediately after watching your release video the Mammoth became my favorite robot this season.

With the performance to boot. As a control engineer, it is very interesting to see all the cool things you guys do.

I did have a couple of questions of my own I am hoping someone on 971 can answer.

I think the most obvious first question is:

1. What are some of the main reasons your team choose to use the Begalebone to read through all the sensors vs using the cRIO?

2. Does the cRIO do any processing? If so, what? (aside from the compressor)

3. How do you handle graceful shut down of the bone? Or do you not care? In particular, do you take any measures to protect the filesystem on an unplanned shutdown because you are writing to the filesystem as well (I assume this because it was mentioned the bone writes a log file)?

4. What is the communication protocol used between the beaglebone and the cRIO?

5. Do you have vision processing on the beaglebone as well?

6. Do I understand correctly that all of your PID loops are on the beaglebone and not run on the cRIO?

7. What language do you run on the bone? what language do you run on the cRIO?

8. What linux distro are you running on the bone?


We used a beaglebone white this year for our vision processing only. All other sensor data (3 encoders, 1 pot, 2 limit switches, and 2 analog IR sensors were all connected to the cRIO.) This gave us 20fps with 100ms lag, well within our requirements of vision detection.

We have a custom fault tolerant TCP link between the bone and cRIO such that if the link ever goes down, it is displayed on our custom dashboard, and the robot continues to operate without a camera. If the link comes up, the server and clients reset, and comms are re-established automatically.

This system worked flawlessly through our 3 in-season competitions. We had problem at our first off-season event event 2 weekends ago. The filesystem on the SD card started to become corrupt and would crash upon startup. This is because we mount the filesystem r/w and do nothing to prevent ungracefull shutdowns. The quick fix was buy a new SD card, and put a clean img on there and then it worked like a champ.

I have planned countermeasures to prevent this failure from happening in the future, I would just like to know your experience with these devices as well.

Thanks in advanced,
Kevin

[AustinSchuh]

Originally Posted by NotInControl

Just off the bat, immediately after watching your release video the Mammoth became my favorite robot this season.

With the performance to boot. As a control engineer, it is very interesting to see all the cool things you guys do.

I did have a couple of questions of my own I am hoping someone on 971 can answer.

I think the most obvious first question is:

1. What are some of the main reasons your team choose to use the Begalebone to read through all the sensors vs using the cRIO?

We do a lot with edge capturing. When a hall effect sensor triggers, we want to know the encoder value at the edge. We use this for zeroing our claws and shooter, used it to locate the frisbees in our helix, etc, etc. This is a perfect application for an interrupt. Unfortunately, if you try to trigger an interrupt on the cRIO, you will randomly reboot it. We tried that in 2012, and spent multiple matches dead (or rebooting in a loop) on the field. We had our problems escalated with FIRST, and were pretty much ignored. We decided that we would never do that again, and brought everything back under our own control.

We ran a FitPC with a custom USB board for sensors in 2012 and 2013, and had problems with the PC and the USB link. We took a leap this year and chose to use a BBB instead with a serial interlink with a custom cape to handle all the encoder decoding. Kernel context switches are really expensive and wouldn't have let us decode any reasonable number of counts/sec.

Originally Posted by NotInControl

2. Does the cRIO do any processing? If so, what? (aside from the compressor)

It does nothing other than listen to the BBB for motor control packets and run the compressor.

Originally Posted by NotInControl

3. How do you handle graceful shut down of the bone? Or do you not care? In particular, do you take any measures to protect the filesystem on an unplanned shutdown because you are writing to the filesystem as well (I assume this because it was mentioned the bone writes a log file)?

We haven't had any corruption. Modern file systems are quite reliable and have good journaling.

Originally Posted by NotInControl

4. What is the communication protocol used between the beaglebone and the cRIO?

UDP packet with all the motor values. If I remember right, the protocol is pretty flexible and essentially says 'talon on port 1 -> 50%, spike on port 5 -> fwd, solenoid 3 -> on' in a list. UDP is the right answer here since we don't want stale values to arrive, and would rather drop packets than get them too late.

Originally Posted by NotInControl

5. Do you have vision processing on the beaglebone as well?

Nope. We are running it at like 80% CPU utilization right now without vision.

Originally Posted by NotInControl

6. Do I understand correctly that all of your PID loops are on the beaglebone and not run on the cRIO?

PID! We have been running full state feedback since 2012, and haven't looked back. All of our logic runs on the BBB. We updated cRIO code once or twice this year when a new WPILib update came out.

Originally Posted by NotInControl

7. What language do you run on the bone? what language do you run on the cRIO?

C++ on both.

Originally Posted by NotInControl

8. What linux distro are you running on the bone?

Debian Wheezy.

Originally Posted by NotInControl

We used a beaglebone white this year for our vision processing only. All other sensor data (3 encoders, 1 pot, 2 limit switches, and 2 analog IR sensors were all connected to the cRIO.) This gave us 20fps with 100ms lag, well within our requirements of vision detection.

We have a custom fault tolerant TCP link between the bone and cRIO such that if the link ever goes down, it is displayed on our custom dashboard, and the robot continues to operate without a camera. If the link comes up, the server and clients reset, and comms are re-established automatically.

This system worked flawlessly through our 3 in-season competitions. We had problem at our first off-season event event 2 weekends ago. The filesystem on the SD card started to become corrupt and would crash upon startup. This is because we mount the filesystem r/w and do nothing to prevent ungracefull shutdowns. The quick fix was buy a new SD card, and put a clean img on there and then it worked like a champ.

I have planned countermeasures to prevent this failure from happening in the future, I would just like to know your experience with these devices as well.

Thanks in advanced,
Kevin

We have at least 2 partitions on the BBB. We put all our logs on one partition, and the root filesystem on another. We try to not write to the root partition so there is little to no risk of corruption. I'd recommend putting /var/ and where ever you put your logs on a separate partition from your executables. Brian can chime in on a bit more of the particulars of our current setup.

We have had a bit of trouble getting the BBB to come up reliably. It seems like the NIC doesn't always boot reliably, and sometimes the processor doesn't boot either. Every match when I boot the robot up on the field, I watch all the light sequences and make sure that the NIC lights are flashing and the CPU lights are flashing correctly. I've had to do a robot reboot once or twice on the field to fix it.

If you have any hard controls questions, those would also be fun to answer. I think this robot has loops more sophisticated than ones I have written for work.

[AustinSchuh]

We typically start writing control loops and state machines before the hardware is done. We do this by doing test driven development. Since all our state feedback loops require a model, we hook up the code that will actually run on the robot (no modifications required) to the model, and add in a bunch of verification to check for collisions. We then run a bunch of tests to verify that, for example, our shooter will shoot and reload correctly, our claws will zero without intersecting from a bunch of different positions, etc. Only once our code passes a bunch of tests do we put it on the real hardware and let it rip. This means that we don't break our nice new hardware during robot bring up, and we have a lot more confidence in the code that we write. It also means that we can write more complicated code and quickly get it to work.

[NotInControl]

Thanks for the responses.

I was trying to keep this conversation at the highschool level, but I am intrigued by your response and here are some of the control centric questions I have. Hopefully the responses will help someone else as well.

Originally Posted by AustinSchuh

We do a lot with edge capturing. When a hall effect sensor triggers, we want to know the encoder value at the edge. We use this for zeroing our claws and shooter, used it to locate the frisbees in our helix, etc, etc. This is a perfect application for an interrupt.

Your approach obviously works for you and it sounds pretty awesome. But I am curious, could you not achieve the same effect using an absolute encoder, or an incremental encoder with a z (index) signal for homing? This should also allow you to get rid of your hall effect sensors.

Originally Posted by AustinSchuh

PID! We have been running full state feedback since 2012, and haven't looked back. All of our logic runs on the BBB. We updated cRIO code once or twice this year when a new WPILib update came out.

This is interesting to me. If I recall correctly, the cheezy poofs also use FSFB for their drivetrain control (or at least that is what I remember from their 2013 code release). I’d like to preface that I have used state feed back controllers a lot in practice, but never in a real application for various reasons which is where my line of questioning comes from.

1. State feedback out of the box does not change the type of system. As a result, it typically is only useful for regulator type systems that do not need to track their inputs unless the algorithm is modified. I assume your control loops need to track a step response or something similar so how do you modify the state feedback controller to have a near zero steady state error? I am familiar with either modifying the SFB controller with an integrator, or using the nbar function to scale the reference input. What do you do to overcome this?

2. When dealing with a full state feedback controller you must provide all of the states. Typically for robotics the states are position, velocity, and or acceleration. Unless you use accelerometers, you are almost always guaranteeing the need to have an estimator apart of your control loop to provide the unmeasured state. This adds uncertainty. What type of estimator do you typically use, and how much error does it add to your control loops? This is the reason I typically use output feedback instead of state feedback because I can almost always guarantee that I can measure the output and don’t have use an estimator.

3. I assume you are designing your State Feedback Controllers on a LTI plant model of your system. Have you noticed that the range of operation you needed out of your system remains in that particular linear range of your plant as described in the LTI model? We use Matlab simmechanics and import solidwork models of our robot systems with inertia properties directly into simulink to design our controllers. The controllers imported are always non-linear (mainly due to friction models) and so we try to cover these cases with gain scheduling, or other control mechanisms to cover most of the non linear system. This method helps us see the non-linearity and try to account for it in our controllers. How do you ensure your controller designed on an LTI plant model, can scale to the non-linear system of your bot? or Is the LTI controller good enough that you don’t care.

4. I assume you use Matlab/simulink to perform pole placement, but what exactly do you code? I assume you transform the u = Kx +r equation into a difference equation a nd calculate the matrices each control loop. Do you program more than this? What do you do to control the output of the controller for our application (saturate to +1 or -1 for driving motor controllers for example). We run a custom PID controller written in Java which takes care of integral windup, derivative noise filtering, and allows for gain scheduling, feedforward term, and output scaling to drive out motor controllers.

I typically use PID for most things, although I have run LQR and MPC controllers in practice. It’s pretty cool to see you guys run SFB controllers for your bot, although PID is just a special case of the state space controller. Do you plan to share your code eventually, or possibly provide excerpts of your control design (i.e state space models, or difference equations)?

Originally Posted by AustinSchuh

We haven't had any corruption. Modern file systems are quite reliable and have good journaling.

The problem we noticed wasn’t with the filesystem itself, but rather the life of the SD card, and having bad sectors on the flash drive. Enough writes, and improper shut downs actually damaged our SD card sectors, where journaling could not repair. The card only lasted about a month and a half with moderate use. I replace the card with a backup image to get the system back to normal. I was wondering if you had any similar experience, but it doesn’t sound so. We plan to make our file system read only in the future, and then write data to a different partition. I can’t imagine that is a solution for you. It looks like you need the file system to be read/write so you can write to the file that holds your gpio pins on the bone correct? Is that not on the Root file system?


Thanks,
Kevin

[James Kuszmaul]

Originally Posted by NotInControl

Your approach obviously works for you and it sounds pretty awesome. But I am curious, could you not achieve the same effect using an absolute encoder, or an incremental encoder with a z (index) signal for homing? This should also allow you to get rid of your hall effect sensors.

I am not personally that familiar with the use of index signals on absolute encoders, but several potential issues could arise:
-The majority (or all?) of indexed encoders will give you one or more index signal(s) per rotation. Most of our encoders move more than a single rotation through their full range of motion. This means that we would only know for sure that we were in one of a few possible places.
-With the hall effect sensors, it is easy to put sensors near or at the limits of the appendage or device. This means that, when zeroing, we are guaranteed not to run into a hard stop.
-If, for whatever reason, the encoder were to slip, then we would have to re-do the zeroing calibration. If we did not notice this before a match, we could easily spend a whole match missing shots by slight amounts or dropping the ball or the such.

Originally Posted by NotInControl

This is interesting to me. If I recall correctly, the cheezy poofs also use FSFB for their drivetrain control (or at least that is what I remember from their 2013 code release). I’d like to preface that I have used state feed back controllers a lot in practice, but never in a real application for various reasons which is where my line of questioning comes from.

1. State feedback out of the box does not change the type of system. As a result, it typically is only useful for regulator type systems that do not need to track their inputs unless the algorithm is modified. I assume your control loops need to track a step response or something similar so how do you modify the state feedback controller to have a near zero steady state error? I am familiar with either modifying the SFB controller with an integrator, or using the nbar function to scale the reference input. What do you do to overcome this?

If I understand your question correctly, then you are asking how we deal with situations where it may be necessary to apply a non-zero voltage to the motors in order to attain the goal state (I'm afraid that my vocabulary in this area is a bit behind my knowledge of how to make our systems work). The way we deal with this is the "Delta U" controller that Austin mentioned earlier; we add the voltage of the motors as the state and make the change in the voltage an input. This allows us to deal with situations where we need a non-zero steady-state voltage.

Originally Posted by NotInControl

2. When dealing with a full state feedback controller you must provide all of the states. Typically for robotics the states are position, velocity, and or acceleration. Unless you use accelerometers, you are almost always guaranteeing the need to have an estimator apart of your control loop to provide the unmeasured state. This adds uncertainty. What type of estimator do you typically use, and how much error does it add to your control loops? This is the reason I typically use output feedback instead of state feedback because I can almost always guarantee that I can measure the output and don’t have use an estimator.

Well, the actual states in our state matrix are just position and velocity (these are all that are necessary for dealing with an ideal motor). We run an observer such that:
If y = Cx + Du = the output of the system (the encoder readings), and x is [[position],[velocity]], then C is [[1, 0]] (and D is zero). This allows us to set up an estimate of x,
x_hat = A*x_hat + B*u + L * (y - y_hat)
x_hat = A*x_hat + B*u + L * (y - C*x_hat - D*u)
Nothing too unusual; a reasonably standard state observer. By placing the poles on A - LC, we can control the aggressiveness of the observer.

Originally Posted by NotInControl

3. I assume you are designing your State Feedback Controllers on a LTI plant model of your system. Have you noticed that the range of operation you needed out of your system remains in that particular linear range of your plant as described in the LTI model? We use Matlab simmechanics and import solidwork models of our robot systems with inertia properties directly into simulink to design our controllers. The controllers imported are always non-linear (mainly due to friction models) and so we try to cover these cases with gain scheduling, or other control mechanisms to cover most of the non linear system. This method helps us see the non-linearity and try to account for it in our controllers. How do you ensure your controller designed on an LTI plant model, can scale to the non-linear system of your bot? or Is the LTI controller good enough that you don’t care.

Assuming LTI is good enough for our applications (on our shooter this year, we gain-scheduled it to use a different controller if it was pushing the springs or not, but in either state, assuming LTI was sufficient).

Originally Posted by NotInControl

4. I assume you use Matlab/simulink to perform pole placement, but what exactly do you code? I assume you transform the u = Kx +r equation into a difference equation a nd calculate the matrices each control loop. Do you program more than this? What do you do to control the output of the controller for our application (saturate to +1 or -1 for driving motor controllers for example). We run a custom PID controller written in Java which takes care of integral windup, derivative noise filtering, and allows for gain scheduling, feedforward term, and output scaling to drive out motor controllers.

We've been using python for pole-placement (using Slycot).
We generally get a voltage out of our controller (as mentioned earlier, this is not always in u), and if the voltage the controller wants to apply is too large, we saturate the motor controllers. Scaling for the actual PWM outputs is dealt with separately (because Talons are conveniently linear, this is just a matter of scaling; pre-2013, with victors, we had fitted functions to deal with the non-linearity). We do not currently add a feedforward value to u in our implementations; using the aforementioned "Delta U" controller deals with this in cases where we might need it (such as when pulling the springs on our shooter back).

Originally Posted by NotInControl

I typically use PID for most things, although I have run LQR and MPC controllers in practice. It’s pretty cool to see you guys run SFB controllers for your bot, although PID is just a special case of the state space controller. Do you plan to share your code eventually, or possibly provide excerpts of your control design (i.e state space models, or difference equations)?

Once our website comes back up, you can find our released code from just before this season started (it should come up just by searching "released code" or the such). We will probably be releasing more documentation regarding our controls at some point, but that is contingent on someone going to the trouble of writing it all up and then editing it to make sure that it is well enough written for release. Hopefully we get something out by the end of the summer, but don't hold your breath.

Originally Posted by NotInControl

The problem we noticed wasn’t with the filesystem itself, but rather the life of the SD card, and having bad sectors on the flash drive. Enough writes, and improper shut downs actually damaged our SD card sectors, where journaling could not repair. The card only lasted about a month and a half with moderate use. I replace the card with a backup image to get the system back to normal. I was wondering if you had any similar experience, but it doesn’t sound so. We plan to make our file system read only in the future, and then write data to a different partition. I can’t imagine that is a solution for you. It looks like you need the file system to be read/write so you can write to the file that holds your gpio pins on the bone correct? Is that not on the Root file system?

I'm not really the person to talk to about this (Austin or Brian can tell you more). I never recall corruption of the SD card causing us issues when driving the robot (and certainly never in any matches), but at one point at Worlds, we noticed some of our logs had gotten corrupted so we replaced the BeagleBone (and SD card) in the robot anyways, just to be safe.

Hopefully I've answered your questions; if I missed the point of any of your questions or if you have any more, feel free to ask and one of us should respond.

[AustinSchuh]

Originally Posted by NotInControl

Thanks for the responses.

I was trying to keep this conversation at the highschool level, but I am intrigued by your response and here are some of the control centric questions I have. Hopefully the responses will help someone else as well.

Funny that you should say that. One of my high school kids already started responding to you before I got home today I prefer on 971 to drag the students up rather than the design or math down. I'm always amazed by what they can learn.

For everyone else, we'll happily try to give you some pointers about how all this works and how we do it. Keep asking questions and we'll keep trying to answer. I've taught FSFB to enough students (James, for example, though after the basics, he kept learning on his own) that I've figured out a reasonably good way to teach it.

Originally Posted by NotInControl

1. State feedback out of the box does not change the type of system. As a result, it typically is only useful for regulator type systems that do not need to track their inputs unless the algorithm is modified. I assume your control loops need to track a step response or something similar so how do you modify the state feedback controller to have a near zero steady state error? I am familiar with either modifying the SFB controller with an integrator, or using the nbar function to scale the reference input. What do you do to overcome this?

James partially addressed this. I really really really don't like integral control. It is a significant amount of work to do right. For most, if not all, of our systems, I try it first without integral control and see if I can make the controller stiff enough to keep the error within acceptable bounds. Turns out that was good enough for our drivetrain and claw this year. This year, James did the modeling for all of our systems, and I tuned the final constants. He did a great job.

I've done integral control multiple ways with FSFB. My favorite is to add an integrator to the input to the plant, and then observe the applied voltage in the observer. This unfortunately puts one of the poles for the controller in the observer. On the other hand, if the system is responding exactly as expected, the commanded voltage will be the same as the observed voltage, and you won't get any integral windup. I find that it works really well. One of my professors in a MPC class called this trick Delta U control. The other classic trick is to integrate the state that you want to have zero steady state error, and stabilize that. You can model that into your plant. That has windup problems with a step input, which I dislike.

Originally Posted by NotInControl

2. When dealing with a full state feedback controller you must provide all of the states. Typically for robotics the states are position, velocity, and or acceleration. Unless you use accelerometers, you are almost always guaranteeing the need to have an estimator apart of your control loop to provide the unmeasured state. This adds uncertainty. What type of estimator do you typically use, and how much error does it add to your control loops? This is the reason I typically use output feedback instead of state feedback because I can almost always guarantee that I can measure the output and don’t have use an estimator.

The states we always use are position and velocity. You are correct that an estimator is needed. If tuned improperly, it can add a significant amount of noise into the velocity estimate. If tuned correctly, the estimate is quite good and gives us lots of freedom in what we do. Our observer poles this year were something like 0.05+-0.01j for our claw (dt = 0.01s). That's pretty fast. It helps that our sensors are very carefully designed into the system and we work hard to keep the backlash/non-lti behavior out of the system.

Originally Posted by NotInControl

3. I assume you are designing your State Feedback Controllers on a LTI plant model of your system. Have you noticed that the range of operation you needed out of your system remains in that particular linear range of your plant as described in the LTI model? We use Matlab simmechanics and import solidwork models of our robot systems with inertia properties directly into simulink to design our controllers. The controllers imported are always non-linear (mainly due to friction models) and so we try to cover these cases with gain scheduling, or other control mechanisms to cover most of the non linear system. This method helps us see the non-linearity and try to account for it in our controllers. How do you ensure your controller designed on an LTI plant model, can scale to the non-linear system of your bot? or Is the LTI controller good enough that you don’t care.

State feedback gives us enough knobs to make our controllers aggressive enough and robust enough to noise that this hasn't been an issue. I'm sure we could do better if we spent more time on it, but honestly it has been good enough that it hasn't been worth messing with. As James said, we gain schedule where it makes the most sense or we actually notice a problem. We gain scheduled the number of frisbees in our indexer in 2013. Controllers are remarkably robust too. I think one of the reasons that we are able to run controllers that are as aggressive as we do is because the observer is so good at filtering out the noise from our sensors and keeping our velocity estimates clean and undelayed.

Originally Posted by NotInControl

4. I assume you use Matlab/simulink to perform pole placement, but what exactly do you code? I assume you transform the u = Kx +r equation into a difference equation a nd calculate the matrices each control loop. Do you program more than this? What do you do to control the output of the controller for our application (saturate to +1 or -1 for driving motor controllers for example). We run a custom PID controller written in Java which takes care of integral windup, derivative noise filtering, and allows for gain scheduling, feedforward term, and output scaling to drive out motor controllers.

SI units all the way. We convert everything to radians, seconds, meters, volts, amps, ... We output voltage from our controllers and then convert that to +- 1 and then saturate that. Starting this year, we have been using set theory from MPCs to deal with saturation in systems with more than 1 input like our claw.

The controller is of the form U = K(R - X).
z X = (A - BK) X + BK R

The place function (we re-implemented it in python) takes the A matrix, B matrix and the desired poles, and places the eigenvalues of A-BK where we want them.

Originally Posted by NotInControl

I typically use PID for most things, although I have run LQR and MPC controllers in practice. It’s pretty cool to see you guys run SFB controllers for your bot, although PID is just a special case of the state space controller. Do you plan to share your code eventually, or possibly provide excerpts of your control design (i.e state space models, or difference equations)?

I'm probably not orthodox here, but I pretty much lump LQR and FSFB in the same bucket. They are 2 different ways of placing your poles, with different problems and insights. We ran LQR on our DT this year because we couldn't figure out how to get python to place the left and right poles in the same spot with direct pole placement. I tend to tune with LQR and then print the poles out to see if they are where I want them to be.

On top of the code that we have released, every now and then, some of our controls code gets ported to Java and shows up on a local blue robot. 254 has been running our flywheel controller from 2012 since 2012 on all their bots. The awesome part was that all they had to do this year was to get a step response, re-tune the model, and it worked perfectly. I don't think they ran our drivetrain controller in 2013, but I could be wrong. I wrote the first FSFB drivetrain controller with one of 254's students in 2011.

We do all of our controls design first in Python where the cost of iteration is cheap, and then port that to C++. If all you want to understand is our controls, start by reading the Python. Take the time to read and understand our claw controller. It is my favorite loop of all of them.

I'm not sure what our release plan is, but since you've expressed interest, I'll see if we can release earlier rather than wait like we have in the past.

Originally Posted by NotInControl

The problem we noticed wasn’t with the filesystem itself, but rather the life of the SD card, and having bad sectors on the flash drive. Enough writes, and improper shut downs actually damaged our SD card sectors, where journaling could not repair. The card only lasted about a month and a half with moderate use. I replace the card with a backup image to get the system back to normal. I was wondering if you had any similar experience, but it doesn’t sound so. We plan to make our file system read only in the future, and then write data to a different partition. I can’t imagine that is a solution for you. It looks like you need the file system to be read/write so you can write to the file that holds your gpio pins on the bone correct? Is that not on the Root file system?

We must have gotten lucky. Sorry to hear that you had troubles.

GPIO is done by sending all that information to the BBB from a serial connection to our cape. Regardless, GPIO is done through /sys/ which is a sysfs filesystem. Kernel context switches are killer on a system like this, so we work hard to minimize them.

Hope that helps.

[BrianSilverman]

A couple corrections/clarifications. Sorry for taking so long; I didn't realize more stuff got posted (still learning how to use CD)...

Originally Posted by AustinSchuh

It does nothing other than listen to the BBB for motor control packets and run the compressor.

It also takes joystick data and forwards it to the BBB. We had to go through the cRIO there both because of the rules and because the DS sends the packets there.

Originally Posted by AustinSchuh

UDP packet with all the motor values. If I remember right, the protocol is pretty flexible and essentially says 'talon on port 1 -> 50%, spike on port 5 -> fwd, solenoid 3 -> on' in a list. UDP is the right answer here since we don't want stale values to arrive, and would rather drop packets than get them too late.

That was the old version back when we had still ran some stuff on the cRIO, so it had to be flexible. The current version just sends an array of bytes across (the bytes that the cRIO writes out to the FPGA) for the motor controllers and 1 byte (8 bits) for the solenoids. We decided to minimize the amount of processing on the cRIO because of all the problems we've had there.

Also, despite using UDP, we still have issues with stale values because the bridge queues packets going between wired ports when it has trouble with the wireless communications (sounds weird, but it does).

Originally Posted by AustinSchuh

Nope. We are running it at like 80% CPU utilization right now without vision.

More like 60% recently, but still too much to comfortably fit vision.

Originally Posted by AustinSchuh

We have at least 2 partitions on the BBB. We put all our logs on one partition, and the root filesystem on another. We try to not write to the root partition so there is little to no risk of corruption. I'd recommend putting /var/ and where ever you put your logs on a separate partition from your executables. Brian can chime in on a bit more of the particulars of our current setup.

Everything except logs (including /var/, although on second thought that's probably a bad idea) is on the internal eMMC in one partition, and the logs are on the uSD card (which means they have their own partition).

Originally Posted by AustinSchuh

We have had a bit of trouble getting the BBB to come up reliably. It seems like the NIC doesn't always boot reliably, and sometimes the processor doesn't boot either. Every match when I boot the robot up on the field, I watch all the light sequences and make sure that the NIC lights are flashing and the CPU lights are flashing correctly. I've had to do a robot reboot once or twice on the field to fix it.

To be pedantic, the chip that doesn't like coming up is just a PHY.

Originally Posted by James Kuszmaul

I am not personally that familiar with the use of index signals on absolute encoders, but several potential issues could arise:
-The majority (or all?) of indexed encoders will give you one or more index signal(s) per rotation. Most of our encoders move more than a single rotation through their full range of motion. This means that we would only know for sure that we were in one of a few possible places.
-With the hall effect sensors, it is easy to put sensors near or at the limits of the appendage or device. This means that, when zeroing, we are guaranteed not to run into a hard stop.
-If, for whatever reason, the encoder were to slip, then we would have to re-do the zeroing calibration. If we did not notice this before a match, we could easily spend a whole match missing shots by slight amounts or dropping the ball or the such.

Using an index pin doesn't get you much, because you still have to have some kind of interrupt for counting it, you still have to position something that'll be different between robots (and can't move at all), and you still need some way to know which index pulse you've hit.

Hitting hard stops isn't as big of a deal because we always make sure to zero at low enough power that it can stall for several seconds without breaking anything (until somebody disables it). However, it is still nice to not have to worry about what position things start in.

Originally Posted by James Kuszmaul

Once our website comes back up, you can find our released code from just before this season started (it should come up just by searching "released code" or the such). We will probably be releasing more documentation regarding our controls at some point, but that is contingent on someone going to the trouble of writing it all up and then editing it to make sure that it is well enough written for release. Hopefully we get something out by the end of the summer, but don't hold your breath.

The code itself is actually on a different server (that still works) here.

Originally Posted by James Kuszmaul

I'm not really the person to talk to about this (Austin or Brian can tell you more). I never recall corruption of the SD card causing us issues when driving the robot (and certainly never in any matches), but at one point at Worlds, we noticed some of our logs had gotten corrupted so we replaced the BeagleBone (and SD card) in the robot anyways, just to be safe.

I think there is a good chance this was due to wearing out the uSD card. That corruption didn't happen after a hard shutdown; it corrupted the file while I was looking at it. The only other explanations I can think of are that the card was falling out or there was something else wrong electrically on that BBB. It didn't matter too much in the end, but it did make us nervous.

[AustinSchuh]

Originally Posted by BrianSilverman

To be pedantic, the chip that doesn't like coming up is just a PHY.

To be even more pedantic, I've seen the BBB also not come up on the field...