Hibner: Software Testing Via Simulation
 

At the FIRST Robotics Conference I will be showing how to test your control software using simulation. The presentation will be Thursday at 8:00 a.m.

Here's the abstract:

The presentation aims to cover the following topics:
1) why use simulation?
2) How to create a simulation environment to test your software.
3) How to create simple dynamic models of your robot system in order to test more complicated feedback control.
4) Taking your software from the simulation to the cRIO (and vice versa).

The presentation will be highly interactive and audience participation will be highly encouraged (Pavlov would be proud). The presentation will start with a clean slate, and a simple simulation example will be created. As the presentation progresses, the simulation will be expanded to cover more complex systems. As with my other presentation, all of the examples will be developed live. That should hopefully result in a good, fun learning experience for everyone (and perhaps some fun at my expense).

If time allows, the final example will be simulation of autonomous software using a simple robot drive system model.

If there is anything you would like to see addressed, feel free to post it in this thread.

Re: Hibner: Software Testing Via Simulation
 

This is a topic that is of great interest to me and that I would like to share with the teams I work with. Is there any chance that this could be taped and made available online? Any slides that could be posted would be useful but with the way you plan on creating your examples live I think video would be the best way to experience the presentation for those of us who can't be in STL.

Re: Hibner: Software Testing Via Simulation
 

What language will the simulation be created in?

Re: Hibner: Software Testing Via Simulation
 

All of my material from the conference as well as the example code will be posted to chiefdelphi.com after I return from St. Louis.

Collen Shaver will have to answer questions regarding video.

The examples will be done in LabVIEW, but I will also touch on how to do simulations in C++ and Java.

Re: Hibner: Software Testing Via Simulation
 

I forgot to add one thing to the original post:

If your team had a software mishap, such as your robot tried to destroy itself, you had a software bug that was difficult to solve, a broken sensor caused your robot to go haywire, or you tried to do something that never quite worked out, I would love to hear about it. If the story is interesting enough or makes a good example, I'll try to include it as part of the presentation. I'm always looking for good, real-world examples to use.

Re: Hibner: Software Testing Via Simulation
 

I'm not sure about how the timing of this presentation will fit into my own schedule, so I'll post a story here.

Thursday, April 7, 2011: VCU Day 1
We put the bot on the field after readjusting the chain runs for reliability after a demoralizing performance in DC. In our first practice match, the robot did a very dangerous robot dance -- the lift was up to 10', the bot would go-turn-stop-go-turn-stop and do so very violently while doing a sweeping left turn (tough to describe...). It only progressed worse as autonomous continued due to the side-to-side rocking. Finally the bot stopped due to an internal timer (9s) and the lift came back down. We were pretty sure the welds on the lift would pop since the entire base of the lift was twisting as this went on. We were lucky. On the plus side the chains didn't pop off during that entire episode, reassuring us that they were fine for the rest of VCU, and we're now 100% sure our c.g. is "low enough".

It turns out that the double-throw pneumatic valve wiring to the shifting transmission cylinders went bad while we were fiddling with something else. One side of the drive train was stuck in high gear while the other side was stuck in low gear. The 'drive straight' 'P' on the gyro PID wasn't set nearly aggressively enough, so the software couldn't compensate for the 2.56:1 ratio difference in mechanical speed that resulted between the left side and the right side. 100% PWM signal to both sides resulted in very different speeds. Later we tuned the gyro's PID and ran 11 consecutive perfect autonomous modes.

It's a good case for simulations that test error conditions or mechanical failures on the bot.

Re: Hibner: Software Testing Via Simulation
 

Thank you for doing this presentation. It gave me a lot of ideas for next year.

Re: Hibner: Software Testing Via Simulation
 

If we weren't able to attend for certain reasons (Busy driving the robot or at a driver meeting), is there a way we can get information about this? I am very interested but I wasn't allowed to go because there was a drivers meeting.

Re: Hibner: Software Testing Via Simulation
 

Chris indicated he'd have all the presentation materials and examples from both his sessions posted in CD-Media possibly around the middle of next week or so.

Re: Hibner: Software Testing Via Simulation
 

I have posted the presentation material and code samples here:

http://www.chiefdelphi.com/media/papers/2494

Sorry that it took so long.


The material and examples for the autonomous presentation should be posted within the next two days.

Re: Hibner: Software Testing Via Simulation
 

This simulation technique is awesome! We were doing very simple simulations to test logic and various code samples but I never fully thought through how to add some virtual models to our testing scheme.

I've already sorted out a few pesky bugs with our PID scheme. :)

A few questions...

What is supposed to be on page 6 of the presentation slides? Its empty.

On page 21 you discuss adding some limits to the model. Did you add these live during the presentation? I don't see this limiting in the sample code.

Page 29 shows some sensor noise modelling. It appears to always add 0 noise. What is a good noise value for different sensors? I'd like to use "D" in my scheme, so realistic noise would be great.

Thanks in advance.

Re: Hibner: Software Testing Via Simulation
 

Woo hoo. Someone is finding it useful :)


There was going to be a live code demo there. The presentation was going to be over the time limit so I never actually got to it during the presentation.

The demo was the big nasty LabVIEW implementation of the state machine on page 5. With the way LabVIEW does its case structures, it's hard to convey the message on a single page of Powerpoint and multiple pages would've put everyone to sleep during the presentation, so I was planning on going through it live. The point of it was that the state machine was ridiculously complicated and it's easy to see how you could have a lot of bugs in the code, and the simulation technique would allow you to find the vast majority of them.

No, I didn't add them during the presentation. They can be just a little bit tricky and require a few iterations to get them right, and I didn't want to look like a bumbling fool in front of an audience. I probably should have added an example to the code samples.

In order to add limits, you can use a "In Range and Coerce" block to limit the distance traveled to a range. Use the Coerced? boolean from the previous sentence to force the output of the 1st order filters to zero so the integrators stop integrating. That's basic concept behind it.

You are correct, the example is always adding zero. That was me turning off the noise model.

To be honest, I don't know any good noise estimates off the top of my head. What I would do is make some special LabVIEW code that samples your sensor every 1 msec and and takes a running maximum and minimum. You can use the maximum and minimum to set the noise parameter.

If you need a quick ball-park estimate, you can probably use 0.5% to 1% of the signal range. Therefore, if you're using a 300 deg/s gyro you can use about 1.5 to 3 deg/s for the noise amplitude. That should get you started until you can get better numbers.

If you're using encoders and D control, I'd highly recommend using the simple encoder model. When the encoder increments the counter it can cause some trouble with D control. You may find that you'll need to add a filter.

Re: Hibner: Software Testing Via Simulation
 

Thanks for the answers!

We typically use potentiometers for feedback. Any modeling recommendations for a potentiometer? It seems fairly straightforward to me.

Can you test for equivalent motor command? I.e. I could put the actual robot arm into a know position, say parallel with the ground.....then apply a known PWM to it's motor until it is held in place and not falling. Would this value then be equal to the equivalent motor command to simulate gravity??

Re: Hibner: Software Testing Via Simulation
 

I don't do anything to model potentiometers - I just use the angle calculated by the simple model as my sensor feedback. I do usually use a noise model (if you have your cRIO handy, use the noise testing method in my last post. If not, 2 degrees of noise is a good starting point).

If your arm is already built, that is exactly how we do it. You only need to do the test in one position, and the rest is proportional to the sine of the angle from vertical. If you do the test with the arm horizontal, then the equivalent motor command is EquivAtHoriz * sin(angleFromVertical). If you look at the arm model example, this is exactly what was done.

If you don't have the arm already build, you can do some rough calculations to get a good starting point. If you don't feel like rough calculations, using 0.2 seems to work pretty well for getting your software debugged.