Chief Delphi - Potential Git and CI integration

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Programming (http://www.chiefdelphi.com/forums/forumdisplay.php?f=51)

- - Potential Git and CI integration (http://www.chiefdelphi.com/forums/showthread.php?t=138433)

Re: Potential Git and CI integration

Quote:

Originally Posted by Jaci (Post 1551176)

I'm inclined to agree on this one.

Quote:

Originally Posted by Jaci (Post 1551176)

The main reason I say this is because of things like Motion Profiling, PID and any kind of Feedback. While, yes, it is possible to test, the tests often don't reflect actual operation, or do a very poor job of it. That being said, that's probably just my opinion weighing in.

I'll push back on this one. I see no reason why you can't tune a good robot model such that the same PID parameters work in simulation. It's a bit of work, but people have done it with the Atlas robot, Ardupilot, and Baxter, which are all far more complicated and dynamic robots.

Quote:

Originally Posted by Jaci (Post 1551176)

This seems more viable, but it's usefulness would depend a lot on what gazebo sends back. If you're sending back things like what speed the motors are, or if a port is on or off, that's often quite useless from a testing perspective. Something more useful would be things like displacement of the Robot. Things like that can be compared against 'what it should be', and could practically test things like PID loops and motion profiles.

This I think is gazebo's strong point. You can get literally anything. You want to test that your auto drives forward 5 meters? Great, just get the world pose before and after. You want to test that your LaunchCatapult command works? Cool, put a ball in and check where it lands, or even just get the pose of the joint. You can also piggy back on any sensors your robot has, and check something things like driveUntilWall, by comparing absolute distance between the robot and a wall, versus what your rangefinder says and where your robot stops.
That said, I still think even testing "is my motor spinning" is a huge improvement over what most teams do (no testing at all).

Re: Potential Git and CI integration

I think there is real value in testing with a simulator. I don't think it's the right tool for typical unit tests, and I probably wouldn't use it to tune PID constants either; the real value in the simulator is in being able to see all the pieces work together and interact with each other and the environment. Off the top of my head, I might put together an interface that looked something like this:

Code:

struct Object_status{

        //Put position, orientation, speed, here

        std::map<std::string,double> etc; //extra in case the object has extra configuration like part that move relative to the others

};



struct Object_setup{

        Object_status status;

        std::string path_to_model;

};



struct Robot_setup{

        Object_setup object;

        std::string path_to_executable;

};



using Name=std::string;



struct Initial_setup{

        std::map<Name,Object_setup> objects;

        std::map<Name,Robot_setup> robots;

};



using Status=std::map<Name,Object_status>;



typedef bool (*Done_callback)(Status);



Status run_sim(Initial_setup,Done_callback);

Re: Potential Git and CI integration

Quote:

Originally Posted by Peter Mitrano (Post 1551180)

That said, I still think even testing "is my motor spinning" is a huge improvement over what most teams do (no testing at all).

Absolutely. This is the huge value of a easy to use simulator, and it allows our programming team to reliably create code before the robot is done and after it's in the bag.

In python, it's far more important to do off-robot testing because it's a dynamic language -- there's a large risk of crashing due to misspelled variables, etc. Because of this we've had excellent test/simulation support for several years now. In the course of development for the pyfrc simulator/test harness, we have a set of 'default' tests that just do the following things:

Run autonomous mode (with support for running multiple modes)
Run teleop mode
Run a 'full match'
Run a 'full match' with a monkey bashing on the controls randomly

And this meets the needs for most of our robot code -- eg, make sure it doesn't crash due to misspelling of a variable name.

However, we've found there is *some* value in junit-style tests -- but usually we only go to the trouble of writing those when you've got a complex state machine that you want to make sure deals with the edge cases correctly.

Re: Potential Git and CI integration

Quote:

Originally Posted by Jaci (Post 1551176)

For a robotics program, this seems like the wrong way to do things. JUnit has it's place, but I don't think FRC is it. Unless your code (and your framework) are specifically designed for isolated testing, unit testing is not really suited for a robotics / embedded software environment. The main reason I say this is because of things like Motion Profiling, PID and any kind of Feedback. While, yes, it is possible to test, the tests often don't reflect actual operation, or do a very poor job of it. That being said, that's probably just my opinion weighing in.

JUnit != Unit Testing. JUnit is a testing framework. It does a great job of giving you tools to check that the expected results are correct, organizing test cases and reporting failures. You are welcome to use it only for unit tests, but it can be used for any type of testing where there is a need for it's features. WPILib uses JUnit running on a roboRIO to make sure PID loops and sensors are all working.

Testing all depends on what you put in it. If you are going to try to mock out a robot subsystem to do unit testing, you are going to have a bad day. Figuring out by hand what a system should return is hard. If you write a simulated subsystem with simulated sensors to support more of an integration test, you can get some very powerful results. You can model almost all FRC subsystems as a motor attached to a mass.

We do pretty exhaustive automated testing of all our control loops and motion profiles, all inside google-test in C++. Last time I counted, we had 20+ tests for our superstructure, and many more tests for things like motion profiles. If you can't control a model of your robot pretty well, you have no hope of controlling the real thing. Unit tests are also very helpful at helping you understand when smaller units of code are performing as expected. This enabled us to have all the software we wrote before the hardware was done this year up and running on the robot in 24 hours, with no modifications after calibrating pots and measuring limit locations. Our arm and flywheel controllers all worked pretty well and were stable, just from the quality of our simulations. That's pretty powerful and well worth the investment.

Motion profiles are actually much easier to test than controllers. You can compute ideal trajectories and see if your generator successfully generates and follows them. You can also write tests making sure they have the bounded acceleration characteristics you expect and bounded velocity characteristics.

I believe that FRC should try to teach some of the best practices in industry to our students. Testing is a key part of developing and maintaining production software. I write both unit tests and integration tests in both industry (safety critical vehicle automation) and at robotics, and it drastically helps the code quality in both locations.