Unit/Integration testing JAva code for FRC

[otherguy]

Our programming team is typically made of ~8 students and two mentors. We manage our code in github projects. Our workflow is usually something like the following: Students are assigned development of specific functionality (subsystems/commands). They develop this code in a branch, commit their changes to the branches, and when they are done they submit a pull request. A mentor or lead student reviews the changes and make comments if there are problems. Once the code is deemed complete it gets pulled into our master branch. Master is the branch which should always be safe to deploy to the hardware.
We have had success with this workflow over the past few years, but it's completely dependent upon good 'peer review' of code to catch problems before getting pulled into Master. Inevitably, people are going to miss bugs, especially when it comes to integration of classes (e.g. command passes in a negative value to drive a motor forward, but the method being called requires a positive value to do that).

Last year we ran into a few problems with code changes getting pulled into our master branch which, that when integrated with other units, didn't work correctly.
There weren't any show stoppers, but I don't want to depend on impeccable code review from a few select individuals to find all the problems, especially when changes are being made at competition.

I'm interested in knowing what experiences other teams have with unit/integration test tools.
What have you used?
What has/hasn't worked and why?

Links to code/examples of your tool of choice in use would be appreciated.

Thanks!

[JesseK]

Whether it's code review or unit tests, code quality takes time. It isn't free, simple or void of errors. They're only as good as the person behind them. With that said...

Considering that a large portion of robot code is mathematic, it's easy to simulate the error and boundary conditions using loops over a range, and a null or two. Some of the rest of the code is pure logic based upon binary flags: read a button input, if 1 do X, if 0 do Y. Unit tests would set the initial conditions, call specific methods, and test the results.

Architecturally (and security-wise, but that's outside FRC) the worst part about unit tests are ensuring that all method/variable declarations are package-privileged so static unit test classes may call them without needing to modify the original source for variable injection.

The code base for a simple FRC robot is small enough that any unit tests would suffice without tools. Add vision, speed ramping, etc - it's easy to go straight for something like the Poofs' Sim-FRC-2015

At work, I've used the pull-request (aka Github) model using Atlassian Stash and another workflow with Gerrit. If Gerrit is used with Jenkins (or any number of other continuous integration setups) then it can automatically kick off a build based upon a code review. Based upon how Gerrit is configured, it can deny merging of the review if the unit tests within Jenkins failed. Also Gerrit used Git's namespacing mechanic to stage the pending changes, then provided a URL in order to fetch & checkout the changes in a local repo. This was insanely useful if someone made changes but then became unavailable to respond to comments. This is what I consider a 'premium' setup and it requires a LOT of time.

I haven't found an equivalent for Stash, other than having unit test results copy-pasted into a pull request. I can pull the changes since I am admin/integrator, but if I didn't have that role I don't know how I could pull the changes to test locally.

With Github, you should be able to get a link to the patchset of a pull request, then you could fetch the change, run the unit test, post results and continue with the merge (or not) from there.

[Cel Skeggs]

We have a similar setup - four students who pull-request into master in our GitHub repo.

The primary way that we have people test their code is through our emulator system (looks something like this) and that seems to catch anything really obvious. Beyond that, anything that has slipped through that either gets caught during code-review or gets fixed later once we start running our code on the actual robot - most of the issues we run into tend to be directions or states getting reversed, and those are relatively easy to fix.

In the case of your example: "a command passes in a negative value to drive a motor forward, but the method being called requires a positive value to do that" - simply adding error checking code to the method means that the bug gets uncovered immediately. Then, the code reviewer can just make sure that they have appropriate checks on any new methods, and a significant portion of bugs will get caught.

We haven't used unit testing in our robot code itself, but our framework has quite a few to test its core components. To get this to work, we had to figure out a few of the same things as you'll need for actual testing - notably, we needed a way to mock out the passage of time. So now our unit tests can make the tested components use fake time and change it as necessary. We've tested a number of time-dependent components successfully with this system. We use JUnit for this - see an example of our ExpirationTimer test here.

One strategy which might be useful would be to "record" certain "sessions" of running a robot (in an emulator or on a real robot) and check that the new version of the code plays back the same sequence (or at least a close-enough sequence) when it gets run with the same input data.

[randallh]

We've used the GitHub pull-request model for several years, and it works well. But I do agree with most of what Colby and JesseK have mentioned. As a professional open source developer, I insist on good unit tests in all the projects I run, and where possible continuous integration (via Jenkins, Travis, or something similar).

Our team has made extensive use of unit tests, but the WPILib makes this very challenging; see our presentation on this topic at the 2015 FIRST World Championship Conference. Because it is so difficult to write tests for WPILib robot projects, in 2015 we ended up creating our own API of hardware interfaces just so that we can unit test without the WPILib classes. This past summer we extracted all of that code and started the Strongback open source Java library. Hopefully you can benefit from at least its ideas, if not by directly using the library. Get started by reading our Using Strongback online book, which documents the library, how to use it, and how you can write normal unit tests with it.

But having unit tests is of very little benefit if they are not used. Robot projects that use Strongback can use and run JUnit tests via the excellent JUnit Eclipse plugins, but they also have Ant tasks to run all of the unit tests inside the project. We recommend developers do this very frequently -- run them from within Eclipse after you change code, and then run them before committing and certainly before creating a PR.

We'd like to enhance Strongback's Ant tasks to dynamically download the WPILib libraries, making it possible to set up Travis to automatically build and run all tests as soon as a PR is created. We hope to have this soon, and will document how to set everything up.