Python Testing Strategies -- How did you do it?

[virtuald]

It seems to me that python is a great language, but it is a very dynamic language. And while that is one of the things that makes it so awesome -- it's also one of the things that makes thorough testing absolutely vital since misspelling a variable name can crash your program if you're not prepared for this.

One way that we dealt with this is by making sure that all of the complex interactions of our robot was separated out in modules, and that the complexity stayed out of the main loop. Then, functions were called in the main loop, but each function was surrounded by a try..except block that *ideally* makes it so that if one component of our robot fails (ie, I accidentally added the wrong variable name to the arm code), that the other parts would still run. So the blocks would all look something like this:

try:
    # call some complicated function in another module
except:
    # swallow any exceptions if we're in a real match...
    if not wpilib.DriverStation.GetInstance().IsFMSAttached():
        raise

Another strategy that we used was creating a test harness and a 'fake' wpilib that contained all of the classes and functions that our code used from the real wpilib. Then our code would just import wpilib or our fake one depending on whether it was actually in the robot or not.

try:
    import wpilib
except:
    import fake_wpilib as wpilib

Then the test harness would import the robot, call run(), and then use the robot object to call Disabled(), Autonomous(), OperatorControl() and so on in the correct order.

The fake wpilib contained all the classes/functions from wpilib that we used -- except parameters that didn't really matter for our testing would just not be used, and many functions were just implemented as 'pass'. For example, our watchdog class was implemented like this:

class Watchdog(object):
    
    def Feed(self):
        pass
        
    def SetEnabled(self, enable):
        pass
        
    def SetExpiration(self, period):
        pass

Now, other parts of wpilib were implemented in such a way that the Set() or Get() functions set/got their values from the class internally, then our test harness would just reach into the class (since there is no such thing as 'private' variables in python) and make sure that whatever was expected to happen at a particular point in time actually happened. Similarly, we could reach into the classes and set the values to provoke a particular action from our code. So we would have a joystick class that looks like this...

class Joystick(object):
    
    def __init__(self, port):
        self.x = 0
        self.y = 0
        self.z = 0
        
        # trigger, top, 3... 
        self.buttons = [ False, False, False, False, False, False, False, False, False, False ]
        
    def GetRawButton(self, number):
        return self.buttons[number-1]
        
    def GetTop(self):
        return self.buttons[1]
        
    def GetTrigger(self):
        return self.buttons[0]
        
    def GetX(self):
        return self.x
        
    def GetY(self):
        return self.y
        
    def GetZ(self):
        return self.z

And then the test harness would just reach into the robot and set the appropriate things to simulate a button being pressed, or the joystick being moved.. whatever.

As you can see, the chief advantage of this is that we could run such a program directly on our programming laptop before the program is loaded onto the cRio, and hopefully catch any syntax errors or whatever before we try it on the robot. We added some simple state machines to model how the robot might be stimulated by various sensors and whatever, and caught a few bugs that way too. Of course, there's all sorts of things that one could do with this approach.

I'll probably publish our team's source code later this week, then you can look at it yourself. Any other useful strategies that others used?