Python Testing Strategies -- How did you do it?

[virtuald]

It seems to me that python is a great language, but it is a very dynamic language. And while that is one of the things that makes it so awesome -- it's also one of the things that makes thorough testing absolutely vital since misspelling a variable name can crash your program if you're not prepared for this.

One way that we dealt with this is by making sure that all of the complex interactions of our robot was separated out in modules, and that the complexity stayed out of the main loop. Then, functions were called in the main loop, but each function was surrounded by a try..except block that *ideally* makes it so that if one component of our robot fails (ie, I accidentally added the wrong variable name to the arm code), that the other parts would still run. So the blocks would all look something like this:

try:
    # call some complicated function in another module
except:
    # swallow any exceptions if we're in a real match...
    if not wpilib.DriverStation.GetInstance().IsFMSAttached():
        raise

Another strategy that we used was creating a test harness and a 'fake' wpilib that contained all of the classes and functions that our code used from the real wpilib. Then our code would just import wpilib or our fake one depending on whether it was actually in the robot or not.

try:
    import wpilib
except:
    import fake_wpilib as wpilib

Then the test harness would import the robot, call run(), and then use the robot object to call Disabled(), Autonomous(), OperatorControl() and so on in the correct order.

The fake wpilib contained all the classes/functions from wpilib that we used -- except parameters that didn't really matter for our testing would just not be used, and many functions were just implemented as 'pass'. For example, our watchdog class was implemented like this:

class Watchdog(object):
    
    def Feed(self):
        pass
        
    def SetEnabled(self, enable):
        pass
        
    def SetExpiration(self, period):
        pass

Now, other parts of wpilib were implemented in such a way that the Set() or Get() functions set/got their values from the class internally, then our test harness would just reach into the class (since there is no such thing as 'private' variables in python) and make sure that whatever was expected to happen at a particular point in time actually happened. Similarly, we could reach into the classes and set the values to provoke a particular action from our code. So we would have a joystick class that looks like this...

class Joystick(object):
    
    def __init__(self, port):
        self.x = 0
        self.y = 0
        self.z = 0
        
        # trigger, top, 3... 
        self.buttons = [ False, False, False, False, False, False, False, False, False, False ]
        
    def GetRawButton(self, number):
        return self.buttons[number-1]
        
    def GetTop(self):
        return self.buttons[1]
        
    def GetTrigger(self):
        return self.buttons[0]
        
    def GetX(self):
        return self.x
        
    def GetY(self):
        return self.y
        
    def GetZ(self):
        return self.z

And then the test harness would just reach into the robot and set the appropriate things to simulate a button being pressed, or the joystick being moved.. whatever.

As you can see, the chief advantage of this is that we could run such a program directly on our programming laptop before the program is loaded onto the cRio, and hopefully catch any syntax errors or whatever before we try it on the robot. We added some simple state machines to model how the robot might be stimulated by various sensors and whatever, and caught a few bugs that way too. Of course, there's all sorts of things that one could do with this approach.

I'll probably publish our team's source code later this week, then you can look at it yourself. Any other useful strategies that others used?

[Jeanne Boyarsky]

Dustin,
Thanks for sharing this. I was hoping there was an easier way to do it of course, but knowing one team did this implies there is not. Have you considered contributing your library of stubs to the existing Python libraries (or packaging it separately) so other teams can use it? I think that is going to prove even more valuable than your "real code"

Jeanne

[virtuald]

Originally Posted by Jeanne Boyarsky

Dustin,
Thanks for sharing this. I was hoping there was an easier way to do it of course, but knowing one team did this implies there is not. Have you considered contributing your library of stubs to the existing Python libraries (or packaging it separately) so other teams can use it? I think that is going to prove even more valuable than your "real code"

Jeanne

I think I'm going to release it all together [real + stubs + test harness] for the moment, since it's pretty specific to our team's robot right now. And the other thing too -- it's not like it's particularly hard to do this stuff once you have a good idea how to implement it. I think it took me 3 hours total to write the test harness/fake wpilib and such.

In the past I made a more full-featured test harness for WPILib/C++, but I haven't really been motivated to do that for python yet. Maybe I'll play with PyQT at some point.

[blakeelias]

Originally Posted by virtuald

And then the test harness would just reach into the robot and set the appropriate things to simulate a button being pressed, or the joystick being moved.. whatever.

That's awesome! I was looking for a way to automatically make a dummy library, but I couldn't find anything that would simulate the interactions between all the classes. It looks like yours catches any syntax errors and even some runtime errors -- better than a local compiler!

[blakeelias]

You also hacked winpdb to run on the cRIO, right? How did that work out?

[virtuald]

Originally Posted by blakeelias

That's awesome! I was looking for a way to automatically make a dummy library, but I couldn't find anything that would simulate the interactions between all the classes. It looks like yours catches any syntax errors and even some runtime errors -- better than a local compiler!

Yeah, my intent was to use it sorta like a compiler, so we wouldn't have to worry so much about runtime errors. But -- it's not automatically generated by anything, we created the harness and classes by writing the code ourselves. But once it was implemented, it was really easy to add new things to it, and was a great little set of tests.

Originally Posted by blakeelias

You also hacked winpdb to run on the cRIO, right? How did that work out?

Well, it worked, though we ended up not needing to use it particularly often at all. For python I've just been finding it easier to use print statements to debug things. Once or twice I've used winpdb... but, our code isn't particularly complex, so print() works just fine

[DjMaddius]

Wow, I really like this idea. I've started preparing this years event as we are doing a build for LabVIEW and Python, Ive been doing research and this will be VERY useful. Using this with some sort of Dynamic UI with tkinter for the fake_wpilib class would be awesome. It would require some time to work out all the bugs in making it very dynamic and yet still be able to use it as simple as you have it, but its worth the time over the next off-season to get this done.

[tux]

One thing that I like about Python is the simplicity of debugging with print statements. I see that the sample code has some prints to show what is happening ("Running teleop()", etc).

Where does that output go? How do I tail the output?

[virtuald]

Originally Posted by tux

One thing that I like about Python is the simplicity of debugging with print statements. I see that the sample code has some prints to show what is happening ("Running teleop()", etc).

Where does that output go? How do I tail the output?

You can use the netconsole that comes with the FRC Tools to look at the output, or alternatively you can use my NetConsole implemented in python.

[tux]

I got the Python netconsole working, but that does not allow 2-way communication, right?
So, how do you type "reboot" when needed?

I tried the FRC NetConsole, but could not get it to work even 1-way.

[connor.worley]

Originally Posted by tux

I got the Python netconsole working, but that does not allow 2-way communication, right?
So, how do you type "reboot" when needed?

I tried the FRC NetConsole, but could not get it to work even 1-way.

You can reimage your cRIO to get it working, just make sure to check "Enable NetConsole."

[virtuald]

Originally Posted by tux

I got the Python netconsole working, but that does not allow 2-way communication, right?
So, how do you type "reboot" when needed?

I tried the FRC NetConsole, but could not get it to work even 1-way.

The FRC NetConsole is two way... I never got around to making the python NetConsole do two way (but patches are accepted!).

In order to receive the UDP communications for the NetConsole, I believe the subnet mask has to match on the cRio and the receiving PC (and possibly the router too). I've had problems with the FRC NetConsole when they didn't match.

[rbmj]

Originally Posted by virtuald

The FRC NetConsole is two way... I never got around to making the python NetConsole do two way (but patches are accepted!).

In order to receive the UDP communications for the NetConsole, I believe the subnet mask has to match on the cRio and the receiving PC (and possibly the router too). I've had problems with the FRC NetConsole when they didn't match.

I have successful two-way communication in a python netconsole:

https://github.com/rbmj/netconsole

It works on linux (I still need to test it on windows). I tried to be cross-platform, but sometimes things don't work as they're supposed to.

There are some issues with the console picking up packets that were sent out to broadcast, even though they're on a different port (so I suspect that they're being echoed by the cRIO to signal receipt). If someone wants to take a closer look at that and submit a patch...

[tux]

So, is typing "reboot" to the netconsole faster than using the reset button on the cRIO, or is it essentially the same thing?

[rbmj]

In my experience it's the same.