Chief Delphi - Python Testing Strategies -- How did you do it?

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Python (http://www.chiefdelphi.com/forums/forumdisplay.php?f=187)

- - Python Testing Strategies -- How did you do it? (http://www.chiefdelphi.com/forums/showthread.php?t=94419)

Python Testing Strategies -- How did you do it?

It seems to me that python is a great language, but it is a very dynamic language. And while that is one of the things that makes it so awesome -- it's also one of the things that makes thorough testing absolutely vital since misspelling a variable name can crash your program if you're not prepared for this.

One way that we dealt with this is by making sure that all of the complex interactions of our robot was separated out in modules, and that the complexity stayed out of the main loop. Then, functions were called in the main loop, but each function was surrounded by a try..except block that *ideally* makes it so that if one component of our robot fails (ie, I accidentally added the wrong variable name to the arm code), that the other parts would still run. So the blocks would all look something like this:

Code:

try:

    # call some complicated function in another module

except:

    # swallow any exceptions if we're in a real match...

    if not wpilib.DriverStation.GetInstance().IsFMSAttached():

        raise

Another strategy that we used was creating a test harness and a 'fake' wpilib that contained all of the classes and functions that our code used from the real wpilib. Then our code would just import wpilib or our fake one depending on whether it was actually in the robot or not.

Code:

try:

    import wpilib

except:

    import fake_wpilib as wpilib

Then the test harness would import the robot, call run(), and then use the robot object to call Disabled(), Autonomous(), OperatorControl() and so on in the correct order.

The fake wpilib contained all the classes/functions from wpilib that we used -- except parameters that didn't really matter for our testing would just not be used, and many functions were just implemented as 'pass'. For example, our watchdog class was implemented like this:

Code:

class Watchdog(object):

    

    def Feed(self):

        pass

        

    def SetEnabled(self, enable):

        pass

        

    def SetExpiration(self, period):

        pass

Now, other parts of wpilib were implemented in such a way that the Set() or Get() functions set/got their values from the class internally, then our test harness would just reach into the class (since there is no such thing as 'private' variables in python) and make sure that whatever was expected to happen at a particular point in time actually happened. Similarly, we could reach into the classes and set the values to provoke a particular action from our code. So we would have a joystick class that looks like this...

Code:

class Joystick(object):

    

    def __init__(self, port):

        self.x = 0

        self.y = 0

        self.z = 0

        

        # trigger, top, 3... 

        self.buttons = [ False, False, False, False, False, False, False, False, False, False ]

        

    def GetRawButton(self, number):

        return self.buttons[number-1]

        

    def GetTop(self):

        return self.buttons[1]

        

    def GetTrigger(self):

        return self.buttons[0]

        

    def GetX(self):

        return self.x

        

    def GetY(self):

        return self.y

        

    def GetZ(self):

        return self.z

And then the test harness would just reach into the robot and set the appropriate things to simulate a button being pressed, or the joystick being moved.. whatever.

As you can see, the chief advantage of this is that we could run such a program directly on our programming laptop before the program is loaded onto the cRio, and hopefully catch any syntax errors or whatever before we try it on the robot. We added some simple state machines to model how the robot might be stimulated by various sensors and whatever, and caught a few bugs that way too. Of course, there's all sorts of things that one could do with this approach.

I'll probably publish our team's source code later this week, then you can look at it yourself. Any other useful strategies that others used? :)

Re: Python Testing Strategies -- How did you do it?

Dustin,
Thanks for sharing this. I was hoping there was an easier way to do it of course, but knowing one team did this implies there is not. Have you considered contributing your library of stubs to the existing Python libraries (or packaging it separately) so other teams can use it? I think that is going to prove even more valuable than your "real code"

Jeanne

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by Jeanne Boyarsky (Post 1051006)

I think I'm going to release it all together [real + stubs + test harness] for the moment, since it's pretty specific to our team's robot right now. And the other thing too -- it's not like it's particularly hard to do this stuff once you have a good idea how to implement it. I think it took me 3 hours total to write the test harness/fake wpilib and such.

In the past I made a more full-featured test harness for WPILib/C++, but I haven't really been motivated to do that for python yet. Maybe I'll play with PyQT at some point.

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by virtuald (Post 1050980)

And then the test harness would just reach into the robot and set the appropriate things to simulate a button being pressed, or the joystick being moved.. whatever.

That's awesome! I was looking for a way to automatically make a dummy library, but I couldn't find anything that would simulate the interactions between all the classes. It looks like yours catches any syntax errors and even some runtime errors -- better than a local compiler!

Re: Python Testing Strategies -- How did you do it?

You also hacked winpdb to run on the cRIO, right? How did that work out?

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by blakeelias (Post 1051088)

Yeah, my intent was to use it sorta like a compiler, so we wouldn't have to worry so much about runtime errors. But -- it's not automatically generated by anything, we created the harness and classes by writing the code ourselves. But once it was implemented, it was really easy to add new things to it, and was a great little set of tests.

Quote:

Originally Posted by blakeelias (Post 1051101)

You also hacked winpdb to run on the cRIO, right? How did that work out?

Well, it worked, though we ended up not needing to use it particularly often at all. For python I've just been finding it easier to use print statements to debug things. Once or twice I've used winpdb... but, our code isn't particularly complex, so print() works just fine :)

Re: Python Testing Strategies -- How did you do it?

Wow, I really like this idea. I've started preparing this years event as we are doing a build for LabVIEW and Python, Ive been doing research and this will be VERY useful. Using this with some sort of Dynamic UI with tkinter for the fake_wpilib class would be awesome. It would require some time to work out all the bugs in making it very dynamic and yet still be able to use it as simple as you have it, but its worth the time over the next off-season to get this done.

Re: Python Testing Strategies -- How did you do it?

One thing that I like about Python is the simplicity of debugging with print statements. I see that the sample code has some prints to show what is happening ("Running teleop()", etc).

Where does that output go? How do I tail the output?

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by tux (Post 1124425)

You can use the netconsole that comes with the FRC Tools to look at the output, or alternatively you can use my NetConsole implemented in python.

Re: Python Testing Strategies -- How did you do it?

I got the Python netconsole working, but that does not allow 2-way communication, right?
So, how do you type "reboot" when needed?

I tried the FRC NetConsole, but could not get it to work even 1-way.

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by tux (Post 1127424)

You can reimage your cRIO to get it working, just make sure to check "Enable NetConsole."

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by tux (Post 1127424)

The FRC NetConsole is two way... I never got around to making the python NetConsole do two way (but patches are accepted!).

In order to receive the UDP communications for the NetConsole, I believe the subnet mask has to match on the cRio and the receiving PC (and possibly the router too). I've had problems with the FRC NetConsole when they didn't match.

Re: Python Testing Strategies -- How did you do it?

Quote:

Originally Posted by virtuald (Post 1127538)

I have successful two-way communication in a python netconsole:

https://github.com/rbmj/netconsole

It works on linux (I still need to test it on windows). I tried to be cross-platform, but sometimes things don't work as they're supposed to.

There are some issues with the console picking up packets that were sent out to broadcast, even though they're on a different port (so I suspect that they're being echoed by the cRIO to signal receipt). If someone wants to take a closer look at that and submit a patch...

Re: Python Testing Strategies -- How did you do it?

So, is typing "reboot" to the netconsole faster than using the reset button on the cRIO, or is it essentially the same thing?

Re: Python Testing Strategies -- How did you do it?

In my experience it's the same.