Suggestion for next year's WPILib

[Radical Pi]

A little idea I came up with after hearing how the LabVIEW teams have Autonomous handled. Basically in LabVIEW the Autonomous function is called asynchronously, and then terminated by the communications code at the end of autonomous.

I was thinking, why can't C++ teams do this as well? If the Autonomous function was simply called as a new Task, then it can easily be terminated by the main task when autonomous ends, avoiding the whole problem of autonomous code holding teleop from executing.

Additionally, if the autonomous code happens to run into a fatal error and die, the robot would be given a second chance. Since only the autonomous task would be terminated and not the main robot task, the teleop code will still be started on time and the robot is given a sort of second chance.

Most of the code to do this is already in WPILib. The Task class is already there to create the task. All that would need to be done is a few teaks to the SimpleRobot class and everything would work.

Comments?

[Joe Hershberger]

This is quite doable. We talked about doing it that way from the start, but through some (probably unwarranted) paranoia, the approach was not taken. Perhaps it could be an option at the least.

-Joe

[mikets]

It's good in theory but I do worry students may not understand multi-threaded programming. They need to learn inter-thread synchronization and exclusive access (e.g. semaphores). BTW, is it true that semaphores in vxWorks is not counting semaphores? I can't acquire the semaphore again in the same thread?

[Greg McKaskle]

I believe that VxWorks has all the semaphore types you could possibly want. If you are using ones wrapped by WPILib, they may be Boolean(noncounting), I don't have access to the code. Take a look.

Greg McKaskle

[gvarndell]

Originally Posted by mikets

BTW, is it true that semaphores in vxWorks is not counting semaphores? I can't acquire the semaphore again in the same thread?

Yes, vxWorks has counting semaphores -- had them forever.
See either the application or the kernel programmer's guides for more info.

Just gotta plug this one more time, Wind River Workbench has Help, and it's searchable -- give it a try.

[Slix]

If implemented properly, programmers shouldn't even need to make their code thread-safe... at least, if I understand it properly... (I'm newish to C++ and completely new to WPILib)

Inside StartCompetition(), the main task would start an autonomous task that runs Autonomous(). Then, the main task would wait until IsAutonomous() is no longer true. After that happens, it would stop the autonomous task and go on. Since the only thing that the main task would do is check IsAutonomous(), then the autonomous task could run without any worries about race conditions (so all that teams must know about Autonomous() is that it might stop in the middle of their code).

There is a problem though.. I haven't tested what stopping a task does. Does the task still unwind the call stack and call all destructors, or does it just stop immediately? If destructors are run, then does Task::Stop() block until the task is really dead? (The main task shouldn't continue until the autonomous task is truly dead so that there are no possible race conditions). If destructors aren't run, then the program becomes pretty unpredictable and there is no way to run any cleanup code.

Not sure if above is completely correct..

EDIT: I was checking the WindRiver manual for information about how tasks are deleted. Unfortunately, it seems that stopping a task does not unwind the call stack. Stopping a task ends it right where it is. (It also doesn't release any semaphores; I could not tell whether Synchronized protects against that) It would be ideal if stopping a task threw an exception in the task so that all destructors would be called, but since that doesn't seem possible, then some AutonomousEnd() function or something should be called right after the task is stopped to allow for some cleanup code... Better than nothing.

[Greg McKaskle]

I won't profess to know VxWorks inside and out, but killing threads is commonly a dangerous task if you do not control what the task is up to and allow it to get to a good place first. This is why LV Abort is somewhat cooperative and doesn't simply kill threads. It is still reasonable to do this within WPILib, just not as simple to pull off.

Greg McKaskle

[mikets]

I agree with Greg. A task either died unexpectedly or got killed is not a planned event. Therefore, the state of the whole program could be in jeopardy. If the task had allocated global resources, they were not released. If you restart the task, you may end up with duplicate resouces. The result could be totally unpredictable. In fact our code avoids spinning new threads unless the task is timing critical (need accurate timed execution). Even with most of the time critical scenarios, we found ways to do things coorpoeratively instead of needing to create new threads. I assume the threads in vxWorks are scheduled preemptively. Preemptive multi-tasking can interrupt you at any point in your code. That's why you need semaphores to protect a certain code region from being re-entered. Dealing with semaphore improperly will cause deadlock. That's not easy to debug especially if the deadlock is caused by some race conditions that don't reproduce predictably.

BTW, does anybody know the interval of a time slice in vxWorks? I found that if I created a Notifier with a given period, say 5 ms, my Notifier got called in 20+ ms instead. I suspect it has something to do with the timer tick interval in vxWorks. Am I correct? If so, what is that interval, 20 ms?

Thanks.

[slavik262]

Originally Posted by Greg McKaskle

I won't profess to know VxWorks inside and out, but killing threads is commonly a dangerous task if you do not control what the task is up to and allow it to get to a good place first. This is why LV Abort is somewhat cooperative and doesn't simply kill threads. It is still reasonable to do this within WPILib, just not as simple to pull off.

Greg McKaskle

It's not just VxWorks, but pretty much any OS - killing off a thread without giving it a chance to release any resources it might be using is generally bad. It's not that difficult, however, to make a thread that just loops until you send a message to it to exit. Then to kill the thread, you just send the exit message, the thread gets out of its main loop, cleans up, and exits.

Also, for a nice explanation of the VxWorks multitasking model, share and enjoy.

[Jared Russell]

This is exactly how we did our autonomous code this year (in Java). Basically, we decided that the "iterative"-style template makes the most intuitive sense for teleoperation, but that the "simple robot"-style sequential template leads to easier-to-understand-and-debug autonomous operation.

As a result, we used the iterative template, but created a separate class - AutonomousThread - that is kicked off by autonomousInit() (and killed by disabledInit() or teleopInit() ).

Here's where it gets cool - when the AutonomousThread starts, it is passed an array of type "Action", which is a custom interface that we made. Each Action has an "execute()" and a "cancel()" method. The convention is that every Action will return from "execute()" when it is done (e.g. DriveDistance would return when the robot has covered the desired distance) or when it has been canceled. We implemented "cancel()" as a method that would set a flag inside the Action that would then be polled within the "execute()" loop, since there really isn't a good/safe way to kill threads from afar otherwise.

So AutonomousThread looks a little bit like this:

public void run()
{
  for( int i = 0; i < mActions.length; i++ )
  {
    mCurrentAction = i;
    if( !mCancelled )
    {
      mActions[i].execute();
    }
  }
}

public synchronized void cancel()
{
  mCancelled = true;
  mActions[mCurrentAction].cancel();
}

Ain't polymorphism grand? (Even though this is Java, the same technique would work fine in C++).

Now, where it gets REALLY cool is where you can serialize Actions to a file on a PC, FTP it to the cRIO, and load a new autonomous mode without re-deploying...

If there is sufficient interest, I'll post our team's code at the end of the season.

[mikets]

Killing a thread is not too much a concern because it could be a controlled event (i.e. you can write code to monitor a termination event and clean up). It is the unexpected termination such as a fault that could be a problem although one can also use structured exception handling to do the clean up. However, it can clean up properly only if the code is expecting the fault (i.e. putting the code in the try block). In either case, if there are bugs in the code that do not expect a certain scenario, you can still hang the thread or fault without catching it.

Our code is also based on Iterative robot but we don't start a new thread. Instead, we have a structure that does cooperative multi-tasking. We also develop an event notification system and a generic state machine that makes the autonomous code almost script like. We could go further and write a parser to read a script from file too, but don't have time to do it this season.

The autonomous code looks something like this:

switch (state)
{
    case 0:
        // Do something such as drive forward 3 ft.
        // At the end of the drive, it will generate an event.
        nextstate = 1;
        WaitForEvents(event, nextstate);
        break;
    case 1:
        ...
        break;
}

The WaitForEvents is not blocking, so the code returns for executing other stuff until an event unblocks it.

[Greg McKaskle]

If the auto thread is politely behaved and checking a cancel or other notification, what is the benefit in forking a thread and killing it. It seems to me that the big benefit of killing the thread is to handle code that is spin-locked, yielded, looping to home in on a target, or otherwise not well-behaved.

Statistically, killing the thread will usually work, but if the thread has acquired a mutex, allocated I/O or other other OS resources, how is a chaperone to know what to delete, what to leave open, what to unlock. Plus depending on the implementation, the lock and other items may have thread-local storage, which means the info has been torn down and the chaperone doesn't have the ability to clean up. If enough shadowing is done with WPILib, those resources can be made safe for killing. OS resources would still be an issue, but should be sufficiently rare I'd think.

Greg McKaskle

[wireties]

Originally Posted by Greg McKaskle

If the auto thread is politely behaved and checking a cancel or other notification, what is the benefit in forking a thread and killing it. It seems to me that the big benefit of killing the thread is to handle code that is spin-locked, yielded, looping to home in on a target, or otherwise not well-behaved.

Statistically, killing the thread will usually work, but if the thread has acquired a mutex, allocated I/O or other other OS resources, how is a chaperone to know what to delete, what to leave open, what to unlock. Plus depending on the implementation, the lock and other items may have thread-local storage, which means the info has been torn down and the chaperone doesn't have the ability to clean up. If enough shadowing is done with WPILib, those resources can be made safe for killing. OS resources would still be an issue, but should be sufficiently rare I'd think.

Greg McKaskle

VxWorks will release mutex semaphores when a task dies if that flag is set when the semaphore is created. But in general, VxWorks does no cleanup (freeing memory, closing files, etc) when a task dies.

Hope this helps

[wireties]

Originally Posted by mikets

It's good in theory but I do worry students may not understand multi-threaded programming. They need to learn inter-thread synchronization and exclusive access (e.g. semaphores). BTW, is it true that semaphores in vxWorks is not counting semaphores? I can't acquire the semaphore again in the same thread?

VxWorks has counting semaphores, binary semaphores and mutex semaphores. The C functions to create them are semCCreate, semBCreate and semMCreate. After creation semTake and semGive are used for all 3 types. I think the POSIX semaphores are really VxWorks counting semaphores. The POSIX mutexes are implemented using VxWorks mutex semaphores.

VxWorks also has public (inter-process) versions of all these types but we don't use them for FIRST (processes are not used, just kernel mode tasks/threads)

Hope this helps

[dmlawrence]

Originally Posted by Radical Pi

Additionally, if the autonomous code happens to run into a fatal error and die, the robot would be given a second chance. Since only the autonomous task would be terminated and not the main robot task, the teleop code will still be started on time and the robot is given a sort of second chance.

Couldn't this be done in the current system using some well-placed exception handling?

Originally Posted by mikets

It's good in theory but I do worry students may not understand multi-threaded programming.

From the perspective of a student, I think it's completely doable to teach multithreading. Maybe not easy -- but if it was easy it wouldn't be FIRST...