Multi-threading issue?

[adamdb]

We are trying to run the camera code on a separate thread. We are basically using the Target class as-is from the CircleTrackerDemo program and have pulled out the relevant bits of code from the main program that call it. The code is as follows:

public class CameraThread extends Thread
{

    private double kScoreThreshold = .01;
    private AxisCamera cam;
    private Target[] targets;
    private boolean targetFound = false;
    private boolean active = false;

    public CameraThread()
    {
        // Setup camera
        cam = AxisCamera.getInstance();
        cam.writeResolution(AxisCamera.ResolutionT.k320x240);
        cam.writeBrightness(0);
    }

    public void run()
    {
        while (true)
        {
            if (active)
            {
                if (cam.freshImage())
                {
                    ColorImage image;
                    try
                    {
                        image = cam.getImage();

                        Thread.yield();
                        targets = Target.findCircularTargets(image);
                        Thread.yield();
                        image.free();
                        if (targets.length == 0 || targets[0].m_score < kScoreThreshold)
                        {
                            this.targetFound = false;
                        }
                        else
                        {
                            this.targetFound = true;
                        }
                    }
                    catch (AxisCameraException ex)
                    {
                        ex.printStackTrace();
                    }
                    catch (NIVisionException ex)
                    {
                        ex.printStackTrace();
                    }
                }
            }
            else
            {
                targetFound = false;

                try
                {
                    Thread.sleep(5);
                }
                catch (InterruptedException ex)
                {
                    ex.printStackTrace();
                }
            }
        }
    }

    public void setActive(boolean active)
    {
        this.active = active;
    }

    public boolean isActive()
    {
        return active;
    }

    public boolean isTargetFound()
    {
        return targetFound;
    }

    public Target[] getTargets()
    {
        return targets;
    }
}

As you can see we set this up so we can tell the thread whether or not to be "active", which just determines whether or not it gets a new image from the camera and goes through the ellipse detection routines.

Our problem is that anytime the "active" flag is true in this routine and it is doing its thing getting images from the camera and detecting targets (which does work), our motor outputs become "jerky" as though the processor is off doing something else and can't feed the motor outputs. The motors actually turn off briefly before resuming their previous speed. We are using all Jaguars for motor control and are using the CAN network. I would think that once we send the command out the CAN network to a motor that the Jaguars would keep the speed the same unless told differently.

We do have the watchdog timeout set at .75 seconds, and we never see the "Watchdog not fed" message. We tried adding frequent "Thread.yield()" statements in this class and in the Target class, but if anything that seemed to make the problem worse. We tried taking all of the Thread.yield() calls out entirely, and tried liberal Thread.sleep(1) calls (which usually works in multi-threaded PC apps) with no change in behavior.

This thread can be running, but with the active flag set to false and we do not see the issue, so it has to be something to do with the camera code itself.

So, does this look like it should work? Is anyone else seeing similar issues using code from the CircleTrackerDemo app? Is anyone else multi-threading the camera code in a similar or different way?

As always any suggestions are appreciated.

[timothyb89]

How are you actually running the thread code? From the looks of things, your code could be simply calling the run() method and not Thread.start()- doing that would prevent a new thread from starting and the camera code would just run in the same thread as everything else, causing the problems you're describing.

Instead of

public class CameraThread extends Thread {
    ...
}

try using:

public class CameraThread implements Runnable {
    ...
}

and then start the thread with (either in CameraThread's constructor or wherever you make an instance of it):

new Thread(someCameraThreadInstance).start();

Hopefully that could clear up some issues- although there's technically nothing wrong with just extending Thread, it can lead to some issues down the line that aren't generally a problem when just using Runnable.

If you need some examples, we've done something a lot like what you're trying to do with our code: Camera controller and thread starting code (line 64).

[Geek 2.0]

Why is it necessary to yield/sleep the thread? I would assume it's because we want to let the processor catch up with us, but I'm not really sure. I'm just starting to learn about threads and such, so I don't know much at this point.

[derekwhite]

Here are some things to try:

1) Try a smaller image size (k160x120). This will vastly reduce the amount of data flowing through the system (camera->cRIO->jpeg decompress->B&W conversion->findTargets, and cRIO->dashboard too).

2) Experiment with different threshold settings for the CurveOptions in Target. Try changing 45 -> 100, but watch what happens to accuracy. This will speed up findTargets.

3) Double check how many frames per second the camera itself is sending. The cRIO may be able to process 2-15 fps, so there's not much point in generating more frames than needed. There is not a Java interface to max fps, but you can use the web interface.

4) Limit how many fps YOU are trying to process:
As a start, you can set the bar low at 2 fps (or 500ms). Then you can time how long your image processing took, subtract that from 500, then sleep for the remainder. That will leave CPU time for other robot tasks. If the other suggestions work out for you you can try for more fps.

[derekwhite]

Geek 2.0:

As far as yield goes, it's a way to tell the system to pick a thread to run. The system may pick the same thread to run, or it may pick a different thread. I believe that some Java VMs ignore the yield method completely. The Java running on the cRIO DOES pay attention to the yield method, but it may not be necessary (more suggestions later).

The sleep() method is very useful, especially for robots. It tells the system to stop running a thread for a certain number of milliseconds. Consider the original poster's code, minus the camera code and the sleep:

   public void run() {
        while (true) {
            if (active) {
               // camera stuff
            } else  {
                targetFound = false;
            }
        }
    }

This is now a classic "busy loop". It will burn CPU cycles until the system decides to run another thread.

Two classic cases that come up, especially in robotics, are doing a task either periodically or in response to some event.

For the periodic case, you can use sleep() directly, or use the java.util.Timer class.

For the event driven case, there are 3 possibilities:
1) The event comes from Java. One Java thread is trying to tell another Java thread that something happened. The best approach is to use wait/notify.

2) The event is coming externally (from C code or the system), and there is a C function that waits until the event occurs (socket reads, Semaphore.take, etc).

3) The event is coming externally (from C code or the system), and there is NOT a C function that waits until the event occurs. cam.getImage(), for example. In this case, the code needs to "poll" (keep checking) for the event, but do so in a way that doesn't use all the CPU. So this bought us back to a period task.

Finally I'll mention that for Java on the cRIO, Thread.yield calls can be helpful if you have a thread that is doing a lot of straight-line code (not a lot of loop branches), and not any:
- sleep() calls
- notify() calls
- IO
- calls to "blocking external functions" (which include the camera and vision processing functions and talking to the driver station).