[Thad House]

Originally Posted by virtuald

We did our image processing in python using OpenCV in a single thread, and didn't have any lag problems. Here are some thoughts that may help:

• Memory allocation of large images can get expensive. Don't allocate new images each time you read something in. Instead, find the size of the image, allocate a numpy array to store the image, and process it. Reuse the same array each time you grab a new image. Similarly, when doing processing steps that return a new image (like splitting, or colorspace transforms, or whatever), allocate your buffers once, and reuse the same buffers over and over again by using the 'dst' parameter.
• Be wary of accessing too many non-local variables in python. In our robot code (which is running python), we found that accessing a lot of things at module level instead of local level caused noticeable slowness. Presumably this is because each time python needs to do multiple lookups in each scope to find things.
• Keep in mind that in python 2.7, because of the GIL multithreading doesn't buy you a lot unless you are I/O bound

It works in a single thread on a computer because a powerful computer can do all the tracking faster then 33ms. When I put the code on the beaglebone, it takes about 60 ms just to run the threshold and morphology commands. So that causes the camera buffer, which gets a new frame every 33ms, to lag. if the beaglebone had more power a single thread would work, but because it does not the 2 threads are needed.

As for grabbing the image directly, openCV has no way of just opening a jpg over the network, so some other library had to be used. I was using urllib because that was the only one i could find working. And it would take 70 ms just to connect to the camera and download the image to an already allocated buffer.