Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Programming (http://www.chiefdelphi.com/forums/forumdisplay.php?f=51)
-   -   Fixing opencv lag (http://www.chiefdelphi.com/forums/showthread.php?t=123390)

Thad House 18-12-2013 23:14
Fixing opencv lag
 
I am trying to get camera tracking working on my beaglebone black. I want to use the axis camera to do this, and I can get that part working. The problem is that the tracking is limited to about 15 frames per second, and the camera is sending at 30fps. This is causing images to just get stored in the buffer, which then starts lagging badly. Does anybody know how to either shrink the buffer down to 1 image, read from the back of the buffer, or loop through the buffer to the end quickly. Or if there are any other suggestions that would help too.

Also this is unrelated, but does anybody know why no networking programs allow reading from the back of the buffer. I have wanted to do that MANY times, but nothing ever supports it, and just curious if anybody knows why that is.

otherguy 19-12-2013 00:11
Re: Fixing opencv lag
 
This is basically what I was going to write, but Zaphod said it better...
Quote:
...have a separate capture thread, in addition to your [image processing] thread. The capture thread keeps reading in frames from the buffer whenever a new frame comes in and stores it in a "recent frame" image object. When the [image processing] thread needs the most recent frame, it locks a mutex for thread safety, copies the most recent frame into another object and frees the mutex so that the capture thread continues reading in new frames.
http://stackoverflow.com/questions/1...-reading-issue

Thad House 19-12-2013 00:15
Re: Fixing opencv lag
 
Yeah I was just about to post that I read that. I did it without the locks because I don't know how to do locks in python, but thats coming up next.

Thanks

billbo911 19-12-2013 01:42
Re: Fixing opencv lag
 
Quote:
Originally Posted by sst.thad (Post 1314457)
I am trying to get camera tracking working on my beaglebone black. I want to use the axis camera to do this, and I can get that part working. The problem is that the tracking is limited to about 15 frames per second, and the camera is sending at 30fps. This is causing images to just get stored in the buffer, which then starts lagging badly. Does anybody know how to either shrink the buffer down to 1 image, read from the back of the buffer, or loop through the buffer to the end quickly. Or if there are any other suggestions that would help too.
Is there a reason you are not just requesting one frame and then processing it before requesting the next?
In other words, don't allow the camera to "free run".
You might also try setting the Axis to run at 15 fps. As long as you can process at 15 fps, you should not be filling the buffer.

Thad House 19-12-2013 02:06
Re: Fixing opencv lag
 
I tried just requesting a single frame at a time, but it would take 100ms from requesting the image to actually receive the image. It was a little too much for me.

As for limiting the fps it wasnt working for some reason.

wireties 19-12-2013 02:18
Re: Fixing opencv lag
 
Quote:
Originally Posted by sst.thad (Post 1314490)
I tried just requesting a single frame at a time, but it would take 100ms from requesting the image to actually receive the image. It was a little too much for me.

As for limiting the fps it wasnt working for some reason.
Just toss every other frame? Have you tried using C or C++ instead of Python?

Thad House 19-12-2013 02:46
Re: Fixing opencv lag
 
Quote:
Originally Posted by wireties (Post 1314493)
Just toss every other frame? Have you tried using C or C++ instead of Python?
I know the python libraries. And the threaded way works just fine, and most likely better.

Greg McKaskle 19-12-2013 05:49
Re: Fixing opencv lag
 
The WPI libraries in LV include vision code. The code uses parallel loops -- results in parallel threads -- in case the user has set the camera faster than they can process. This is effective for keeping the lag to a minimum.

You may also want to lower the camera frame rate, though. First, there is some overhead involved in processing the frames, though hopefully you aren't decompressing the images or doing any of the expensive things. Second, the two thread approach will result in much less certainty about when the image was acquired. If you are trying to estimate time, it is likely better to run the camera at a slower, but known rate that doesn't buffer images.

The other approach of requesting an image only when you want one was used in older versions of WPILib. It worked/works far better on the 206 model than on the newer Axis cameras. There is indeed quite a bit of setup overhead and the mjpeg approach is almost required to get above 10fps.

Greg McKaskle

Thad House 19-12-2013 14:20
Re: Fixing opencv lag
 
Quote:
Originally Posted by Greg McKaskle (Post 1314511)
The WPI libraries in LV include vision code. The code uses parallel loops -- results in parallel threads -- in case the user has set the camera faster than they can process. This is effective for keeping the lag to a minimum.

You may also want to lower the camera frame rate, though. First, there is some overhead involved in processing the frames, though hopefully you aren't decompressing the images or doing any of the expensive things. Second, the two thread approach will result in much less certainty about when the image was acquired. If you are trying to estimate time, it is likely better to run the camera at a slower, but known rate that doesn't buffer images.

The other approach of requesting an image only when you want one was used in older versions of WPILib. It worked/works far better on the 206 model than on the newer Axis cameras. There is indeed quite a bit of setup overhead and the mjpeg approach is almost required to get above 10fps.

Greg McKaskle
We are using a Beaglebone black, which makes us run openCV. If it could run labVIEW we would use it. For some reason limiting the FPS was not working.

I ran the threading overnight last night, and it ran for 12 hours without missing any frames or starting to lag. How I had it set up was grab the image in the fast looping thread, and store that. Then in the slow thread i would grab a frame as fast as the tracking could loop, which was about 60ms. Because at 30fps that is a new frame every 33ms, that means at maximum the image is 2 frames old, which is perfectly fine.

The reason we are on a beaglebone instead of the dashboard is i have heard stories of FTA's requiring dashboards to get shut off, and we don't want to be stuck if that happens. So we want it to be onboard.

Greg McKaskle 19-12-2013 23:25
Re: Fixing opencv lag
 
Totally understand about the approach. I have a bone at my desk for precisely this sort of experimentation. I was relating it to how LV implemented it in order to validate your approach.

I'd be interested to hear if FTAs need to limit dashboards this year. Some venue's are less wifi friendly, so it may still happen, but it is not the expected route.

Greg McKaskle

virtuald 20-12-2013 13:48
Re: Fixing opencv lag
 
We did our image processing in python using OpenCV in a single thread, and didn't have any lag problems. Here are some thoughts that may help:
  • Memory allocation of large images can get expensive. Don't allocate new images each time you read something in. Instead, find the size of the image, allocate a numpy array to store the image, and process it. Reuse the same array each time you grab a new image. Similarly, when doing processing steps that return a new image (like splitting, or colorspace transforms, or whatever), allocate your buffers once, and reuse the same buffers over and over again by using the 'dst' parameter.
  • Be wary of accessing too many non-local variables in python. In our robot code (which is running python), we found that accessing a lot of things at module level instead of local level caused noticeable slowness. Presumably this is because each time python needs to do multiple lookups in each scope to find things.
  • Keep in mind that in python 2.7, because of the GIL multithreading doesn't buy you a lot unless you are I/O bound

virtuald 20-12-2013 13:52
Re: Fixing opencv lag
 
Quote:
Originally Posted by sst.thad (Post 1314490)
I tried just requesting a single frame at a time, but it would take 100ms from requesting the image to actually receive the image. It was a little too much for me.
Were you reusing the same image buffer each time, or were you just using the returned image buffer from read()? If you are allocating a new image each time (the default behavior), that explains why you had so much lag receiving the image.

Thad House 20-12-2013 16:38
Re: Fixing opencv lag
 
Quote:
Originally Posted by virtuald (Post 1315010)
We did our image processing in python using OpenCV in a single thread, and didn't have any lag problems. Here are some thoughts that may help:
  • Memory allocation of large images can get expensive. Don't allocate new images each time you read something in. Instead, find the size of the image, allocate a numpy array to store the image, and process it. Reuse the same array each time you grab a new image. Similarly, when doing processing steps that return a new image (like splitting, or colorspace transforms, or whatever), allocate your buffers once, and reuse the same buffers over and over again by using the 'dst' parameter.
  • Be wary of accessing too many non-local variables in python. In our robot code (which is running python), we found that accessing a lot of things at module level instead of local level caused noticeable slowness. Presumably this is because each time python needs to do multiple lookups in each scope to find things.
  • Keep in mind that in python 2.7, because of the GIL multithreading doesn't buy you a lot unless you are I/O bound
It works in a single thread on a computer because a powerful computer can do all the tracking faster then 33ms. When I put the code on the beaglebone, it takes about 60 ms just to run the threshold and morphology commands. So that causes the camera buffer, which gets a new frame every 33ms, to lag. if the beaglebone had more power a single thread would work, but because it does not the 2 threads are needed.

As for grabbing the image directly, openCV has no way of just opening a jpg over the network, so some other library had to be used. I was using urllib because that was the only one i could find working. And it would take 70 ms just to connect to the camera and download the image to an already allocated buffer.

billbo911 20-12-2013 17:04
Re: Fixing opencv lag
 
Quote:
Originally Posted by virtuald (Post 1315011)
Were you reusing the same image buffer each time, or were you just using the returned image buffer from read()? If you are allocating a new image each time (the default behavior), that explains why you had so much lag receiving the image.
I would definitely like to hear more about this approach.
I am no expert on the efficient use of OpenCV, and what you are proposing just might be the key to get rid of the minor delay we are seeing in our images. We can process 15-20 fps, but there is a small, but noticeable lag in our images.

We are using a USB camera and a PCDuino, but otherwise it is mostly the same.

Thad House 20-12-2013 18:41
Re: Fixing opencv lag
 
Quote:
Originally Posted by billbo911 (Post 1315140)
I would definitely like to hear more about this approach.
I am no expert on the efficient use of OpenCV, and what you are proposing just might be the key to get rid of the minor delay we are seeing in our images. We can process 15-20 fps, but there is a small, but noticeable lag in our images.

We are using a USB camera and a PCDuino, but otherwise it is mostly the same.
Does the lag show up before processing or after processing. If it shows up before processing that could be a solution. If it shows up after processing then its the actual processing time that could be causing the lag. One way I liked to test this is to put a timestamp at the beginning of the processing, then another at key points in the processing. Then you can calculate the difference and see how much time each process is taking. I used this on ours to see that the morphology and thresholding took about 50ms to do on a 320x240 image. This would cause any processed image to be more behind then that, at which point is easily visible.

gluxon 20-12-2013 22:30
Re: Fixing opencv lag
 
Request a video stream from VAPIX with the "fps" HTTP parameter. Details are on page 12 of the VAPIX API doc -> http://www.axis.com/files/manuals/va...37_en_1307.pdf

And if it helps, there's an incomplete Node.js implementation of VAPIX that supports a video stream with the slowed down fps you need. https://github.com/gluxon/node-vapix

This answers your question, although I feel like the problem will still occur, just over a longer period of time with this change.

Edit: I would actually go with what virtuald said in his post for the solution. Sounds like a more complete way of handling things.

virtuald 21-12-2013 12:06
Re: Fixing opencv lag
 
Quote:
Originally Posted by billbo911 (Post 1315140)
I would definitely like to hear more about this approach.
I am no expert on the efficient use of OpenCV, and what you are proposing just might be the key to get rid of the minor delay we are seeing in our images. We can process 15-20 fps, but there is a small, but noticeable lag in our images.

We are using a USB camera and a PCDuino, but otherwise it is mostly the same.
Our implementation is available on my website if you want to look at it. Look for kwarqs2013.py.

virtuald 21-12-2013 12:11
Re: Fixing opencv lag
 
Quote:
Originally Posted by sst.thad (Post 1315111)
As for grabbing the image directly, openCV has no way of just opening a jpg over the network.
Not true. If you have ffmpeg support enabled (which it is by default in most opencv builds), you can retrieve the mjpeg stream directly from the camera via FFMPEG. Something similar to the following (error checking omitted):

Code:
vc = cv2.VideoCapture()
vc.open('http://%s/mjpg/video.mjpg' % camera_host)

h = vc.get(cv2.cv.CV_CAP_PROP_FRAME_WIDTH)
w = vc.get(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT)

capture_buffer = np.empty(shape=(h, w, 3), dtype=np.uint8)

while True:
    retval, img = vc.read(capture_buffer)

magnets 21-12-2013 12:43
Re: Fixing opencv lag
 
Quote:
Originally Posted by sst.thad (Post 1314668)
The reason we are on a beaglebone instead of the dashboard is i have heard stories of FTA's requiring dashboards to get shut off, and we don't want to be stuck if that happens. So we want it to be onboard.
This.

In 2012, I had to tell a student that the vision software they worked on was not allowed because the FTA wouldn't let us use it. Then, we were stupid enough to try vision again in 2013, thinking that the "bandwidth limit" (which doesn't work) would prevent the FTA from banning the dashboard, but no, the FTA said we weren't allowed to use our vision stuff yet again, but this time, because a robot on the field had stopped working "because we sabotaged their network connection". What really happened? The other teams battery fell out. Not our problem, but we still couldn't use vision. Vision is not a guaranteed ability for robots. Nor are dashboards, or any amount of communication.

billbo911 21-12-2013 13:42
Re: Fixing opencv lag
 
Quote:
Originally Posted by magnets (Post 1315682)
... Vision is not a guaranteed ability for robots. Nor are dashboards, or any amount of communication.
I think the crux of this thread points out the error in your statement.

Yes, it is really a shame when a feature that has been pushed by FIRST is taken away, for any reason, and that feature is part of the major design a team has undertaken. The point of this thread, on the other hand, is how to avoid this possibility entirely.

Here is why this thread addresses this. If the vision processing is done by a separate on-board system, Beaglebone in this case and PCDuino in ours, it removes the wireless network entirely. All acquisition of images, processing and communication of target information remains on the robot. Honestly, for much less than $200, any team can do this! It just takes time to learn how, and the information is already readily available.

Thad House 21-12-2013 14:39
Re: Fixing opencv lag
 
Quote:
Originally Posted by virtuald (Post 1315655)
Not true. If you have ffmpeg support enabled (which it is by default in most opencv builds), you can retrieve the mjpeg stream directly from the camera via FFMPEG. Something similar to the following (error checking omitted):

Code:
vc = cv2.VideoCapture()
vc.open('http://%s/mjpg/video.mjpg' % camera_host)

h = vc.get(cv2.cv.CV_CAP_PROP_FRAME_WIDTH)
w = vc.get(cv2.cv.CV_CAP_PROP_FRAME_HEIGHT)

capture_buffer = np.empty(shape=(h, w, 3), dtype=np.uint8)

while True:
    retval, img = vc.read(capture_buffer)
Yeah I know I can recieve the MJPEG stream like this, and that is what I am doing. The camera still pushes frames into the stream at 30 FPS. If the vc.read does not occur faster then 30 times a second, The buffer will start to backlog and lag. In fact your code was what I started from, and I was doing exactly that. If you put at 50ms delay at the end of your processing function you will see that it starts to lag.

When people were talking about requesting images, they are talking about not opening the stream, but just directly requesting a single image, and working with that, so the camera does not push a buffer. Thats the part that does not work natively in openCV.


All times are GMT -5. The time now is 22:39.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi