MJPG lag OpenCV

[yash101]

As I was demonstrating my program to my mentor (it was the first time he saw it running live off a camera), I was experiencing constant lag of around 10 seconds. This was with the M1013, at http://10.11.65.13/mjpg/video.mjpg. Our M1011 worked fine last time I tried it and this just won't budge. Any suggestions? Both cameras have the same stream settings.

I did eradicate the possibility that maybe the processing might be too slow, but I was getting no lag on my internal camera. I also tried regular images and there was no lag at 60FPS.

[AlexBrinister]

Are you using TCP or UDP?

Alex Brinister

[Greg McKaskle]

The lag is caused by your camera running faster than the downstream pieces. Over time the buffers in the camera, network, and in your code store more and more images that they have arrived, but not been processed, until it either reaches a steady state, or until the buffers start to overflow.

I'd suggest measuring how long it takes to process an image. If it is 25ms, then your camera can theoretically run at 40fps to max out the system. But in practice it is wise to leave a bit of a safety margin. It is common to shoot for 80% utilization. This means that instead of 1/.025, you would set the camera to .8/.025, or 32 fps. This will hopefully allow the processing to catch up and be able to wait just a bit for new frames to come in.

Be sure to measure the time of the processing with realistic content. Some image processing algorithm steps are a constant amount of work. For 1,000,000 pixels, they will always perform N computation. Many are dependent on what they find in those pixels. So some datasets will not take 25ms, but may take 30 or 40.

Greg McKaskle

[yash101]

I am calculating FPS already and my processing speed is 3 times the camera frame rate cap of 15fps. With every module disabled, I am updating the display at 60fps but receiving at 15. I am going to write some code to adjust the delays to lock onto a frame rate!

[yash101]

So I wrote an engine that exploits my FPS engine and attempts to lower and raise the delay between each iteration to match a specific variable that can be set by a single keystroke. After this season, I will release my code because I have all sorts of feature selections and those features we will never use that could be useful for teaching freshmen the importance of configuration files and the ability to reconfigure on the go.

Actually, let me post a snippet of code

PHP Code:

the variables I can change on the go:

bool readConfig() {
    msg("===Config File Read Started===");
    //create a new ifstream from IOS to read the file
    ifstream fin(confLocationToUse);
    //check if the file is open
    if (fin.is_open()) {
        //read the file to the variables
        //one variable is read per line
        fin >> trackTargets;
        fin >> trackBalls;
        fin >> trackOtherRobots;
        fin >> useNiftyInterface;
        fin >> calculateDistance;
        fin >> unmodifiedFeed;
        fin >> showFPS;
        fin >> onboard;
        fin >> captype;
        fin >> capdev;
        fin >> imageWidth;
        fin >> imageHeight;
        fin >> camFOV;
        fin >> pi;
        fin >> uri;
        fin >> staticTarget_width;
        fin >> staticTarget_height;
        fin >> staticTarget_distanceFromFloor;
        fin >> dynamicTarget_width;
        fin >> dynamicTarget_height;
        fin >> dynamicTarget_distanceFromFloor;
        fin >> arena_ballheight;
        fin >> arena_ballwidth;
        fin >> cam_distanceFromFront;
        fin >> cam_distanceFromFloor;
        fin >> redAlliance;
        fin >> target_hueslidermin;
        fin >> target_hueslidermax;
        fin >> target_satslidermin;
        fin >> target_satslidermax;
        fin >> target_valslidermin;
        fin >> target_valslidermax;
        fin >> target_accuracy;
        fin >> target_blurKernelSquare;
        fin >> target_minArea;
        fin >> target_maxArea;
        fin >> ball_hueslidermin;
        fin >> ball_hueslidermax;
        fin >> ball_satslidermin;
        fin >> ball_satslidermax;
        fin >> ball_valslidermin;
        fin >> ball_valslidermax;
        fin >> ball_accuracy;
        fin >> ball_blurKernelSquare;
        fin >> ball_minarea;
        fin >> ball_maxarea;
        fin >> ball_minsides;
        fin >> ball_maxsides;
        fin >> maxfframes;
        fin >> lockFPS;
        //close the file properly to save the changes and dismount the file, preventing corruption
        fin.close();
        //debug
        msg("Configuration Read Successfully!");
        //return a true for succeed
        return 1;
    } else {
        msg("Read Failure!");
        return 0;
    }
}


The feature selection:
void featureList() {
    //clear the screen
    nl(32);
    //list the available features and their statuses
    //draw a lovely interface
    msg("====================================");
    msg("===========Feature List=============");
    msg("Name           ||   ID   || Enabled ");
    msg("===============++========++=========");
    msg("Track Targets? ||    1   ||    " + bool2str(trackTargets));
    msg("------------------------------------");
    msg("Track Balls?   ||    2   ||    " + bool2str(trackBalls));
    msg("------------------------------------");
    msg("track Robots?  ||    3   ||    " + bool2str(trackOtherRobots));
    msg("------------------------------------");
    msg("calc Distance? ||    4   ||    " + bool2str(calculateDistance));
    msg("------------------------------------");
    msg("Interface?     ||    5   ||    " + bool2str(useNiftyInterface));
    msg("------------------------------------");
    msg("LiveFeed       ||    6   ||    " + bool2str(unmodifiedFeed));
    msg("------------------------------------");
    msg("saves directory||    7   ||    1");
    msg("------------------------------------");
    msg("Show FPS       ||    8   ||    " + bool2str(showFPS));
    msg("------------------------------------");
    msg("FPS Engine     ||    9   ||    " + bool2str(fpsEngine));
    msg("====================================");
    msg("0 = done; press the ID with [RETURN]");
    msg("         to toggle value!           ");
    msg("====================================");
    msg("==========Saves Directory===========");
    msg(image_uri);
    msg("====================================");
} 


I am overpowering my program to the point where no team (except those who dare) would ever use every single one of them. Hopefully when my code is released, it will be quite educational for EVERYONE. I even have detailed comments almost every other line of code!

I have some exports of what the program can do so far too

[Greg McKaskle]

But doesn't the attached image show that your code is taking close to 70ms to process, and therefore your camera shouldn't run more than 12 or 13 fps?

Greg McKaskle

[yash101]

That is partially true. That isn't the same environment. All my modules were disabled so I can get up to 30fps.

I have attached a screenshot of my program running no modules and processing with a locked framerate of 30fps and one with the fps locking engine module disabled.

the one with it enabled is at ~10fps. There is around 60fps when the engine is disabled.


With all modules disabled, the program will max out at 61FPS. I don't know why, but I think it might be because it is exceeding what my GPU can do!

[Greg McKaskle]

To summarize why I think you had 10 seconds of lag, it was because your code requested the camera send more images into the processing code than it was able to consume. This caused the images to sit in the buffer and age. You had 10 seconds of aged packets sitting in the buffer. If you change processing, you may need to change the camera rate.

Do you have enough info to solve your problem?

Greg McKaskle

[yash101]

The camera was maxxed out at 15 fps so that is probably not the problem!

[Greg McKaskle]

Can you explain the lag you were seeing? Do you know how to fix it?

If so, you can ignore the rest of my post.

I'm not sure what you mean by "maxxed out at 15fps". My fundamental question is still -- How many images was the camera acquiring and sending to the processing device, and how many images were being processed in a second?

I think you'll find that you were acquiring more images than you were processing. That leads to buffered images which we interpret as lag because the images being presented are old ones from many seconds ago. Do you have a different explanation for the lag?

Greg McKaskle

[billbo911]

I've been following this thread for a while with quite a bit of interest.
Although our hardware is different, and our code is not nearly as complex, we to see a small, less than a quarter of a second, amount of lag. It is consistent and doesn't change.

Our hardware is a PCDuino v1, running OpenCV 2.3.1 and processing images from a Microsoft LifeCam over USB.

The question I keep coming back to is, is the camera free running and I am just grabbing a frame as needed, or are we actually requesting each frame one at a time. Here is some pseudo code that shows our approach. (Yash, is this how you are acquiring images?)

height = 240
width = 320

capture=cv.CaptureFromCAM(-1) ##run from camera
cv.SetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_WIDTH, width)
cv.SetCaptureProperty(capture, cv.CV_CAP_PROP_FRAME_HEIGHT, height)

while True:
rawImage =cv.QueryFrame(capture)

Do all your processing......

cv.ShowImage('Camera', rawImage)
etc......

I would hope this either just grabs the latest frame from the camera, or requests just one frame. If it is allowing the processor to buffer all the unused frames, then we are SLOWING down the process considerably and need to take a better approach.

Currently we can acquire and process approx. 20 fps. Again, there appears to be about .25 sec. lag between acquisition and display of images.

[yash101]

There is a constant lag of 10 seconds. The 15fps is the max I allow the camera to send. Also, I actually think the program is already running at 15 fps. I have a timer and I just do a frequency calculation, so running at ~50ms lag, I get 60fps. I believe that the program is actually running at 15fps already. I doubt the problem is with my processing because when I run from my internal camera with 4 times the resolution, I get very little lag.

Here is my retrieval code:

PHP Code:

CvCapture *cam1 = NULL; //null required on linux otherwise crash will occur
cam1 = cvCaptureFromFile("http://10.11.65.13/mjpg/video.mjpg");
while(1) {
    Mat rawImg = cvQueryFrame(cam1);
    if(trackTargets) {

    }
    if(trackBalls) {

    }
    if(trackRobots) {

    }
    imshow("Window", rawImg);
    waitKey(1);  //1 is lowest possible
} 


[Greg McKaskle]

The docs seem to say that cv.QueryFrame returns "the just grabbed frame". If you have only 0.25 seconds of lag from an external camera through an embedded card to a PC and to a display, I would think you are fine. To measure the lag more accurately, consider something that the embedded board can control and a measurement made on the video capture. This will allow you to turn on an LED and determine how long afterwards it takes for the board to see the LED. In the past, I've done this using some clamps to hold everything in place and done the simplest code that measure the delay from a pixel's green value go from below a threshold to above the threshold.

If you have ten second lag, that seems like a significant issue. Running the LED test will help determine whether this is possibly caused by the transmission to the display or caused by the capture.

Greg McKaskle

[billbo911]

Originally Posted by Greg McKaskle

The docs seem to say that cv.QueryFrame returns "the just grabbed frame". If you have only 0.25 seconds of lag from an external camera through an embedded card to a PC and to a display, I would think you are fine......

Greg McKaskle

Thanks for looking into that Greg. My network here at work does not allow me access to the docs.opencv.org site for some odd security reason.

That .25 sec. is actually from an external USB webcam, connected directly to the PCDuino, and using TCP to deliver the "X" center of the target to the cRio.
So, as you said, I think we're fine.

[yash101]

I figured it out. It is really chance. With minimal changes, I was getting .25s of lag. I think the future looks good especially if we can get an SBC!

I am using C++, by the way (and I suggest that everyone do so). It is not too frequent after your basic program is done, to make changes, so a configuration file will work. It takes 1 second to compile on our slowest computer so it isn't too bad and the speed increase might be worth it! I don't know if CvCapture has this lag problem when your process too slow. I noticed when my FPS dropped too low, I would suddenly start getting tons of lag.

It could also be a camera issue because I typically don't have this problem running our older, M1011 camera! As soon as I started to use the M1013 camera, bad things started to happen!