Pi 3 - Slow cvsink.grabFrame() times

Greetings!

This year, our team got a Raspberry Pi 3 and a Logitech C922 camera for learning vision processing. We’ve got a pretty basic java program running thanks to the folks on here, but we’re seeing that the CVSink.getFrame() method is taking 3 times as long as each frame should be coming in for a 30-fps stream.

Here’s the Java code we’re using to get these results:


UsbCamera frontCamera = new UsbCamera("FrontCam", frontCamera.setResolution(640, 480);
frontCamera.setFPS(30);
...
CvSink imageSink = new CvSink("CV Image Grabber");
imageSink.setSource(frontCamera);
...
while (true) {
	long startTime = System.nanoTime();

	long frameTime = imageSink.grabFrame(inputImage);
	if (frameTime == 0) 
	{
		System.out.println(imageSink.getError());
		continue;
	}

	long elapsedTime = System.nanoTime() - startTime;
	System.out.println("grabFrame Time: " + elapsedTime/1000000 + "ms");
}

And the output we’re getting:


grabFrame Time: 98ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms
grabFrame Time: 99ms

Is this a limitation of the Pi 3’s USB camera driver, or could the grabFrame() method really be this slow? Would using something like Python or C++ be faster? Or is there a better way of doing this in Java?

Since ntcore 1.0.2 does not yet support USB cameras, I haven’t had a way to test this code with the camera hooked up to a Windows machine with the correct driver installed.

Any input / pointing us in the right direction is much appreciated! :smiley:

In general, to measure the amount of time something takes you should perform the action multiple times, then average the results. I would get the time, grab 100 frames, then divide the resulting ms by 100 to get average ms.

There’s really not much grabFrame is doing other than waiting for the camera to provide the image and decompressing it if necessary. The ~100 ms time makes me think the camera is only providing frames at 10 FPS. Can you run v4l2-ctl —list-formats-ext and paste the output here?

I’d run an average but I don’t have access to the Pi right now.

I ran this for about 3 minutes and watched the results of these prints; about 98% were 99ms, with a few jumps to 120ms or 60ms.

Unfortunately, I don’t have access to the Pi at the moment to run this command. Sorry! I’ll try to get access to it tomorrow to provide this information.

That being said, in the same program I *do *have an MjpegServer from this same camera set up *before *the CvSink declaration which is able to stream either 640x480 or 1280x720 at 30FPS without issue. I omitted this from the original code but here is the program with the Mjpeg added in:


// frontStream runs at 30FPS without issue accessed from a remote browser
MjpegServer frontStream = new MjpegServer("Front Camera", 1185);
UsbCamera frontCamera = new UsbCamera("FrontCam", 0);
frontCamera.setResolution(640, 480);
frontCamera.setFPS(30);
frontStream.setSource(frontCamera);
...
CvSink imageSink = new CvSink("CV Image Grabber");
imageSink.setSource(frontCamera);
...
// This loop averages ~100ms for each frame
while (true) {
	long startTime = System.nanoTime();

	long frameTime = imageSink.grabFrame(inputImage);
	if (frameTime == 0) 
	{
		System.out.println(imageSink.getError());
		continue;
	}

	long elapsedTime = System.nanoTime() - startTime;
	System.out.println("grabFrame Time: " + elapsedTime/1000000 + "ms");
}

Could the MjpegServer be causing the problem? If so, would running the MJPEG on a separate thread solve this issue or would this be a hardware limitation?

Thanks for the replies!

Are you sure the MJPEG stream is at 30 fps? How did you test that? Essentially the MJPEG server has nearly the same code as the code you’re running, and both wait on the same condition variable for the next frame coming from the camera. The only differences between the two are:

  • CvSink must decompress the image from JPEG
  • CvSink must copy the decompressed image into an OpenCV mat
  • Your code is running in Java

It feels unlikely that any of these steps will result in such deterministic behavior.

The MJPEG server is already running on a separate thread (the cscore library is internally multithreaded). I doubt the result will change if you comment out the MJPEG server, but it’s probably worth testing just to be sure.

It might also be informative to print out the frame time (the value returned from GrabFrame). This frame time is set by the capture thread immediately after the frame is received from the Linux driver layer. I would think the frame time deltas will match the ~100ms time you’re seeing.

Even if the camera is set up to run at 30 fps, it’s not always going to deliver frames at that rate. In particular, settings like auto exposure will result in varying frame rates based on the brightness of the image. Switching to manual exposure can help. However, how stable the numbers you’re getting are tend to make me thing this isn’t the case for you.

The other thing worth trying is setting the camera video mode to YUYV format instead of the default MJPEG (use SetVideoMode instead of SetResolution on the UsbCamera object to do this).

To double-check what video mode the camera is set in, you can run v4l2-ctl --all while your capture program is running.

Sorry, missed a “~”. I’m estimating based on the visible frame rate. There a jitters here and there, but the stream is smooth as butter for the most part.

I commented out the MjpegServer code and this resulted in about a 10ms increase in grabFrame speed. Progress!

Yes, frame time deltas returned from grabFrame() are consistent with the measured ~100ms.

I set the camera to YUYV using the following lines:

UsbCamera frontCamera = new UsbCamera("FrontCam", 0);
frontCamera.setVideoMode(VideoMode.PixelFormat.kYUYV, 640, 480, 30);

This brought the frame time deltas from grabFrame() down to an average of 68ms over a test of 100 frames! Getting there, but I’d really love to get this down much farther. Is it possible with Java? If not, would C++ or Python be faster?

Here’s the output I get from that command.

Thanks for all the help so far!

How did you compile OpenCV and cscore for the rPi3? What optimization settings did you use / packages did you install (e.g. OpenCV uses optimized routines for a number of operations if lapack / ATLAS is available during build). Debug vs release configuration can also make a difference.

We didn’t have to compile OpenCV ourselves. Should we have done this? Sorry, very new to this whole thing :smiley:

The program we’re running is a modified version of this:

In this, the dependencies.gradle file grabs the OpenCV 3.1.0 jar found at the following link:
http://first.wpi.edu/FRC/roborio/maven/release/org/opencv/opencv-jni/3.1.0/opencv-jni-3.1.0-linux-arm-raspbian.jar

After looking at your settings again, I noticed you have auto exposure and auto white balance turned on. These can have a dramatic effect on how fast the camera feeds you frames. Try adding these two lines to your code before entering the loop:


  frontCamera.setExposureManual(10);
  frontCamera.setWhiteBalanceManual(50);

In local testing with a HD3000 camera, if I don’t have these lines, I get frames about every 130 ms if I’m looking at a dark scene, and about every 30 ms if I’m looking at a brightly lit scene. With the above settings, I consistently get frames every 31-35 ms.

Thanks for that! I was able to get this down to 40 ms on my end with those settings. Adding in the actual image-processing steps (HSV, Erode, Dilate, find Contours, filter Contours), I was getting a total loop time of ~450 ms, which is kinda disappointing but it’s not a Jetson so I guess I can’t expect too much from the little guy.

I went ahead and re-installed the OS on the Pi, this time using Jessie Lite, installed Java, then ran this same code, which brought it down to ~430ms for the entire loop on average with no noticeable difference in grabFrame() times. Meh.

Can we go deeper?

I ran a heavily-watered down version of this same program with just the MJPEG server running and I noticed in the “supported video modes” area of the MJPEG page in a browser, the MJPEG format looks to support a higher frame rate and resolution then YUYV; up to 1280x720 and 60FPS, which is this camera’s max specs and the reason we purchased it!

I switched the camera’s video mode to MJPEG and changed the frame rate to 60 and…

VOILA!

The processing loop is now about ~110ms faster, sitting at ~320 ms, with grabFrame() only taking 10-12ms!

I don’t know much about the types of video compression or how they differ from one another, but it looks like MJPEG format with exposure and white balance manually set, as well as 640x480 at 60fps was the way to go for the C922.

I’m not sure how much switching to Jessie Lite may have helped in the end result, but I’m sure it’s much better for system resource usage since the only user program that will be running on the Pi is vision.

If you can point us to any useful resources about optimizing those processing times, it would be much appreciated! Otherwise we can consider this issue solved!

Thanks for all your help on this! I owe you a beer

One thing I’ve noticed when using both Logitech and Microsoft cameras is that poor lighting will strongly affect framerate if auto exposure is enabled (which it is, by default). On a Jetson TX1, it’s the difference between 30fps (Logitech c270’s max framerate) and 15fps. I’d strongly recommend using manual exposure to decrease read times if the room you’re testing in is dark.

We get about 47 to 50 fps fully processed with a RPi, and RPi Camera 2. We use a 640 x 480 image. You can’t use the cheap knockoffs CSI cameras. There is a setting for high speed mode on the Pi Camera. We run the camera grab in a thread and process in another using CV2. We are using python so Pyimagesearch.com was very helpful. He has a high frame rate tutorial.