Poor Framerate with PSEYE Camera and OpenCV

I have been trying to get a decent framerate from our new webcam for weeks now and I don’t understand why it seems to be artificially limited to 50 fps. Running the command
gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw, width=640, height=480, framerate=60/1 ! fpsdisplaysink I can clearly see that the camera is capable of 60 fps. If I check in v4l2-ctl --all I can also see that

Video input : 0 (ov534: ok)
Format Video Capture:
	Width/Height      : 640/480
	Pixel Format      : 'YUYV'
	Field             : None
	Bytes per Line    : 1280
	Size Image        : 614400
	Colorspace        : sRGB
	Transfer Function : Default
	YCbCr Encoding    : Default
	Quantization      : Default
	Flags             : 
Streaming Parameters Video Capture:
	Capabilities     : timeperframe
	Frames per second: 60.000 (60/1)
	Read buffers     : 2

However, when using VideoCapture in OpenCV, I cannot seem to get over 10 fps

System.out.println("Hello Word!");
System.loadLibrary(Core.NATIVE_LIBRARY_NAME);
VideoCapture camera = new VideoCapture();
camera.open("gst-launch-1.0 v4l2src device=/dev/video1 ! video/x-raw, width=640, height=480, framerate=60/1  ! appsink");
int height = (int) camera.get(CAP_PROP_FRAME_HEIGHT);
int width = (int) camera.get(CAP_PROP_FRAME_WIDTH);
Mat frame = new Mat();

OUTER:
while(true) {
    long mils = System.currentTimeMillis();
    for(int i = 0; i < 10; i++) {
        boolean success = camera.read(frame);
        if (!success) {
            System.out.println("Can't read frame");
            break OUTER;
        }
    }
    long delta = System.currentTimeMillis() - mils;
    double framesPerSecond = 10.0 / delta * 1000;
    System.out.println("FPS: " + framesPerSecond);
}

Outputs

Hello Word!
FPS: 46.875
FPS: 51.85825410544512
FPS: 50.04170141784821
FPS: 49.3421052631579
FPS: 50.04170141784821
FPS: 50.0
FPS: 50.718512256973796
FPS: 50.04170141784821
FPS: 50.04170141784821
FPS: 50.04170141784821
FPS: 49.3421052631579

Does anyone have an explanation / solution to this?

Out of curiosity, why do you want a framerate higher than 50 fps? The main robot control loop runs at 50 Hz, after all.

I’m not sure what’s wrong with what you’re doing, but I just tested and cscore can get 60 fps from the PSEYE (code below). I suspect for some reason in your test the camera isn’t getting set into 60 fps mode by the v4l2_src driver, as there is a 50 fps mode too.

int main() {
  cs::UsbCamera camera{"usbcam", 1};
  camera.SetVideoMode(cs::VideoMode::kYUYV, 640, 480, 60);
  cs::CvSink cvsink{"cvsink"};
  cvsink.SetSource(camera);

  cv::Mat test;
  for (;;) {
    uint64_t start = wpi::Now();
    for (int i=0; i<10; ++i) {
      uint64_t time = cvsink.GrabFrame(test);
      if (time == 0) {
        std::cout << "error: " << cvsink.GetError() << std::endl;
        continue;
      }
    }
    uint64_t delta = wpi::Now() - start;
    std::cout << "FPS: " << (10.0 / delta * 1000000) << std::endl;
  }
}

Out of curiosity, what happens when you try a 320x240 mode at even higher FPS (150 or 187)? cscore will happily do even the highest FPS there.

This is literally the reciprocal of the “your eyes can see only 24fps” argument…

More frames = smoother image = better response times and less worry about dropped frames. Similar idea if you were to play an FPS Game, more frames = smoother image = better chance to react quickly to the enemy.

I should have been clearer. I am running this code on an NVidia Jetson TX1 so I don’t have the CSCore library (or any other WPILib libraries) on it. I simply installed OpenCV4Tegra on it and worked from there.

Alright so I think i’m making some headway on diagnosing this.
I have moved to some c++ code to cross out anything that could result from java slowdowns. My current code is

#include <stdio.h>
#include <opencv2/opencv.hpp>
using namespace cv;

double time_diff( const timespec &t1, const timespec &t2 ) {
  return ((t2.tv_sec-t1.tv_sec)*1000000000 + (t2.tv_nsec-t1.tv_nsec)) / 1000000000.0;
}

int main(int argc, char *argv[]) {
  VideoCapture cap("gst-launch-1.0 v4l2src device=/dev/video0 ! video/x-raw, width=640, height=480, framerate=60/1 ! appsink"); // open the default camera
  if(!cap.isOpened())  // check if we succeeded
    return -1;
  Mat frame;
  timespec t_start, t_end;
  for(int tests = 0; tests < 10; tests++) {
      clock_gettime(CLOCK_REALTIME, &t_start);
      for (int i = 1;i < 60; i++) {
          cap.read(frame);
      }
      clock_gettime(CLOCK_REALTIME, &t_end);
      std::cout << "FPS ~= " << 60.0 / time_diff(t_start, t_end) << std::endl;
  }
}

When I start the gstreamer pipeline with 60 fps set, I get wildly varying fps:

FPS ~= 8.82575
FPS ~= 7.10674
FPS ~= 11.0755
FPS ~= 17.0689

vs When I start the pipeline with 50 fps set:

FPS ~= 51.4685
FPS ~= 50.9493
FPS ~= 50.8228
FPS ~= 50.906

Also, when I set the resolution to 320x240 I can get a stable 187 fps

It’s a little off-topic but it’s fairly easy to build cscore and the rest of wpilib on a coprocessor–just clone the allwpilib github repo and build with cmake (not gradle).

Does the same thing happen if you use VideoCapture directly on /dev/video0 rather than using the gst pipeline?

If I use /dev/video0 instead of a gstreamer pipeline it behaves the same as when I use gstreamer and set the fps to 60.