MJPG Streamer input from opencv in C++

[jreneew2]

Hello!

I am setting up a vision co-processor with a raspberry pi 3 and I am having a trouble getting image data from opencv to mjpg-streamer. The reason I want to do this is so I can see the output of the opencv processing right on the smart dashboard.

The specific issue is that there is no image showing up in the web server for mjpg-streamer. The page is continuously loading so I think its just not displaying the stream.

I can even watch the avi file if I run omxplayer on the file.

Here is my opencv code:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <cscore.h>
#include <networktables/NetworkTable.h>

void VisionThread() {

	cv::VideoCapture vidCap(0);
	
	cv::Mat frame;
	cv::Mat output;

	cv::namedWindow("Display Window", cv::WINDOW_AUTOSIZE);
	
	if(!vidCap.isOpened()) {
		std::cout << "Can't open video stream!" << "\n";
	}
	else {
		vidCap.set(CV_CAP_PROP_FRAME_WIDTH,640);
		vidCap.set(CV_CAP_PROP_FRAME_HEIGHT,480);
	}

	vidCap >> frame; 
		
	cv::VideoWriter outStream("output/out.avi", CV_FOURCC('M','J','P','G'), 30, frame.size(), true);

	while(true) {
		vidCap >> frame;
		cv::circle(frame, cv::Point(320, 240), 5, cv::Scalar(255, 0, 0));
		
		if(outStream.isOpened()) {
			outStream.write(frame);
		} 
		else {
			std::cout<<"Can't write to file!" << "\n";
		}
		
		cv::imshow("Display Window", frame);
		if(cv::waitKey(30) >= 0) break;
	}	
}

int main(int argc, const char** argv) {
	
	VisionThread();
	
    return 0;
}

Here is the command I am running to start mjpg streamer:

mjpg_streamer -i "input_file.so -f /home/pi/vision_code/output"  -o "output_http.so -w /usr/local/www"

Any help is appreciated!

Thanks,
Drew

[Wishbonea]

You should look at this thread on raspberrypi.org. It helped us alot last year when we had a similar issue with mjpg-streamer.
https://www.raspberrypi.org/forums/v...ic.php?t=45178

The way we ended up getting it working last year was by using this fork of mjpg-streamer. It has an opencv input plugin which should allow for lower latency streaming of your opencv output.
https://github.com/robotpy/mjpg-streamer

I haven't looked much into it, but the new Camera Server library made by WPILib seems very promising for streaming opencv.
https://github.com/wpilibsuite/cscore

[Thad House]

Originally Posted by jreneew2

Hello!

I am setting up a vision co-processor with a raspberry pi 3 and I am having a trouble getting image data from opencv to mjpg-streamer. The reason I want to do this is so I can see the output of the opencv processing right on the smart dashboard.

The specific issue is that there is no image showing up in the web server for mjpg-streamer. The page is continuously loading so I think its just not displaying the stream.

I can even watch the avi file if I run omxplayer on the file.

Here is my opencv code:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <cscore.h>
#include <networktables/NetworkTable.h>

void VisionThread() {

	cv::VideoCapture vidCap(0);
	
	cv::Mat frame;
	cv::Mat output;

	cv::namedWindow("Display Window", cv::WINDOW_AUTOSIZE);
	
	if(!vidCap.isOpened()) {
		std::cout << "Can't open video stream!" << "\n";
	}
	else {
		vidCap.set(CV_CAP_PROP_FRAME_WIDTH,640);
		vidCap.set(CV_CAP_PROP_FRAME_HEIGHT,480);
	}

	vidCap >> frame; 
		
	cv::VideoWriter outStream("output/out.avi", CV_FOURCC('M','J','P','G'), 30, frame.size(), true);

	while(true) {
		vidCap >> frame;
		cv::circle(frame, cv::Point(320, 240), 5, cv::Scalar(255, 0, 0));
		
		if(outStream.isOpened()) {
			outStream.write(frame);
		} 
		else {
			std::cout<<"Can't write to file!" << "\n";
		}
		
		cv::imshow("Display Window", frame);
		if(cv::waitKey(30) >= 0) break;
	}	
}

int main(int argc, const char** argv) {
	
	VisionThread();
	
    return 0;
}

Here is the command I am running to start mjpg streamer:

mjpg_streamer -i "input_file.so -f /home/pi/vision_code/output"  -o "output_http.so -w /usr/local/www"

Any help is appreciated!

Thanks,
Drew

You should be using the cscore classes to handle the USB cameras and the opencv streams. Especially since you already have cscore linked.They are much more reliable then the stock opencv vidcapture objects. Take a look at the UsbCamera, CvSink, CvSource and MjpegServer classes in cscore. I can post an example how how to use them in code in a bit, but it's fairly simple. Create the UsbCamera and a CvSink, set the cvSink source to be the USB camera. Create a cvsource and an MjpegServer. Set the MjpegServer source to the cvsource. Then in your loop use CvSink grabframe to get the frame, and Cvsource putframe to stream your final opencv image

[jreneew2]

Alright, I wrote up some new code and I am getting some linking errors around cscore. Here is my make file.

Here is the error Im getting

/tmp/ccVjljw2.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'

(And a bunch more like this)

# the compiler
CC = g++

#compiler flags:
CFLAGS = -std=c++0x `pkg-config --cflags --libs opencv` -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore  
TARGET = test

all: $(TARGET)

$(TARGET): $(TARGET).cpp
	$(CC) $(CFLAGS) -o $(TARGET) $(TARGET).cpp


clean:
	$(RM) (TARGET)
	$(RM) (CAMSERV)

Here is my code

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <networktables/NetworkTable.h>
#include <cscore.h>

void processImage() {
}

void VisionThread() {

	cs::UsbCamera camera = cs::UsbCamera("cam0", 0);
	cs::CvSink sink = cs::CvSink("sink");
	sink.SetSource(camera);
	cs::CvSource source = cs::CvSource("src", cs::VideoMode::PixelFormat::kMJPEG, 640, 480, 30);
	cs::MjpegServer mjpegServer = cs::MjpegServer("server", 1182);
	mjpegServer.SetSource(source);
	
	cv::Mat input;
	cv::Mat output;
	
	while(true) {
		sink.GrabFrame(input);
		cv::cvtColor(input, output, cv::COLOR_BGR2GRAY);
		source.PutFrame(output);
	}
}

int main(int argc, const char** argv) {
	
	VisionThread();
	
    return 0;
}

Thank you for the help!

- Drew

[Thad House]

Originally Posted by jreneew2

Alright, I wrote up some new code and I am getting some linking errors around cscore. Here is my make file.

SNIP

Thank you for the help!

- Drew

Hmm that's interesting. Something I noticed is that you have -std=c++0x set. We build everything with -std=c++1y. That might explain that. Try changing to that. Also if you are interested, at the link afterward is all the compiler options we set. https://github.com/wpilibsuite/Visio...ins/arm.gradle

[jreneew2]

Originally Posted by Thad House

Hmm that's interesting. Something I noticed is that you have -std=c++0x set. We build everything with -std=c++1y. That might explain that. Try changing to that. Also if you are interested, at the link afterward is all the compiler options we set. https://github.com/wpilibsuite/Visio...ins/arm.gradle

That didn't seem to make a difference sadly. I also tried adding the arguments you used in your build and no difference. Here is the full output

g++ -std=c++1y `pkg-config --cflags --libs opencv` -Wformat=2 -Wall -Wextra -Werror -pedantic -Wno-psabi -Wno-unused-parameter -fPIC -rdynamic -Wno-error=deprecated-declarations -pthread -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore   -o test test.cpp
/tmp/ccEiQ61k.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSource::~VideoSource()':
test.cpp:(.text._ZN2cs11VideoSourceD2Ev[_ZN2cs11VideoSourceD5Ev]+0x40): undefined reference to `cs::ReleaseSource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::UsbCamera::UsbCamera(llvm::StringRef, int)':
test.cpp:(.text._ZN2cs9UsbCameraC2EN4llvm9StringRefEi[_ZN2cs9UsbCameraC5EN4llvm9StringRefEi]+0x3c): undefined reference to `cs::CreateUsbCameraDev(llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::CvSource(llvm::StringRef, cs::VideoMode::PixelFormat, int, int, int)':
test.cpp:(.text._ZN2cs8CvSourceC2EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii[_ZN2cs8CvSourceC5EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii]+0x5c): undefined reference to `cs::CreateCvSource(llvm::StringRef, cs::VideoMode const&, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::PutFrame(cv::Mat&)':
test.cpp:(.text._ZN2cs8CvSource8PutFrameERN2cv3MatE[_ZN2cs8CvSource8PutFrameERN2cv3MatE]+0x38): undefined reference to `cs::PutSourceFrame(int, cv::Mat&, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::~VideoSink()':
test.cpp:(.text._ZN2cs9VideoSinkD2Ev[_ZN2cs9VideoSinkD5Ev]+0x40): undefined reference to `cs::ReleaseSink(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::SetSource(cs::VideoSource)':
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x54): undefined reference to `cs::SetSinkSource(int, int, int*)'
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x7c): undefined reference to `cs::SetSinkSource(int, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::MjpegServer::MjpegServer(llvm::StringRef, llvm::StringRef, int)':
test.cpp:(.text._ZN2cs11MjpegServerC2EN4llvm9StringRefES2_i[_ZN2cs11MjpegServerC5EN4llvm9StringRefES2_i]+0x4c): undefined reference to `cs::CreateMjpegServer(llvm::StringRef, llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::CvSink(llvm::StringRef)':
test.cpp:(.text._ZN2cs6CvSinkC2EN4llvm9StringRefE[_ZN2cs6CvSinkC5EN4llvm9StringRefE]+0x30): undefined reference to `cs::CreateCvSink(llvm::StringRef, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::GrabFrame(cv::Mat&) const':
test.cpp:(.text._ZNK2cs6CvSink9GrabFrameERN2cv3MatE[_ZNK2cs6CvSink9GrabFrameERN2cv3MatE]+0x38): undefined reference to `cs::GrabSinkFrame(int, cv::Mat&, int*)'
collect2: error: ld returned 1 exit status
Makefile:11: recipe for target 'test' failed
make: *** [test] Error 1

[Thad House]

Originally Posted by jreneew2

That didn't seem to make a difference sadly. I also tried adding the arguments you used in your build and no difference. Here is the full output

g++ -std=c++1y `pkg-config --cflags --libs opencv` -Wformat=2 -Wall -Wextra -Werror -pedantic -Wno-psabi -Wno-unused-parameter -fPIC -rdynamic -Wno-error=deprecated-declarations -pthread -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore   -o test test.cpp
/tmp/ccEiQ61k.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSource::~VideoSource()':
test.cpp:(.text._ZN2cs11VideoSourceD2Ev[_ZN2cs11VideoSourceD5Ev]+0x40): undefined reference to `cs::ReleaseSource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::UsbCamera::UsbCamera(llvm::StringRef, int)':
test.cpp:(.text._ZN2cs9UsbCameraC2EN4llvm9StringRefEi[_ZN2cs9UsbCameraC5EN4llvm9StringRefEi]+0x3c): undefined reference to `cs::CreateUsbCameraDev(llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::CvSource(llvm::StringRef, cs::VideoMode::PixelFormat, int, int, int)':
test.cpp:(.text._ZN2cs8CvSourceC2EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii[_ZN2cs8CvSourceC5EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii]+0x5c): undefined reference to `cs::CreateCvSource(llvm::StringRef, cs::VideoMode const&, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::PutFrame(cv::Mat&)':
test.cpp:(.text._ZN2cs8CvSource8PutFrameERN2cv3MatE[_ZN2cs8CvSource8PutFrameERN2cv3MatE]+0x38): undefined reference to `cs::PutSourceFrame(int, cv::Mat&, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::~VideoSink()':
test.cpp:(.text._ZN2cs9VideoSinkD2Ev[_ZN2cs9VideoSinkD5Ev]+0x40): undefined reference to `cs::ReleaseSink(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::SetSource(cs::VideoSource)':
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x54): undefined reference to `cs::SetSinkSource(int, int, int*)'
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x7c): undefined reference to `cs::SetSinkSource(int, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::MjpegServer::MjpegServer(llvm::StringRef, llvm::StringRef, int)':
test.cpp:(.text._ZN2cs11MjpegServerC2EN4llvm9StringRefES2_i[_ZN2cs11MjpegServerC5EN4llvm9StringRefES2_i]+0x4c): undefined reference to `cs::CreateMjpegServer(llvm::StringRef, llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::CvSink(llvm::StringRef)':
test.cpp:(.text._ZN2cs6CvSinkC2EN4llvm9StringRefE[_ZN2cs6CvSinkC5EN4llvm9StringRefE]+0x30): undefined reference to `cs::CreateCvSink(llvm::StringRef, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::GrabFrame(cv::Mat&) const':
test.cpp:(.text._ZNK2cs6CvSink9GrabFrameERN2cv3MatE[_ZNK2cs6CvSink9GrabFrameERN2cv3MatE]+0x38): undefined reference to `cs::GrabSinkFrame(int, cv::Mat&, int*)'
collect2: error: ld returned 1 exit status
Makefile:11: recipe for target 'test' failed
make: *** [test] Error 1

Hmm that sounds like the link to cscore still isn't being setup right. Try instead of -L/home/pi/cscore/Linux/arm and -lcscore, try -L/home/pi/cscore/Linux/arm/libcscore.a

That should force the link. Sorry this is a lot of toubleshooting. I've only really tested the gradle setup, and unless you specifically needing to custom compile either ntcore or cscore, I'd recommend using that, since that will make sure to have all the linker settings right. If you want to try that, clone the repo to that link I sent for the build arguments, and use the Cpp folder. Basic setup is go into build.gradle, comment out ext.builtType = "windows", and uncomment the 3 lines below the comment that says uncomment these to build natively on a Pi. Then just replace the main.cpp in the src file with your main, and that should work. Your file works when testing on my end, so it's definitely something not being right in the make link script.

[jreneew2]

Originally Posted by Thad House

Hmm that sounds like the link to cscore still isn't being setup right. Try instead of -L/home/pi/cscore/Linux/arm and -lcscore, try -L/home/pi/cscore/Linux/arm/libcscore.a

That should force the link. Sorry this is a lot of toubleshooting. I've only really tested the gradle setup, and unless you specifically needing to custom compile either ntcore or cscore, I'd recommend using that, since that will make sure to have all the linker settings right. If you want to try that, clone the repo to that link I sent for the build arguments, and use the Cpp folder. Basic setup is go into build.gradle, comment out ext.builtType = "windows", and uncomment the 3 lines below the comment that says uncomment these to build natively on a Pi. Then just replace the main.cpp in the src file with your main, and that should work. Your file works when testing on my end, so it's definitely something not being right in the make link script.

Its now working. I can see the output of the opencv stream right on my driverstation. I did what you said above and it worked flawlessly. Thank you for all the help and debugging. The framerate is actually wonderful. Almost no lag.

- Drew

"You must spread some Reputation around before giving it to Thad House again."