Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   C/C++ (http://www.chiefdelphi.com/forums/forumdisplay.php?f=183)
-   -   MJPG Streamer input from opencv in C++ (http://www.chiefdelphi.com/forums/showthread.php?t=153361)

jreneew2 10-01-2017 06:46

MJPG Streamer input from opencv in C++
 
Hello!

I am setting up a vision co-processor with a raspberry pi 3 and I am having a trouble getting image data from opencv to mjpg-streamer. The reason I want to do this is so I can see the output of the opencv processing right on the smart dashboard.

The specific issue is that there is no image showing up in the web server for mjpg-streamer. The page is continuously loading so I think its just not displaying the stream.

I can even watch the avi file if I run omxplayer on the file.

Here is my opencv code:
Code:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <cscore.h>
#include <networktables/NetworkTable.h>

void VisionThread() {

        cv::VideoCapture vidCap(0);
       
        cv::Mat frame;
        cv::Mat output;

        cv::namedWindow("Display Window", cv::WINDOW_AUTOSIZE);
       
        if(!vidCap.isOpened()) {
                std::cout << "Can't open video stream!" << "\n";
        }
        else {
                vidCap.set(CV_CAP_PROP_FRAME_WIDTH,640);
                vidCap.set(CV_CAP_PROP_FRAME_HEIGHT,480);
        }

        vidCap >> frame;
               
        cv::VideoWriter outStream("output/out.avi", CV_FOURCC('M','J','P','G'), 30, frame.size(), true);

        while(true) {
                vidCap >> frame;
                cv::circle(frame, cv::Point(320, 240), 5, cv::Scalar(255, 0, 0));
               
                if(outStream.isOpened()) {
                        outStream.write(frame);
                }
                else {
                        std::cout<<"Can't write to file!" << "\n";
                }
               
                cv::imshow("Display Window", frame);
                if(cv::waitKey(30) >= 0) break;
        }       
}

int main(int argc, const char** argv) {
       
        VisionThread();
       
    return 0;
}

Here is the command I am running to start mjpg streamer:
Code:

mjpg_streamer -i "input_file.so -f /home/pi/vision_code/output"  -o "output_http.so -w /usr/local/www"
Any help is appreciated!

Thanks,
Drew

Wishbonea 10-01-2017 10:08

Re: MJPG Streamer input from opencv in C++
 
You should look at this thread on raspberrypi.org. It helped us alot last year when we had a similar issue with mjpg-streamer.
https://www.raspberrypi.org/forums/v...ic.php?t=45178

The way we ended up getting it working last year was by using this fork of mjpg-streamer. It has an opencv input plugin which should allow for lower latency streaming of your opencv output.
https://github.com/robotpy/mjpg-streamer

I haven't looked much into it, but the new Camera Server library made by WPILib seems very promising for streaming opencv.
https://github.com/wpilibsuite/cscore

Thad House 10-01-2017 15:59

Re: MJPG Streamer input from opencv in C++
 
Quote:

Originally Posted by jreneew2 (Post 1628422)
Hello!

I am setting up a vision co-processor with a raspberry pi 3 and I am having a trouble getting image data from opencv to mjpg-streamer. The reason I want to do this is so I can see the output of the opencv processing right on the smart dashboard.

The specific issue is that there is no image showing up in the web server for mjpg-streamer. The page is continuously loading so I think its just not displaying the stream.

I can even watch the avi file if I run omxplayer on the file.

Here is my opencv code:
Code:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <cscore.h>
#include <networktables/NetworkTable.h>

void VisionThread() {

        cv::VideoCapture vidCap(0);
       
        cv::Mat frame;
        cv::Mat output;

        cv::namedWindow("Display Window", cv::WINDOW_AUTOSIZE);
       
        if(!vidCap.isOpened()) {
                std::cout << "Can't open video stream!" << "\n";
        }
        else {
                vidCap.set(CV_CAP_PROP_FRAME_WIDTH,640);
                vidCap.set(CV_CAP_PROP_FRAME_HEIGHT,480);
        }

        vidCap >> frame;
               
        cv::VideoWriter outStream("output/out.avi", CV_FOURCC('M','J','P','G'), 30, frame.size(), true);

        while(true) {
                vidCap >> frame;
                cv::circle(frame, cv::Point(320, 240), 5, cv::Scalar(255, 0, 0));
               
                if(outStream.isOpened()) {
                        outStream.write(frame);
                }
                else {
                        std::cout<<"Can't write to file!" << "\n";
                }
               
                cv::imshow("Display Window", frame);
                if(cv::waitKey(30) >= 0) break;
        }       
}

int main(int argc, const char** argv) {
       
        VisionThread();
       
    return 0;
}

Here is the command I am running to start mjpg streamer:
Code:

mjpg_streamer -i "input_file.so -f /home/pi/vision_code/output"  -o "output_http.so -w /usr/local/www"
Any help is appreciated!

Thanks,
Drew

You should be using the cscore classes to handle the USB cameras and the opencv streams. Especially since you already have cscore linked.They are much more reliable then the stock opencv vidcapture objects. Take a look at the UsbCamera, CvSink, CvSource and MjpegServer classes in cscore. I can post an example how how to use them in code in a bit, but it's fairly simple. Create the UsbCamera and a CvSink, set the cvSink source to be the USB camera. Create a cvsource and an MjpegServer. Set the MjpegServer source to the cvsource. Then in your loop use CvSink grabframe to get the frame, and Cvsource putframe to stream your final opencv image

jreneew2 12-01-2017 16:39

Re: MJPG Streamer input from opencv in C++
 
Alright, I wrote up some new code and I am getting some linking errors around cscore. Here is my make file.

Here is the error Im getting
Code:

/tmp/ccVjljw2.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'

(And a bunch more like this)

Code:

# the compiler
CC = g++

#compiler flags:
CFLAGS = -std=c++0x `pkg-config --cflags --libs opencv` -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore 
TARGET = test

all: $(TARGET)

$(TARGET): $(TARGET).cpp
        $(CC) $(CFLAGS) -o $(TARGET) $(TARGET).cpp


clean:
        $(RM) (TARGET)
        $(RM) (CAMSERV)

Here is my code

Code:

#include <opencv2/core/core.hpp>
#include <opencv2/highgui/highgui.hpp>
#include <opencv2/imgproc.hpp>
#include <iostream>
#include <ntcore.h>
#include <networktables/NetworkTable.h>
#include <cscore.h>

void processImage() {
}

void VisionThread() {

        cs::UsbCamera camera = cs::UsbCamera("cam0", 0);
        cs::CvSink sink = cs::CvSink("sink");
        sink.SetSource(camera);
        cs::CvSource source = cs::CvSource("src", cs::VideoMode::PixelFormat::kMJPEG, 640, 480, 30);
        cs::MjpegServer mjpegServer = cs::MjpegServer("server", 1182);
        mjpegServer.SetSource(source);
       
        cv::Mat input;
        cv::Mat output;
       
        while(true) {
                sink.GrabFrame(input);
                cv::cvtColor(input, output, cv::COLOR_BGR2GRAY);
                source.PutFrame(output);
        }
}

int main(int argc, const char** argv) {
       
        VisionThread();
       
    return 0;
}

Thank you for the help!

- Drew

Thad House 12-01-2017 23:28

Re: MJPG Streamer input from opencv in C++
 
Quote:

Originally Posted by jreneew2 (Post 1630157)
Alright, I wrote up some new code and I am getting some linking errors around cscore. Here is my make file.

SNIP

Thank you for the help!

- Drew

Hmm that's interesting. Something I noticed is that you have -std=c++0x set. We build everything with -std=c++1y. That might explain that. Try changing to that. Also if you are interested, at the link afterward is all the compiler options we set. https://github.com/wpilibsuite/Visio...ins/arm.gradle

jreneew2 13-01-2017 10:13

Re: MJPG Streamer input from opencv in C++
 
Quote:

Originally Posted by Thad House (Post 1630337)
Hmm that's interesting. Something I noticed is that you have -std=c++0x set. We build everything with -std=c++1y. That might explain that. Try changing to that. Also if you are interested, at the link afterward is all the compiler options we set. https://github.com/wpilibsuite/Visio...ins/arm.gradle

That didn't seem to make a difference sadly. I also tried adding the arguments you used in your build and no difference. Here is the full output

Code:

g++ -std=c++1y `pkg-config --cflags --libs opencv` -Wformat=2 -Wall -Wextra -Werror -pedantic -Wno-psabi -Wno-unused-parameter -fPIC -rdynamic -Wno-error=deprecated-declarations -pthread -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore  -o test test.cpp
/tmp/ccEiQ61k.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSource::~VideoSource()':
test.cpp:(.text._ZN2cs11VideoSourceD2Ev[_ZN2cs11VideoSourceD5Ev]+0x40): undefined reference to `cs::ReleaseSource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::UsbCamera::UsbCamera(llvm::StringRef, int)':
test.cpp:(.text._ZN2cs9UsbCameraC2EN4llvm9StringRefEi[_ZN2cs9UsbCameraC5EN4llvm9StringRefEi]+0x3c): undefined reference to `cs::CreateUsbCameraDev(llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::CvSource(llvm::StringRef, cs::VideoMode::PixelFormat, int, int, int)':
test.cpp:(.text._ZN2cs8CvSourceC2EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii[_ZN2cs8CvSourceC5EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii]+0x5c): undefined reference to `cs::CreateCvSource(llvm::StringRef, cs::VideoMode const&, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::PutFrame(cv::Mat&)':
test.cpp:(.text._ZN2cs8CvSource8PutFrameERN2cv3MatE[_ZN2cs8CvSource8PutFrameERN2cv3MatE]+0x38): undefined reference to `cs::PutSourceFrame(int, cv::Mat&, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::~VideoSink()':
test.cpp:(.text._ZN2cs9VideoSinkD2Ev[_ZN2cs9VideoSinkD5Ev]+0x40): undefined reference to `cs::ReleaseSink(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::SetSource(cs::VideoSource)':
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x54): undefined reference to `cs::SetSinkSource(int, int, int*)'
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x7c): undefined reference to `cs::SetSinkSource(int, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::MjpegServer::MjpegServer(llvm::StringRef, llvm::StringRef, int)':
test.cpp:(.text._ZN2cs11MjpegServerC2EN4llvm9StringRefES2_i[_ZN2cs11MjpegServerC5EN4llvm9StringRefES2_i]+0x4c): undefined reference to `cs::CreateMjpegServer(llvm::StringRef, llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::CvSink(llvm::StringRef)':
test.cpp:(.text._ZN2cs6CvSinkC2EN4llvm9StringRefE[_ZN2cs6CvSinkC5EN4llvm9StringRefE]+0x30): undefined reference to `cs::CreateCvSink(llvm::StringRef, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::GrabFrame(cv::Mat&) const':
test.cpp:(.text._ZNK2cs6CvSink9GrabFrameERN2cv3MatE[_ZNK2cs6CvSink9GrabFrameERN2cv3MatE]+0x38): undefined reference to `cs::GrabSinkFrame(int, cv::Mat&, int*)'
collect2: error: ld returned 1 exit status
Makefile:11: recipe for target 'test' failed
make: *** [test] Error 1


Thad House 13-01-2017 12:47

Re: MJPG Streamer input from opencv in C++
 
Quote:

Originally Posted by jreneew2 (Post 1630472)
That didn't seem to make a difference sadly. I also tried adding the arguments you used in your build and no difference. Here is the full output

Code:

g++ -std=c++1y `pkg-config --cflags --libs opencv` -Wformat=2 -Wall -Wextra -Werror -pedantic -Wno-psabi -Wno-unused-parameter -fPIC -rdynamic -Wno-error=deprecated-declarations -pthread -I/home/pi/wpiutil/include -I/home/pi/ntcore/include -I/home/pi/cscore/include -L/home/pi/wpiutil/Linux/arm -L/home/pi/ntcore/arm/ntcore/build/libs/ntcore/static -L/home/pi/cscore/Linux/arm -lwpiutil -lntcore -lcscore  -o test test.cpp
/tmp/ccEiQ61k.o: In function `cs::VideoSource::VideoSource(cs::VideoSource const&)':
test.cpp:(.text._ZN2cs11VideoSourceC2ERKS0_[_ZN2cs11VideoSourceC5ERKS0_]+0x44): undefined reference to `cs::CopySource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSource::~VideoSource()':
test.cpp:(.text._ZN2cs11VideoSourceD2Ev[_ZN2cs11VideoSourceD5Ev]+0x40): undefined reference to `cs::ReleaseSource(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::UsbCamera::UsbCamera(llvm::StringRef, int)':
test.cpp:(.text._ZN2cs9UsbCameraC2EN4llvm9StringRefEi[_ZN2cs9UsbCameraC5EN4llvm9StringRefEi]+0x3c): undefined reference to `cs::CreateUsbCameraDev(llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::CvSource(llvm::StringRef, cs::VideoMode::PixelFormat, int, int, int)':
test.cpp:(.text._ZN2cs8CvSourceC2EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii[_ZN2cs8CvSourceC5EN4llvm9StringRefENS_9VideoMode11PixelFormatEiii]+0x5c): undefined reference to `cs::CreateCvSource(llvm::StringRef, cs::VideoMode const&, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSource::PutFrame(cv::Mat&)':
test.cpp:(.text._ZN2cs8CvSource8PutFrameERN2cv3MatE[_ZN2cs8CvSource8PutFrameERN2cv3MatE]+0x38): undefined reference to `cs::PutSourceFrame(int, cv::Mat&, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::~VideoSink()':
test.cpp:(.text._ZN2cs9VideoSinkD2Ev[_ZN2cs9VideoSinkD5Ev]+0x40): undefined reference to `cs::ReleaseSink(int, int*)'
/tmp/ccEiQ61k.o: In function `cs::VideoSink::SetSource(cs::VideoSource)':
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x54): undefined reference to `cs::SetSinkSource(int, int, int*)'
test.cpp:(.text._ZN2cs9VideoSink9SetSourceENS_11VideoSourceE[_ZN2cs9VideoSink9SetSourceENS_11VideoSourceE]+0x7c): undefined reference to `cs::SetSinkSource(int, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::MjpegServer::MjpegServer(llvm::StringRef, llvm::StringRef, int)':
test.cpp:(.text._ZN2cs11MjpegServerC2EN4llvm9StringRefES2_i[_ZN2cs11MjpegServerC5EN4llvm9StringRefES2_i]+0x4c): undefined reference to `cs::CreateMjpegServer(llvm::StringRef, llvm::StringRef, int, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::CvSink(llvm::StringRef)':
test.cpp:(.text._ZN2cs6CvSinkC2EN4llvm9StringRefE[_ZN2cs6CvSinkC5EN4llvm9StringRefE]+0x30): undefined reference to `cs::CreateCvSink(llvm::StringRef, int*)'
/tmp/ccEiQ61k.o: In function `cs::CvSink::GrabFrame(cv::Mat&) const':
test.cpp:(.text._ZNK2cs6CvSink9GrabFrameERN2cv3MatE[_ZNK2cs6CvSink9GrabFrameERN2cv3MatE]+0x38): undefined reference to `cs::GrabSinkFrame(int, cv::Mat&, int*)'
collect2: error: ld returned 1 exit status
Makefile:11: recipe for target 'test' failed
make: *** [test] Error 1


Hmm that sounds like the link to cscore still isn't being setup right. Try instead of -L/home/pi/cscore/Linux/arm and -lcscore, try -L/home/pi/cscore/Linux/arm/libcscore.a

That should force the link. Sorry this is a lot of toubleshooting. I've only really tested the gradle setup, and unless you specifically needing to custom compile either ntcore or cscore, I'd recommend using that, since that will make sure to have all the linker settings right. If you want to try that, clone the repo to that link I sent for the build arguments, and use the Cpp folder. Basic setup is go into build.gradle, comment out ext.builtType = "windows", and uncomment the 3 lines below the comment that says uncomment these to build natively on a Pi. Then just replace the main.cpp in the src file with your main, and that should work. Your file works when testing on my end, so it's definitely something not being right in the make link script.

jreneew2 13-01-2017 14:26

Re: MJPG Streamer input from opencv in C++
 
Quote:

Originally Posted by Thad House (Post 1630547)
Hmm that sounds like the link to cscore still isn't being setup right. Try instead of -L/home/pi/cscore/Linux/arm and -lcscore, try -L/home/pi/cscore/Linux/arm/libcscore.a

That should force the link. Sorry this is a lot of toubleshooting. I've only really tested the gradle setup, and unless you specifically needing to custom compile either ntcore or cscore, I'd recommend using that, since that will make sure to have all the linker settings right. If you want to try that, clone the repo to that link I sent for the build arguments, and use the Cpp folder. Basic setup is go into build.gradle, comment out ext.builtType = "windows", and uncomment the 3 lines below the comment that says uncomment these to build natively on a Pi. Then just replace the main.cpp in the src file with your main, and that should work. Your file works when testing on my end, so it's definitely something not being right in the make link script.

Its now working. I can see the output of the opencv stream right on my driverstation. I did what you said above and it worked flawlessly. Thank you for all the help and debugging. The framerate is actually wonderful. Almost no lag.

- Drew

"You must spread some Reputation around before giving it to Thad House again."


All times are GMT -5. The time now is 13:31.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi