I’ll expand on this later (and maybe even some documentation…), but for now know that you can use mjpg-streamer with OpenCV to write simple little filter plugins that can process the image from the webcam, and change what is streamed out via HTTP. You can install the mjpg-streamer-cv or mjpg-streamer-py packages using the instructions on our github repo.
We used it on our robot today with a python script to do image processing and networktables operations (Lifecam 3000, 320x240, 15fps, 30 quality), and it seemed to be about 20% CPU usage. Not too shabby.
In theory, you could use this on a RPi or other platform too, as I’ve pushed the changes (plus some significant build system improvements) to mjpg-streamer upstream.
On a Raspberry Pi 2 with OpenCV 3.1.0 and Numpy 1.10, I get the error
ImportError: numpy.core.multiarray failed to import
Since Mjpg-streamer works fine with the input_uvc plugin and numpy.core.multiarray imports fine in python 2 and 3, I thought I might be using the wrong version of Numpy. So I tried it with Numpy 1.7, 1.9, and 1.10 and even tried a fresh install of OpenCV and Numpy on another SD card, but I still get the same error.
I only spent a few minutes playing with this tonight, but in the current build configuration I’m finding it devilishly difficult to tell it to use Python 2 if Python 3 is installed. You can use “ldd” to figure out which one it decided to link against though.
I also spent some time working on it last night, and we found that cmake was using python 2 as its python interpreter when building. After some troubleshooting, we managed to make it build cvfilter_py.so correctly by setting the PYTHON_EXECUTABLE to /usr/bin/python3 instead of /usr/bin/python. Although this seems to have fixed the issue with cmake not finding python to build cvfilter_py.so, it has not fixed the numpy import error when running cvfilter_py.so.
Running ldd on cvfilter_py.so shows that it has decided to link against libpython3.4m.so.1.0
I feel like I ran into this problem when I first started working on the plugin… looking at __multiarray_api.h, it appears the thing that triggers it is essentially a “import numpy.core.multiarray”, but it might be hiding the actual error message. Perhaps add something like this to line 85 of filter_py.cpp and see if a more revealing error message shows up:
if (PyImport_ImportModule("numpy.core.multiarray") == NULL)
PyErr_Print();
else
fprintf(stderr, "No error occurred!?
");
I added that code to line 85 and sure enough, a much more revealing error popped up. Now running mjpg-streamer gives the error:
Traceback (most recent call last):
File "/usr/local/lib/python3.4/dist-packages/numpy/__init__.py", line 180, in <module>
from . import add_newdocs
File "/usr/local/lib/python3.4/dist-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/usr/local/lib/python3.4/dist-packages/numpy/lib/__init__.py", line 8, in <module>
from .type_check import *
File "/usr/local/lib/python3.4/dist-packages/numpy/lib/type_check.py", line 11, in <module>
import numpy.core.numeric as _nx
File "/usr/local/lib/python3.4/dist-packages/numpy/core/__init__.py", line 14, in <module>
from . import multiarray
ImportError: /usr/local/lib/python3.4/dist-packages/numpy/core/multiarray.cpython-34m.so: undefined symbol: PyType_GenericNew
ImportError: numpy.core.multiarray failed to import
Error loading numpy!
After a lot of troubleshooting, we’re still stuck at this error. We were able to get it working on a laptop Arch Linux install, but still no luck on the Raspberry Pi 2 running Raspbian. Any ideas as to what that error might mean?
Did you install python by hand (compile + install) or via a package? If you installed by hand, maybe reinstall?
Running ldd on the numpy .so files might yield interesting results. Is it pointing to libraries the same place as running ldd on cvfilter_py.so?
Do you have multiple versions of numpy installed (eg, python2 and python3?). Maybe get rid of one of them.
It would be interesting to know the contents of sys.path when the numpy import occurs. You can do this by inserting this before the numpy insert fails:
I’ve been busy getting ready for competition, so I haven’t had much time to work on this. I tried building Python 3.5.1 from source and installing it, but still almost the same error, though this time slightly different. Now instead of PyType_GenericNew being the undefined symbol, PyFloat_Type is, leading me to think that Python might not be exporting symbols correctly as nightpool suggested. So I tried adding -Xlinker -export-dynamic" to the cmake build flags through the MAKE_CXX_FLAGS variable. It didn’t seem to have any effect, so I’m thinking I may have put it in the wrong cmake variable. Any idea what I might be doing wrong?
Bellow is a log of what I tried, hopefully it will help you!
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental/_build$ cmake .. -DPYTHON_EXECUTABLE=/usr/local/bin/python3.5 -DCMAKE_CXX_FLAGS="-Xlinker -export-dynamic"
-- The C compiler identification is GNU 4.9.2
-- The CXX compiler identification is GNU 4.9.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Looking for include file sys/inotify.h
-- Looking for include file sys/inotify.h - found
-- Found PythonLibs: /usr/local/lib/libpython3.5m.a (found version "3.5.1")
-- Found PythonInterp: /usr/local/bin/python3.5 (found version "3.5.1")
-- Found NUMPY: /usr/local/lib/python3.5/site-packages/numpy/core/include
-- Found components for NumPy
-- NUMPY_ROOT_DIR = /usr/local
-- NUMPY_INCLUDES = /usr/local/lib/python3.5/site-packages/numpy/core/include
-- NUMPY_LIBRARIES =
-- NUMPY_API_VERSION = 1.11.0
-- Found PkgConfig: /usr/bin/pkg-config (found version "0.28")
-- checking for module 'libgphoto2'
-- package 'libgphoto2' not found
-- Could NOT find GPHOTO2 (missing: GPHOTO2_LIBRARY GPHOTO2_INCLUDE_DIR)
-- Looking for include file linux/videodev2.h
-- Looking for include file linux/videodev2.h - found
-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Could NOT find SDL (missing: SDL_LIBRARY SDL_INCLUDE_DIR)
--
-- The following features have been enabled:
* PLUGIN_INPUT_FILE , File input plugin
* PLUGIN_INPUT_HTTP , HTTP input proxy plugin
* PLUGIN_INPUT_OPENCV , OpenCV input plugin
* PLUGIN_CVFILTER_CPP , OpenCV example filter
* PLUGIN_CVFILTER_PY , OpenCV python filter
* PLUGIN_INPUT_UVC , Video 4 Linux input plugin
* PLUGIN_OUTPUT_FILE , File output plugin
* PLUGIN_OUTPUT_HTTP , HTTP server output plugin
* PLUGIN_OUTPUT_RTSP , RTSP output plugin
* PLUGIN_OUTPUT_UDP , UDP output stream plugin
-- The following OPTIONAL packages have been found:
* OpenCV
* PythonLibs
* PythonInterp
* Numpy
* Threads
-- The following features have been disabled:
* WXP_COMPAT , Enable compatibility with WebcamXP
* PLUGIN_INPUT_RASPICAM , Raspberry Pi input camera plugin (unmet dependencies)
* PLUGIN_INPUT_PTP2 , PTP2 input plugin (unmet dependencies)
* ENABLE_HTTP_MANAGEMENT , Enable experimental HTTP management option
* PLUGIN_OUTPUT_VIEWER , SDL output viewer plugin (unmet dependencies)
-- The following OPTIONAL packages have not been found:
* Gphoto2
* SDL
-- Configuring done
-- Generating done
-- Build files have been written to: /home/pi/mjpg-streamer/mjpg-streamer-experimental/_build
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental/_build$ make
Scanning dependencies of target mjpg_streamer
5%] Building C object CMakeFiles/mjpg_streamer.dir/mjpg_streamer.c.o
10%] Building C object CMakeFiles/mjpg_streamer.dir/utils.c.o
Linking C executable mjpg_streamer
10%] Built target mjpg_streamer
Scanning dependencies of target input_file
15%] Building C object plugins/input_file/CMakeFiles/input_file.dir/input_file.c.o
Linking C shared library input_file.so
15%] Built target input_file
Scanning dependencies of target input_http
21%] Building C object plugins/input_http/CMakeFiles/input_http.dir/input_http.c.o
26%] Building C object plugins/input_http/CMakeFiles/input_http.dir/misc.c.o
31%] Building C object plugins/input_http/CMakeFiles/input_http.dir/mjpg-proxy.c.o
Linking C shared library input_http.so
31%] Built target input_http
Scanning dependencies of target input_opencv
36%] Building CXX object plugins/input_opencv/CMakeFiles/input_opencv.dir/input_opencv.cpp.o
Linking CXX shared library input_opencv.so
36%] Built target input_opencv
Scanning dependencies of target cvfilter_cpp
42%] Building CXX object plugins/input_opencv/filters/cvfilter_cpp/CMakeFiles/cvfilter_cpp.dir/filter_cpp.cpp.o
Linking CXX shared library cvfilter_cpp.so
42%] Built target cvfilter_cpp
Scanning dependencies of target cvfilter_py
47%] Building CXX object plugins/input_opencv/filters/cvfilter_py/CMakeFiles/cvfilter_py.dir/filter_py.cpp.o
52%] Building CXX object plugins/input_opencv/filters/cvfilter_py/CMakeFiles/cvfilter_py.dir/conversion.cpp.o
Linking CXX shared library cvfilter_py.so
52%] Built target cvfilter_py
Scanning dependencies of target input_uvc
57%] Building C object plugins/input_uvc/CMakeFiles/input_uvc.dir/dynctrl.c.o
63%] Building C object plugins/input_uvc/CMakeFiles/input_uvc.dir/input_uvc.c.o
68%] Building C object plugins/input_uvc/CMakeFiles/input_uvc.dir/jpeg_utils.c.o
73%] Building C object plugins/input_uvc/CMakeFiles/input_uvc.dir/v4l2uvc.c.o
Linking C shared library input_uvc.so
73%] Built target input_uvc
Scanning dependencies of target output_file
78%] Building C object plugins/output_file/CMakeFiles/output_file.dir/output_file.c.o
Linking C shared library output_file.so
78%] Built target output_file
Scanning dependencies of target output_http
84%] Building C object plugins/output_http/CMakeFiles/output_http.dir/httpd.c.o
89%] Building C object plugins/output_http/CMakeFiles/output_http.dir/output_http.c.o
Linking C shared library output_http.so
89%] Built target output_http
Scanning dependencies of target output_rtsp
94%] Building C object plugins/output_rtsp/CMakeFiles/output_rtsp.dir/output_rtsp.c.o
Linking C shared library output_rtsp.so
94%] Built target output_rtsp
Scanning dependencies of target output_udp
[100%] Building C object plugins/output_udp/CMakeFiles/output_udp.dir/output_udp.c.o
Linking C shared library output_udp.so
[100%] Built target output_udp
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental/_build$ cd ..
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental$ sudo make install
make -C _build install
make[1]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[2]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
10%] Built target mjpg_streamer
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
15%] Built target input_file
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
31%] Built target input_http
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
36%] Built target input_opencv
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
42%] Built target cvfilter_cpp
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
52%] Built target cvfilter_py
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
73%] Built target input_uvc
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
78%] Built target output_file
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
89%] Built target output_http
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
94%] Built target output_rtsp
make[3]: Entering directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
make[3]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
[100%] Built target output_udp
make[2]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
Install the project...
-- Install configuration: "Release"
-- Up-to-date: /usr/local/bin/mjpg_streamer
-- Installing: /usr/local/share/mjpg-streamer/www
-- Up-to-date: /usr/local/share/mjpg-streamer/www/rotateicons.png
-- Up-to-date: /usr/local/share/mjpg-streamer/www/bodybg.gif
-- Up-to-date: /usr/local/share/mjpg-streamer/www/stream_simple.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/cambozola.jar
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.ui.tabs.min.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/control.htm
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.ui.core.min.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/static_simple.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/favicon.ico
-- Up-to-date: /usr/local/share/mjpg-streamer/www/javascript_motiondetection.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/java.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/javascript_simple.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.rotate.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/spinbtn_updn.gif
-- Up-to-date: /usr/local/share/mjpg-streamer/www/java_control.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/java_simple.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/stream.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/static.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/videolan.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/LICENSE.txt
-- Up-to-date: /usr/local/share/mjpg-streamer/www/index.html
-- Up-to-date: /usr/local/share/mjpg-streamer/www/JQuerySpinBtn.css
-- Up-to-date: /usr/local/share/mjpg-streamer/www/functions.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/favicon.png
-- Up-to-date: /usr/local/share/mjpg-streamer/www/sidebarbg.gif
-- Up-to-date: /usr/local/share/mjpg-streamer/www/example.jpg
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.ui.widget.min.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/fix.css
-- Up-to-date: /usr/local/share/mjpg-streamer/www/jquery.ui.custom.css
-- Up-to-date: /usr/local/share/mjpg-streamer/www/JQuerySpinBtn.js
-- Up-to-date: /usr/local/share/mjpg-streamer/www/style.css
-- Up-to-date: /usr/local/share/mjpg-streamer/www/javascript.html
-- Up-to-date: /usr/local/lib/mjpg-streamer/input_file.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/input_http.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/input_opencv.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/cvfilter_cpp.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/cvfilter_py.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/input_uvc.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/output_file.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/output_http.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/output_rtsp.so
-- Up-to-date: /usr/local/lib/mjpg-streamer/output_udp.so
make[1]: Leaving directory '/home/pi/mjpg-streamer/mjpg-streamer-experimental/_build'
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental$ export LD_LIBRARY_PATH=.
pi@core2062pi:~/mjpg-streamer/mjpg-streamer-experimental$ sudo ./mjpg_streamer -i "input_opencv.so --filter cvfilter_py.so --fargs ~/mjpg-streamer/mjpg-streamer-experimental/plugins/input_opencv/filters/cvfilter_py/example_filter.py" -o "output_http.so -w ./www"
MJPG Streamer Version: svn rev: 2016.3.0
i: device........... : default
i: Desired Resolution: 640 x 480
i: filter........... : cvfilter_py.so
i: filter args ..... : ~/mjpg-streamer/mjpg-streamer-experimental/plugins/input_opencv/filters/cvfilter_py/example_filter.py
'/usr/local/lib/python35.zip', '/usr/local/lib/python3.5', '/usr/local/lib/python3.5/plat-linux', '/usr/local/lib/python3.5/lib-dynload', '/usr/local/lib/python3.5/site-packages']
Traceback (most recent call last):
File "/usr/local/lib/python3.5/site-packages/numpy/__init__.py", line 180, in <module>
from . import add_newdocs
File "/usr/local/lib/python3.5/site-packages/numpy/add_newdocs.py", line 13, in <module>
from numpy.lib import add_newdoc
File "/usr/local/lib/python3.5/site-packages/numpy/lib/__init__.py", line 3, in <module>
import math
ImportError: /usr/local/lib/python3.5/lib-dynload/math.cpython-35m-arm-linux-gnueabihf.so: undefined symbol: PyFloat_Type
ImportError: numpy.core.multiarray failed to import
Error loading numpy!
This was done on a Debian Minbian install on a Raspberry Pi 3 with Python 3.5.1, Numpy 1.11.0, and Open CV 3.1.0.
Did you ever use the exposure parameter (-ex)? We have the same webcam, but cannot get the stream with the correct exposure. It is definitely calling the opencv filter, because the image has the result of our processing (eg detected contours), but the exposure is stubbornly refusing to be set.
Yeah, OpenCV doesn’t set exposure properly. See this bug report (looks like a fix was merged, so maybe next year it’ll work!).
To set exposure correctly, we use the v4l-ctl executable. I thought I had documented it somewhere, but I can’t seem to find it. Anyways, we found it useful to be able to switch the exposure dynamically via NetworkTables, so I created an object to manage that.
You’ll have to search the forums, but I remember seeing that the lifecam-3000 only supports certain exposure setting values (like 5 and 10), and for all others it just does full exposure. We use 5 I think.