mjpg-streamer now with OpenCV input plugin (+filtering)

[virtuald]

I'll expand on this later (and maybe even some documentation...), but for now know that you can use mjpg-streamer with OpenCV to write simple little filter plugins that can process the image from the webcam, and change what is streamed out via HTTP. You can install the mjpg-streamer-cv or mjpg-streamer-py packages using the instructions on our github repo.

We used it on our robot today with a python script to do image processing and networktables operations (Lifecam 3000, 320x240, 15fps, 30 quality), and it seemed to be about 20% CPU usage. Not too shabby.

In theory, you could use this on a RPi or other platform too, as I've pushed the changes (plus some significant build system improvements) to mjpg-streamer upstream.

[Wishbonea]

Thanks for making this! This is exactly what I have been trying to do for the past week!

The only issue is that when running Mjpg-Streamer with:

export LD_LIBRARY_PATH=.
./mjpg_streamer -i "input_opencv.so --filter cvfilter_py.so --fargs ~/mjpg-streamer/mjpg-streamer-experimental/plugins/input_opencv/filters/cvfilter_py/example_filter.py" -o "output_http.so -w ./www"

On a Raspberry Pi 2 with OpenCV 3.1.0 and Numpy 1.10, I get the error

ImportError: numpy.core.multiarray failed to import

Since Mjpg-streamer works fine with the input_uvc plugin and numpy.core.multiarray imports fine in python 2 and 3, I thought I might be using the wrong version of Numpy. So I tried it with Numpy 1.7, 1.9, and 1.10 and even tried a fresh install of OpenCV and Numpy on another SD card, but I still get the same error.

Is there something I may have missed?

[virtuald]

I added some rpath fixes that should remove the need to set the LD_LIBRARY_PATH anymore, that might be part of the problem.

Which version of python did the compilation select? I've only tested against Python 3 and NumPy 1.10.

[Wishbonea]

I don't have the Raspberry Pi with me right now, but now that I think of it, it may have selected python 2 instead of python 3.

Do you know how I would change which version of python it uses?

[virtuald]

I only spent a few minutes playing with this tonight, but in the current build configuration I'm finding it devilishly difficult to tell it to use Python 2 if Python 3 is installed. You can use "ldd" to figure out which one it decided to link against though.

[Wishbonea]

I also spent some time working on it last night, and we found that cmake was using python 2 as its python interpreter when building. After some troubleshooting, we managed to make it build cvfilter_py.so correctly by setting the PYTHON_EXECUTABLE to /usr/bin/python3 instead of /usr/bin/python. Although this seems to have fixed the issue with cmake not finding python to build cvfilter_py.so, it has not fixed the numpy import error when running cvfilter_py.so.

Running ldd on cvfilter_py.so shows that it has decided to link against libpython3.4m.so.1.0

[virtuald]

I feel like I ran into this problem when I first started working on the plugin... looking at __multiarray_api.h, it appears the thing that triggers it is essentially a "import numpy.core.multiarray", but it might be hiding the actual error message. Perhaps add something like this to line 85 of filter_py.cpp and see if a more revealing error message shows up:

if (PyImport_ImportModule("numpy.core.multiarray") == NULL)
    PyErr_Print();
else
    fprintf(stderr, "No error occurred!?\n");

[Wishbonea]

I added that code to line 85 and sure enough, a much more revealing error popped up. Now running mjpg-streamer gives the error:

Traceback (most recent call last):
  File "/usr/local/lib/python3.4/dist-packages/numpy/__init__.py", line 180, in <module>
    from . import add_newdocs
  File "/usr/local/lib/python3.4/dist-packages/numpy/add_newdocs.py", line 13, in <module>
    from numpy.lib import add_newdoc
  File "/usr/local/lib/python3.4/dist-packages/numpy/lib/__init__.py", line 8, in <module>
    from .type_check import *
  File "/usr/local/lib/python3.4/dist-packages/numpy/lib/type_check.py", line 11, in <module>
    import numpy.core.numeric as _nx
  File "/usr/local/lib/python3.4/dist-packages/numpy/core/__init__.py", line 14, in <module>
    from . import multiarray
ImportError: /usr/local/lib/python3.4/dist-packages/numpy/core/multiarray.cpython-34m.so: undefined symbol: PyType_GenericNew
ImportError: numpy.core.multiarray failed to import
Error loading numpy!

Thanks so much for your help!

[virtuald]

Excellent. It would be nice to fix up the build scripts more, but I think I've spent enough time messing with them. Feel free and issue a PR!

I've added that initial import to a pull request, should be easier to diagnose next time.

[Wishbonea]

After a lot of troubleshooting, we're still stuck at this error. We were able to get it working on a laptop Arch Linux install, but still no luck on the Raspberry Pi 2 running Raspbian. Any ideas as to what that error might mean?

[virtuald]

Did you install python by hand (compile + install) or via a package? If you installed by hand, maybe reinstall?

Running ldd on the numpy .so files might yield interesting results. Is it pointing to libraries the same place as running ldd on cvfilter_py.so?

Do you have multiple versions of numpy installed (eg, python2 and python3?). Maybe get rid of one of them.

It would be interesting to know the contents of sys.path when the numpy import occurs. You can do this by inserting this before the numpy insert fails:

PyRun_SimpleString("import sys; print(sys.path)");

[jrw]

Originally Posted by virtuald

We used it on our robot today with a python script to do image processing and networktables operations (Lifecam 3000, 320x240, 15fps, 30 quality), and it seemed to be about 20% CPU usage. Not too shabby.

Did you ever use the exposure parameter (-ex)? We have the same webcam, but cannot get the stream with the correct exposure. It is definitely calling the opencv filter, because the image has the result of our processing (eg detected contours), but the exposure is stubbornly refusing to be set.

[virtuald]

Originally Posted by jrw

Did you ever use the exposure parameter (-ex)? We have the same webcam, but cannot get the stream with the correct exposure. It is definitely calling the opencv filter, because the image has the result of our processing (eg detected contours), but the exposure is stubbornly refusing to be set.

Yeah, OpenCV doesn't set exposure properly. See this bug report (looks like a fix was merged, so maybe next year it'll work!).

To set exposure correctly, we use the v4l-ctl executable. I thought I had documented it somewhere, but I can't seem to find it. Anyways, we found it useful to be able to switch the exposure dynamically via NetworkTables, so I created an object to manage that.

You'll have to search the forums, but I remember seeing that the lifecam-3000 only supports certain exposure setting values (like 5 and 10), and for all others it just does full exposure. We use 5 I think.