Getting the Kinect to work with OpenCV and libfreenect/My experience with the Kinect

[yash101]

The Kinect is a nice sensor, which has an IR projector, an IR camera and an BGR camera. The IR projector and camera work together to for depth perception because as you go further away, the IR projector's IR dots will be much more scattered. The Kinect also has some great hardware, and is quite competitively priced for what it's got.

Today, when I was hacking around with the Kinect, in Robotics, I learned quite a bit of stuff, and I also got some areas where I need some help and assistance.

So for the software I used, I installed Ubuntu using Wubi (though you can do so any way). After that finished installing, I updated and upgraded the system, so it was running like a hungry lion! I installed some software with the following command:
sudo apt-get install qtcreator pkg-config libopencv-dev python-opencv libfreenect-dev libfreenect-demos chromium-browser flashplugin-installer
That installed a lot of stuff and took about half an hour on our school's wifi (very fast). By the way, skip the last two because that installs a browser and the adobe flash plugin for Firefox/chromium.

So I then blacklisted the Kinect, so it would be accessible to the system.

I was able to successfully run some sample programs, downloaded using apt-get:
freenect-glview

However, I am unable to compile any libfreenect source code! I am using QT Creator as the C++ IDE.

So, I ran pkg-config libfreenect --libs, and got the output I was looking for, so I was able to use it in the project file for QT Creator. However, even though I included the libfreenect.hpp, the which showed up as a valid file, I kept getting random errors every time I tried to compile code.

What did I do wrong, and how could I fix it? Does anyone have some sample code to interface the Kinect, libfreenect with OpenCV for the image processing? I just want to exploit the infrared camera that is in the Kinect! OpenCV can do the rest.

[Greg McKaskle]

Compilers shouldn't really give random errors. They are almost guaranteed to return errors which are hard for people to understand, and small changes in a #include or a build setting can lead to very different errors.

Perhaps this would be easier if you were to include some of those error messages.

Greg McKaskle

[yash101]

I'll try to take screenshots/videos/etc. to show what's going on ;)

[faust1706]

Shot in the dark: you didn't set up up the gcc as the c++ compiler.

http://ratchetrockers1706.org/vision-setup/

[yash101]

Shot in the dark: you didn't set up up the gcc as the c++ compiler.

http://ratchetrockers1706.org/vision-setup/

LOL. That might be it. However, why did OpenCV work, then? That's kinda wierd! Let me tweak those settings and report back! I need to write a whitepaper about this once I'm done, so rookies don't fall into the same hole as I!

[yash101]

OK. So I did change some compiler settings, and create a new project. However, I'm still unable to compile with libfreenect. I did get rid of all the errors that libfreenect gave me, until my compiler gave me 3000+ errors :ahh: :ahh: :ahh:!

I think it was because I was trying out some C code, and the setup was really a mix of both C and C++! Bad me :D

I need to now work on porting this IplImage and CvArr stuff to CvCapture and Mat!

[faust1706]

Do you have the namespaces cv and or std?

You can just put mat img = cvcapture() I think. I'm on my phone so I apologize if my capitalization isn't proper for functions.

Look into cvqueryframe. That is one I have used in the past and you can save the output as a mat.

[MoosingIn3space]

I have attached my sample code for my school's KinectDevice class, written in C++, which provides a simple interface to the Kinect. You simply write

// assuming 'capture' is a KinectDevice
cv::Mat frame;
capture >> frame;

in order to read a frame from the Kinect. Hope it helps.

[yash101]

This is the problem I am facing (below)

[faust1706]

Will you (attempt) to compile through terminal? It _should_ give a more detail error. More to work with.


make clean
make
./(executable name) will execute the program

In case you don't know of forgot ;)

[yash101]

Here's the output:
rm -f main.o
rm -f *~ core *.core
g++ -c -pipe -O2 -Wall -W -D_REENTRANT -DQT_WEBKIT -DQT_NO_DEBUG -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++ -I../TestOpenCVLibFreenect -I/usr/include/qt4/QtCore -I/usr/include/qt4 -I. -I../TestOpenCVLibFreenect -I. -o main.o ../TestOpenCVLibFreenect/main.cpp
g++ -Wl,-O1 -o TestOpenCVLibFreenect main.o -L/usr/lib/x86_64-linux-gnu `pkg-config opencv --libs` libfreenect --libs` -lQtCore -lpthread

Sadly, I don't speak robot (or do I?) so I don't really understand what the problem is!

[POSIXGuru]

Try changing `pkg-config opencv --libs` libfreenect --libs` to `pkg-config --libs opencv libfreenect`

Here's the output:
rm -f main.o
rm -f *~ core *.core
g++ -c -pipe -O2 -Wall -W -D_REENTRANT -DQT_WEBKIT -DQT_NO_DEBUG -DQT_CORE_LIB -DQT_SHARED -I/usr/share/qt4/mkspecs/linux-g++ -I../TestOpenCVLibFreenect -I/usr/include/qt4/QtCore -I/usr/include/qt4 -I. -I../TestOpenCVLibFreenect -I. -o main.o ../TestOpenCVLibFreenect/main.cpp
g++ -Wl,-O1 -o TestOpenCVLibFreenect main.o -L/usr/lib/x86_64-linux-gnu `pkg-config opencv --libs` libfreenect --libs` -lQtCore -lpthread

Sadly, I don't speak robot (or do I?) so I don't really understand what the problem is!

[yash101]

I'm still getting a same/similar error. I am starting to feel as though it is the library, libfreenect, that is doing this! OpenCV still works, and my code is so modular, you change a viariable, it uses an MGPEG stream or an internal camera! That's how I know that the code works!

[faust1706]

wait. Did you try running the program in terminal?

[yash101]

Ya. I ran it in Terminal and I think I pasted the output above and PMed it to you.

I'll try to fresh-install Ubuntu, and install OpenCV and libfreenect a better way. I guess I can say that the instructions on the getting started page are broken, so how do y'all suggest I install libfreenect? Any links to tutorials would be appreciated! Also, what version of Ubuntu should I target? The problem could be because I am not using the latest version of Ubuntu (I'm using what's available through Wubi!)

I'll also try to document the process and start working on my whitepaper!