OpenCV for RoboRio - Release

[NotInControl]

OpenCV For the RoboRio - Release

A few hours before kickoff and were finally able to release this. We have been working very hard to provide a working OpenCV
solution for everyone that not only supports the Axis IP camera we all have, but also the Microsoft HD 3000 USB webcam.

Our github page has everything to help you get starting using OpenCV, including our 2014 vision code which we used throughout
the 2014 season for hot target detection. The example supports reading images from a USB camera, Axis IP camera, or from
a file, by simply changing command line arguments

We have successfully ran this code on The RoboRio, and on a beaglebone black, so you can pick your poison. We also have this running
on the Tegra TK1, and will release a how-to on that very soon.

The OpenCV version we are providing is version 2.4.10 which we then patched to support the various camera settings of the
MS HD 3000 camera. It will also support other USB cameras although we havent tested it. And of course it supports the Axis
IP cameras.

The OpenCV libraries have been cross compiled using GCC 4.6.3 and has VFP_V3 and NEON support.

Some of the options it was compiled with:
TBB, FFMPEG, GSTREAMER, JAVA Bindings, JPEG, VL42, GTK

Java bindings means you can use this in Java as well, although we never tested it, so please report back if you do use it in Java.

Our github page has everything to help you get set up using C++ (Even though we program our Robot in Java, we program our Vision in C++).

Please let me know if you have any questions, we are here to help.

Link to Release: https://github.com/Team2168/2168_Vision_Example

The Program will take an input image like input.jpg (provided by the camera) and produce an output like that of output.jpg.

Regards,
Kevin

Happy New Year, and Happy New Season

And as always, if this is what you were waiting for, please provide you thoughts or any feedback you may have

[Bryce Paputa]

Looks very cool, we'll definitely try the java bindings out. How's the performance on the RoboRIO?

[NotInControl]

not too bad actually... we started doing more controlled vision performance tests a few weeks back, but then that got pushed behind some other priorities.

Our original tests were documented here: http://khengineering.github.io/RoboRio/vision/cameratest/

and still hold some valid data, however, the new openCV libraries we just produced are a bit more streamlined and should run a little bit faster.

It is too early to give a good recommendation on the Rio's performance, but we have been able to get 320x240 at 15fps without any lag using the axis camera under 2014 game scenarios, and that was more than what we needed for last years game.

Everything we have done so far shows that the IP camera outperforms the USB HD 3000 Camera on the Rio, but I don't want to official state that as of yet until we do more controlled observations. We should have a new set of comparative data by the end of week 1 for both IP and USB cameras.

The main page where that information will be held is at http://khengineering.github.io/RoboRio/faq/vision/

[Jared]

Awesome work. I've been planning to attempt to compile OpenCV on the roboRIO itself once we finally get one, but this seems like a better way.

[Jefferson]

Wow. Looks really well done and documented. We'll definitely be using this as a resource this season. I'm assuming there will be some vision challenge this year given the Microsoft camera beta testing.

Good work.

[virtuald]

Good to know it works! We've got numpy packaged for the RoboRIO in opkg format, so OpenCV will probably next on the list of things to package.

[mikhail]

Thanks Kevin! This is really valuable resource you have created.

While our whole team is brainstorming and prototyping I've been "secretly" toying around with the control system. What I've achieved so far is: I cloned the 2168_Vision_Example project from GitHub, successfully built it in Eclipse with the arm-frc-linux-gnueabi toolchain. Then transferred the libraries and the binary onto roboRIO and successfully executed it in there. All the how-to steps are clearly documented in the project's README.md file. Thank you very much, Kevin!

Next will be to modify the 2168's code until it breaks and then start prototyping something my own :D

[Wazzaps]

Any way to do some processing with opencv and then display the results in the dashboard? (Processed image stream)

PS. We are using java.

[Ben Wolsieffer]

I just tried this out today using the Java libraries and I got it to work.

After installing OpenCV to the roboRIO via the instructions on GitHub, I had to do some more work to make it work in Java.

First I had to add:
<var name="classpath" value="${classpath}:/path/to/opencv-2410.jar" />
to the end of the build.xml in the robot code Eclipse project (add it just before the closing </project> tag). This includes the OpenCV classes in the jar file that is uploaded to the roboRIO.

To load the native library in the robot code I had to add this line:
System.load("/usr/local/lib/lib_OpenCV/java/opencv_java2410.so");
Put this somewhere where it will get executed before any OpenCV code (ie. static initializer). The path will obviously be different if you installed OpenCV somewhere else on the roboRIO.

I was able to capture and stream an image from the Microsoft Lifecam HD3000, but it is at a very low resolution and I can't seem to change it. The same code works fine in Linux on my laptop. I'll see if I can figure more out tomorrow.

[Corvo]

I'm currently running two versions of openCV in eclipse, the official 2.4.10 version and this one, and I've installed the java version (for both) as descrbied here (http://docs.opencv.org/trunk/doc/tutorials/introduction/java_eclipse/java_eclipse.html). For some reason I'm getting an error (http://pastebin.com/C02ncsA6) with your version that I'm not getting with the 2.4.10 version from opencv.

It appears that this librabry only runs in 32 bit java and not 64. Is there any way to fix this without having to reinstall 32 bit java?

[NotInControl]

I just tried this out today using the Java libraries and I got it to work.

After installing OpenCV to the roboRIO via the instructions on GitHub, I had to do some more work to make it work in Java.

First I had to add:
<var name="classpath" value="${classpath}:/path/to/opencv-2410.jar" />
to the end of the build.xml in the robot code Eclipse project (add it just before the closing </project> tag). This includes the OpenCV classes in the jar file that is uploaded to the roboRIO.

To load the native library in the robot code I had to add this line:
System.load("/usr/local/lib/lib_OpenCV/java/opencv_java2410.so");
Put this somewhere where it will get executed before any OpenCV code (ie. static initializer). The path will obviously be different if you installed OpenCV somewhere else on the roboRIO.

I was able to capture and stream an image from the Microsoft Lifecam HD3000, but it is at a very low resolution and I can't seem to change it. The same code works fine in Linux on my laptop. I'll see if I can figure more out tomorrow.

This is awesome news. I haven't had the time to test out Java myself (it is 1 item on a long to-do list of mine, but glad to hear it is working for you.

If you are having trouble changing the camera settings, try using the patch we added to OpenCV, we modified some of the v4l calls and exposed a new constructor to the videoCap class.

You can work off of the C++ example here: which will allow you to set FPS, image size, and some quality settings.

https://github.com/Team2168/2168_Vision_Example/blob/master/src/main.cpp#L633-L664

Let me know if this works for you or not.

[NotInControl]

I'm currently running two versions of openCV in eclipse, the official 2.4.10 version and this one, and I've installed the java version (for both) as descrbied here (http://docs.opencv.org/trunk/doc/tutorials/introduction/java_eclipse/java_eclipse.html). For some reason I'm getting an error (http://pastebin.com/C02ncsA6) with your version that I'm not getting with the 2.4.10 version from opencv.

It appears that this librabry only runs in 32 bit java and not 64. Is there any way to fix this without having to reinstall 32 bit java?

Are you trying to run this on your desktop or on a Arm based processor?

The OpenCV libraries we provide are for Arm based processors only so they will only run on a RoboRio, Beaglebone etc or anything running ArmV7 soft eabi. They will not run on an Intel based processor.

Please provide more details of where you are trying to run the program (causing the error), and what you have done thus far getting up to that point so that we can try to help figure it out.

Thanks for trying this out.

[JagCode]

So, this happened (http://i.imgur.com/FiyVuRK.jpg). Not sure what went wrong. :I

We followed the instructions, and the Rio_Beagle directory was successfully created. However, the main.o that was inside was corrupted to kingdom-come.

[Ben Wolsieffer]

So, this happened (http://i.imgur.com/FiyVuRK.jpg). Not sure what went wrong. :I

We followed the instructions, and the Rio_Beagle directory was successfully created. However, the main.o that was inside was corrupted to kingdom-come.

I don't think main.o is necessarily corrupted, it is a binary file so you shouldn't be trying to open it with a text editor.

[kylelanman]

Team 2481 is trying to get this working with our C++ project. Our project settings are what was generated by the FRC C++ plugin.

I'll admit that we don't have a lot of expertise when it comes to modifying the build/compiler/linker configuration.

We followed the instructions in the readme. The sample project compiles but when when we try to add the libraries to our project we aren't able to compile. I'm including the build console and our .cproject file.

Any assistance would be greatly appreciated.

Thanks,

Team 2481

[NotInControl]

So, this happened (http://i.imgur.com/FiyVuRK.jpg). Not sure what went wrong. :I

We followed the instructions, and the Rio_Beagle directory was successfully created. However, the main.o that was inside was corrupted to kingdom-come.

Nothing went wrong actually. It looks like you successfully compiled the code. The two warnings you get are known warnings which everyone on windows will receive and do not affect code output.

Main.o is an object file and is not human readable... it is not corrupt.

The file named 2168_Vision in that folder is the binary that you can transfer to the arm7 embedded device and run it.

I dont see any obvious problems in your posted images.

If you have any probs, please repprt back.

Regards,
Kevin

[NotInControl]

Team 2481 is trying to get this working with our C++ project. Our project settings are what was generated by the FRC C++ plugin.

I'll admit that we don't have a lot of expertise when it comes to modifying the build/compiler/linker configuration.

We followed the instructions in the readme. The sample project compiles but when when we try to add the libraries to our project we aren't able to compile. I'm including the build console and our .cproject file.

Any assistance would be greatly appreciated.

Thanks,

Team 2481

The WPILib project is not set up properly. The linker is not directed to the OpenCv libraries supplied by the Sample project _Libraries folder. You will need to manually add that yourself if you are starting from a WPI template Robot project.

Take a look at the build settings in the Sample Project we supplied, and modify the WPI build settings to include the OpenCV specific changes, such as adding the OpenCV _Libraries path to your WPILib project, and adding the -rpath command to the linker misc settings.

The build settings in the Sample Project should be all that you need to get this up and running in your own project.

[kylelanman]

The WPILib project is not set up properly. The linker is not directed to the OpenCv libraries supplied by the Sample project _Libraries folder. You will need to manually add that yourself if you are starting from a WPI template Robot project.

Take a look at the build settings in the Sample Project we supplied, and modify the WPI build settings to include the OpenCV specific changes, such as adding the OpenCV _Libraries path to your WPILib project, and adding the -rpath command to the linker misc settings.

The build settings in the Sample Project should be all that you need to get this up and running in your own project.

We double checked all the settings again and did not find any problems. We did however find the problem and I'm nearly certain other people will run into this.

In the WPILib.h file on line 8 you will find the following.

#define REAL

In the core.hpp file on line 4132 you will find the following enum definition.

enum
{
NONE=0, //!< empty node
INT=1, //!< an integer
REAL=2, //!< floating-point number
FLOAT=REAL, //!< synonym or REAL
STR=3, //!< text string in UTF-8 encoding
STRING=STR, //!< synonym for STR
REF=4, //!< integer of size size_t. Typically used for storing complex dynamic structures where some elements reference the others
SEQ=5, //!< sequence
MAP=6, //!< mapping
TYPE_MASK=7,
FLOW=8, //!< compact representation of a sequence or mapping. Used only by YAML writer
USER=16, //!< a registered object (e.g. a matrix)
EMPTY=32, //!< empty structure (sequence or mapping)
NAMED=64 //!< the node has a name (i.e. it is element of a mapping)
};

The two definitions of REAL conflict. We removed the #define in WPILib.h and were able to compile.

The best I can tell the #define REAL in WPILib is only used in examples provided for the simulator to distinguish between simulation and real mode so it shouldn't cause a problem to remove it.

Thanks,

Team 2481

[Corvo]

Are you trying to run this on your desktop or on a Arm based processor?

The OpenCV libraries we provide are for Arm based processors only so they will only run on a RoboRio, Beaglebone etc or anything running ArmV7 soft eabi. They will not run on an Intel based processor.

Please provide more details of where you are trying to run the program (causing the error), and what you have done thus far getting up to that point so that we can try to help figure it out.

Thanks for trying this out.

Currently I'm running this on my laptop (desktop) but have plans to migrate over to an Odroid (http://www.hardkernel.com/main/products/prdt_info.php?g_code=G138745696275) (Arm processor) for competition.

As for what I've done thus far, I've already stated it, I've installed it like the official version and run some test code on it that works fine with the main version. I believe the version that's up for download just hasn't been properly packaged.

[mikhail]

Any way to do some processing with opencv and then display the results in the dashboard? (Processed image stream)

PS. We are using java.

I wonder the same but in C++. What would be the best way to do something similar to imshow() but on the driver station's dashboard?

[faust1706]

tl;dr of this thread (copied from http://www.reddit.com/r/FRC/comments/2t0dy4/how_to_compile_opencv_from_source_for_roborio_on/) Credit goes to u/ironmig


Install dependencies
sudo apt-get install build-essential
sudo apt-get install cmake
sudo apt-get install gcc-arm-linux-gnueabi
sudo apt-get install g++-arm-linux-gnueabi
Enter directory that will hold opencv
cd ~/Documents
Clone OpenCV from git
git clone https://github.com/Itseez/opencv.git
Checkout to a recent OpenCV release (change 2.4.10.1 to a recent release)
cd opencv
git checkout tags/2.4.10.1
Set toolchain to correct version of g++ (change line in file or run this complicated one-liner
cd platforms/linux
x=$(dpkg -s g++-arm-linux-gnueabi | grep "Version" | grep -o "..."); sed -i 's/set(GCC_COMPILER_VERSION "."/set(GCC_COMPILER_VERSION \"'$x'\" CACHE STRING "GCC Compiler version"/g' arm-gnueabi.toolchain.cmake
Prepare file for build
mkdir build_roborio
cd build_roborio
cmake -DSOFTFP=ON -DCMAKE_TOOLCHAIN_FILE=../arm-gnueabi.toolchain.cmake ../../..
Build project (this will take a while)
make
Disclaimer* I haven't actually tried this.

[sebas]

I have also confirmed that the Java version works if you make the fixes to the build.xml. Great work, Team 2168!

[Arhowk]

My apologies for not being used to linux development nor C++ (Windows + Java almost exclusively for the past 10 years) but the first time I tried to build, it gave me an error that it couldn't find libc++.so.6 (and another one that I don't remember). Now, when I try to build, I get this message

"Info: Nothing to build for 2168_Vision_Example"

I thought it might be because the builder uses a diff scanner and it's already built, but I tried to modify the source code and it still refused to build.

E/ deleting the Rio-Beagle folder re-gave me the first error

c:/frc/bin/../lib/gcc/arm-frc-linux-gnueabi/4.9.1/../../../../arm-frc-linux-gnueabi/bin/ld.exe: warning: libstdc++.so.6, needed by D:\Miscellaneous Documents\CDT Code\2168_Vision_Example\_Libraries\softFP\lib_Ope nCV/libopencv_highgui.so, not found (try using -rpath or -rpath-link)
c:/frc/bin/../lib/gcc/arm-frc-linux-gnueabi/4.9.1/../../../../arm-frc-linux-gnueabi/bin/ld.exe: warning: libffi.so.6, needed by D:/Miscellaneous Documents/CDT Code/2168_Vision_Example/_Libraries/softFP/lib_OpenCV/libgio-2.0.so.0, not found (try using -rpath or -rpath-link)

should I just ignore it?

[frechetn]

Has there been any developments with the Tegra TK1 as suggested in the original post?

[JohnM]

One thing I do not understand is the whole downloading process. Okays I have the library's set up on the robRIO (I think). Anyways here are my questions.


Will all of openCV's methods for C++ fork on this?
To download code to the robot you have to do the FTP tranfer of the binary? You can't just deploy the code to the robot


I ask the second question is because the provided example I eclipse does not provide an option to deploy to the roboRIO, only an option to run as a local application or something like that.

[NotInControl]

I have also confirmed that the Java version works if you make the fixes to the build.xml. Great work, Team 2168!

Awesome!. I will add that to the Github readme so others can follow it as well.

My apologies for not being used to linux development nor C++ (Windows + Java almost exclusively for the past 10 years) but the first time I tried to build, it gave me an error that it couldn't find libc++.so.6 (and another one that I don't remember). Now, when I try to build, I get this message

"Info: Nothing to build for 2168_Vision_Example"

I thought it might be because the builder uses a diff scanner and it's already built, but I tried to modify the source code and it still refused to build.

E/ deleting the Rio-Beagle folder re-gave me the first error

c:/frc/bin/../lib/gcc/arm-frc-linux-gnueabi/4.9.1/../../../../arm-frc-linux-gnueabi/bin/ld.exe: warning: libstdc++.so.6, needed by D:\Miscellaneous Documents\CDT Code\2168_Vision_Example\_Libraries\softFP\lib_Ope nCV/libopencv_highgui.so, not found (try using -rpath or -rpath-link)
c:/frc/bin/../lib/gcc/arm-frc-linux-gnueabi/4.9.1/../../../../arm-frc-linux-gnueabi/bin/ld.exe: warning: libffi.so.6, needed by D:/Miscellaneous Documents/CDT Code/2168_Vision_Example/_Libraries/softFP/lib_OpenCV/libgio-2.0.so.0, not found (try using -rpath or -rpath-link)

should I just ignore it?

Did you install the WPILib C++ tools? Can you build a WPI C++ project? The message nothing to build means that nothing in the source code changed from the last time you built the program and created the 2168_Vision binary.

Edit the source code, and it will build again.

Has there been any developments with the Tegra TK1 as suggested in the original post?

Yes there has, I need to upload the code. These libraries won't won't on the Tegra because the Tegra used hardFP eabi. I will upload them as soon as I can.

One thing I do not understand is the whole downloading process. Okays I have the library's set up on the robRIO (I think). Anyways here are my questions.


Will all of openCV's methods for C++ fork on this?
To download code to the robot you have to do the FTP tranfer of the binary? You can't just deploy the code to the robot


I ask the second question is because the provided example I eclipse does not provide an option to deploy to the roboRIO, only an option to run as a local application or something like that.

This is a full OpenCV package so all methods are included, we also added a few modules so you get more than just the basic OpenCV. We also patched OpenCV so it supports the Microsoft HD 3000 camera.

To download the code all you need to do is Transfer the binary using FTP, winSCP, or any other method you like. Follow the instructions on the readme to get this running as a standalone project.

This project is not apart of WPILib, it is standalone, which is why there is no deploy. It can be deployed to any Arm7 device, including a beagle bone without any changes.

[brk]

I wonder the same but in C++. What would be the best way to do something similar to imshow() but on the driver station's dashboard?


So I'm trying to do the same in C++.
I'd like to capture/process the stream on the roborio, but then send the processed image back to the driver station.

As far as I can tell, you need to get a JPEG image and send that back. The capture gives me a cv::Mat object back, I manipulate it, but then how to I send that cv::Mat data back to the driver station? Is it already in JPEG format? or do I need to convert it. And where do I access a JPEG stream within the cv::Mat ? I know, I know, this is a CV question, but this is the first time working with CV.


edit - found my answer...
Just need to convert the Mat to an array using cv::imencode() and send that back to the DS.

[DaPasta]

Greetings!

I attempted to get OpenCV working with Java. However, once I pushed the Java code to the roboRio, I get the following error on the driver station:
ERROR Unhandled exception: java.lang.UnsatisfiedLinkError: no opencv_java2410 in java.library.path at [java.lang.ClassLoader.loadLibrary(ClassLoader.java :1857), java.lang.Runtime.loadLibrary0(Runtime.java:870), java.lang.System.loadLibrary(System.java:1119), org.team1515.pastabot.Robot.robotInit(Robot.java:4 1), edu.wpi.first.wpilibj.IterativeRobot.startCompetit ion(IterativeRobot.java:76), edu.wpi.first.wpilibj.RobotBase.main(RobotBase.jav a:234)]

I also added the following line in the roboInit method per instructions based on a previous post:
System.load("/usr/local/lib/lib_OpenCV/java/opencv_java2410.so");

I double checked multiple times that this path was correct.

[Jared]

Greetings!

I attempted to get OpenCV working with Java. However, once I pushed the Java code to the roboRio, I get the following error on the driver station:
ERROR Unhandled exception: java.lang.UnsatisfiedLinkError: no opencv_java2410 in java.library.path at [java.lang.ClassLoader.loadLibrary(ClassLoader.java :1857), java.lang.Runtime.loadLibrary0(Runtime.java:870), java.lang.System.loadLibrary(System.java:1119), org.team1515.pastabot.Robot.robotInit(Robot.java:4 1), edu.wpi.first.wpilibj.IterativeRobot.startCompetit ion(IterativeRobot.java:76), edu.wpi.first.wpilibj.RobotBase.main(RobotBase.jav a:234)]

I also added the following line in the roboInit method per instructions based on a previous post:
System.load("/usr/local/lib/lib_OpenCV/java/opencv_java2410.so");

I double checked multiple times that this path was correct.

This may be due to the robotInit() method being called a short period of time after the robot code actually starts.

You can put the System.load(PATH); line inside of a static initialization block in the main robot class.

It should look something like this, but with CV_PATH replaced by the path you installed to. It can be inserted above your robotInit() method.

static {
System.load(CV_PATH);
}

[mikhail]

edit - found my answer...
Just need to convert the Mat to an array using cv::imencode() and send that back to the DS.

A small code snippet would be very appreciated. What parameters do you use for cv::imencode? Do you use CameraServer class to send the image to the DS? Thank you.

[brk]

To encode the JPEG, i have:

unsigned int SPAMVision::_encodeJpeg(std::vector<uchar> & dst, cv::Mat src, int quality)
{
std::vector<int> param = std::vector<int>(2);
param[0] = CV_IMWRITE_JPEG_QUALITY;
param[1] = quality;

cv::imencode(".jpg", src, dst, param);

return dst.size();
}


And I use this in something very similar to the CameraServer class.

[brk]

I have a new problem...

I'm trying to do this on a beaglebone black.
I thought it should work out of the box, but...

when I run an executable built in eclipse, I get:

error while loading shared libraries: libstdc++.so.6.0.20: cannot open shared object file: No such file or directory

ldd shows libstdc++.so.6 isn't found, as well as some others.
In fact, these are the missing ones (from ldd)

libffi.so.6 => not found
libstdc++.so.6 => not found
libstdc++.so.6.0.20 => not found
libz.so.1 => not found


This will be a problem in that the beaglebone only has libstdc++.so.6.0.17.
Does that mean I need to recompile opencv for this to remove that dependency?

[Sparkyshires]

hello!

Looks like an awesome resource. I'm having an issue though. When I try and compile it it gives errors on all the includes, even string and anything else from the c std library. Do you have any solutions for this?

Thanks again!

[mikhail]

hello!

Looks like an awesome resource. I'm having an issue though. When I try and compile it it gives errors on all the includes, even string and anything else from the c std library. Do you have any solutions for this?

Thanks again!

Yes, there is always a solution. Remove or change the line #8 in WPILib.h which defines "REAL" (#define REAL) to something else, e.g. "#define REAL_ROBOT"

Details: http://www.chiefdelphi.com/forums/showpost.php?p=1430768&postcount=18

[jreneew2]

Hello there! We are up to the point where we can grab image data from the camera and process it using OpenCV. However, we can't figure out how to get that jpeg data on the smartdashboard via CameraServer.

Here is what we have:

std::string videoStreamAddress = "http://169.254.149.22/mjpg/video.mjpg";
Mat frame;

VideoCapture vidcap;
vidcap.open(videoStreamAddress);
vidcap >> frame;

vector<int> param = vector<int>(2);
param[0] = CV_IMWRITE_JPEG_QUALITY;
param[1] = 80;
vector<uchar> img_data;
imencode(".jpg", frame, img_data, param);

// This is where we're stuck :)
RobotMap::cam_server->SetImage();


SetImage seems to take a Image object. We cannot find any real documentation on how to use this object. Any help would be most appreciated.

Thanks,
Team 2053

[Ben Wolsieffer]

The CameraServer is designed to work with the the NI Vision libraries and I don't know how to make it work with OpenCV. I wrote an OpenCV mjpg server a while ago (not for the roboRIO), that you might be able to make work. It requires Boost, which might be difficult to get on the roboRIO. Maybe it will help you with something though.