Vision: what's state of the art in FRC?

[MrRoboSteve]

Our team is wanting to get serious about vision this year, and I'm curious what people think is the state of the art in vision systems for FRC.

Questions:

1. Is it better to do vision processing onboard or with a coprocessor? What are the tradeoffs? How does the RoboRIO change the answer to this question?

2. Which vision libraries? NI Vision? OpenCV? RoboRealm? Any libraries that run on top of any of these that are useful?

3. Which teams have well developed vision codebases? I'm assuming teams are following R13 and sharing out the code.

4. Are there alternatives to the Axis cameras that should be considered? What USB camera options are viable for 2015 control system use? Is the Kinect a viable vision sensor with the RoboRIO?

[billbo911]

Originally Posted by MrRoboSteve

Our team is wanting to get serious about vision this year, and I'm curious what people think is the state of the art in vision systems for FRC.

Questions:

1. Is it better to do vision processing onboard or with a coprocessor? What are the tradeoffs? How does the RoboRIO change the answer to this question?

2. Which vision libraries? NI Vision? OpenCV? RoboRealm? Any libraries that run on top of any of these that are useful?

3. Which teams have well developed vision codebases? I'm assuming teams are following R13 and sharing out the code.

4. Are there alternatives to the Axis cameras that should be considered? What USB camera options are viable for 2015 control system use? Is the Kinect a viable vision sensor with the RoboRIO?

Solid questions Steve!

I can not speak for all of FRC. I can only speak from the perspective of a 3 time award winner for the vision/object tracking system we developed and used in 2014.

I'm going to try to answer your questions, from our perspective, in order.

1) Up through 2014, it was "better" to use a co-processor. Vision tracking on the cRio will work, but unless it is done VERY CAREFULLY and with limited requirements, it could easily max out the processor. We did use it successfully a few years ago, but we really limited the requirements. Since 2013, we started using a PCDuino (This link is to a retired version, but it is the version of the board we used.).
How does the RoboRIO change the answer to this question? Sorry, I can't say either way. We are not a Beta team, so we have no direct experience with the RoboRio. My guess is, it will have "much better" performance, but how much better remains to be seen. (I too am curious!)

2) We used OpenCV running under Ubuntu. Our scripts were written in Python. We will likely stick with this for 2015, but that determination is yet to be made. The PCDuino offers excellent access to the GPIO pins, thus allowing us to do some really neat software tricks.

3) Our code was shared in this post. That post is the main reason we won the "Gracious Professionalism Award" on Curie this year. This code is board specific, but can easily be modified to run on many different boards.

4) Sorry, without being a Beta team, we can not address this question.

[faust1706]

Originally Posted by MrRoboSteve

Our team is wanting to get serious about vision this year, and I'm curious what people think is the state of the art in vision systems for FRC.

Questions:

1. Is it better to do vision processing onboard or with a coprocessor? What are the tradeoffs? How does the RoboRIO change the answer to this question?

2. Which vision libraries? NI Vision? OpenCV? RoboRealm? Any libraries that run on top of any of these that are useful?

3. Which teams have well developed vision codebases? I'm assuming teams are following R13 and sharing out the code.

4. Are there alternatives to the Axis cameras that should be considered? What USB camera options are viable for 2015 control system use? Is the Kinect a viable vision sensor with the RoboRIO?

There are a number of threads that already talk about computer vision in FRC, however, it never hurts to talk about it every once in awhile as the participants of the competition are always changing.

I'll go through your questions one at a time.

1. I do not know if this question has an objective answer. Teams have had success with with the cRIO, off board, or using a second computer like the PI or an o-droid. The trade off for using an on board computer is obviously weight, but PI weighs something like 200g so that shouldn't concern you. You have to somehow communicate what the vision system outputs to your cRIO, for the past 3 years we've been doing a simple udp.

As for off board, I have heard it can be slow, and with the cap on data per team, I personally think it is no long a valid option.

Using the cRIO, and programming it in labview or something, is doable, and there are plenty of resources around to do so, but the truth is it won't be as fast a something running on an SBC.

I haven't looked at the specs of the roboRIO, so my opinion on that shouldn't be taken seriously. I heard it is "linuxy," what have yet to get a clear definition of what that actually means. It would be cool to make your vision program an application and run it on the roboRIO, but I don't know if that is doable.

2. As for vision libraries, any will do, but some have more support than others. I don't know the state of openni, but last time I checked it was no longer funded. OpenCV has been around since....I want to say 1999. It is used in industry and academia and that is the one I suggest using, but of course there exists bias. I played around with roborealm a tad, but it seems too simple, I feel someone using it wouldn't get the fundamental understanding of what is happening, but if you are just looking to figure out how far away you are from something and don't really care about understanding, then I suggest that.

There is always the option of writing your own computer vision library, of course, but I'd be willing to bet money it wouldn't be as encompassing or efficient as existing open sourced cv libraries, like ni and opencv.

3. I like to think computer vision is a little unique. Yeah, you can develop code bases, but the vision task changes so dramatically over the years, except for game piece detection, that I don't think it'd be worth it. Take opencv for example. It is an extremely developed library that now focuses on readability and ease of developing. If you really know what you're doing, you can solve the vision program in 100 lines of code, if you don't know what you're doing, it can take upwards to 2k. You could maybe group some routine operations, such as binary image morphology, into one step, but even then you'd only be saving one line of code.

Once you get a game piece detection code, however, that can just be "pulled from the shelf" and used year after year.

4. As long as your library can get the image from the camera, it really doesn't matter which one you use. The common option is the axis camera. We've used the kinect for 2 years, then this past year we used 3 120 degree cameras (it was more a test of concept for future years). If you want to do some research on hardware, look up: asus xtion, pixy, playstation eye, kinect, and keep and eye out for the kinect one to be "hacked" and thus being usable.

I feel that computer vision is a developing aspect of frc. My freshman year it was logo motion. I didn't ask around in the pits, but I didn't see many cameras to my memory. When 1114 and 254 were doing multiple game piece autonomous (in the einstein finals, mind you), I think it really inspired people I think, as did 254's autonomous routine this year. Just look at all path trajectory stuff that has been posted over the path several months.

This is my 2 cents on the topic. Take it how you will. If you have any questions, or want to see some code from my previous vision solutions (from 2012-2014), I'd be more than happy to go over it with you.

I emailed a first rep requesting they consider an aerial camera for future games, and if they do that, or allow teams to put up their own aerial camera, then on board vision systems become obsolete because essentially you have an "objective" view, instead of a "subjective" one.

[Greg McKaskle]

A few clarifications and a few of my own answers.

NI Vision isn't the same as openni.

The OS on the roboRIO is described in quite a bit of detail here http://www.ni.com/white-paper/14627/en/

The code is on github.

1. There are tradeoffs here that depend on the robot, the strategy, and the team's resources. All have been made to work. The roboRIO is somewhere between 4x and 10x faster. It has two cores and includes NEON instructions for vector processing.

But vision is a hungry beast and will bring any computer to its knees if you just brute force it. The "right" solution depends on where the camera is mounted, where it is pointed, lighting, lenses, and of course processing. Makes it challenging, huh.

2. All of them are capable of solving the FRC imagining problems quite well, IMO.

3. I've seen plenty who make it work.

4. The WPI libraries will continue to support Axis and will support many UVC cameras via NI-IMAQdx drivers. This means that many USB webcams will work. Additionally, we have also tested the Basler USB3 industrial cameras. I have not tested Kinect directly on the roboRIO, though if you google myRIO and Kinect you will see others who have.

Greg McKaskle

[ToddF]

One word: CheesyVision

(From the perspective of a Mech E)

[faust1706]

Originally Posted by Greg McKaskle

But vision is a hungry beast and will bring any computer to its knees if you just brute force it. The "right" solution depends on where the camera is mounted, where it is pointed, lighting, lenses, and of course processing. Makes it challenging, huh.

I don't think people realize this. After the image is acquired, it is literally pure mathematics. If you aren't careful, as in don't put in a wait function, your computer will try to execute the program as fast as possible, and the program is basically in an infinite loop until you somehow close the program. (I believe the majority of teams don't gracefully exit their program after a match, but instead simply power off).

To give an example, when you have a binary image you desire, as in your targets are white (1) and everything else is black (0), you preform the operation described in this paper (http://wenku.baidu.com/view/6cb52ede...aa811dad5.html) This paper was published in 1983. There do exist other methods, but the computational cost is about the same. (See also: http://en.wikipedia.org/wiki/Edge_detection).
Basically what this step does is reduces a 640x480 (or whatever resolution image you have) matrix to a sequence of curves that allow you to do cool things, such as recursively approximate a polygon to guess how many sides it has.

[yash101]

What is the purpose? Do you want to process a handful of images at the beginning of the match, or do you want to continually process images and perform AI like pathfinding? Those questions are extremely important for answering many of your questions. The first option should not require more power than the RoboRIO offers. The second option would probably benefit from a coprocessor. These tiny boards can offer a lot of power, easily many times the available oomph of the RoboRIO.

Running OpenCV on the RoboRIO is far from realistic. The hard disk space is quite limited, and as it seems to me, there will be missing pieces of linux which will make it extremely difficult to compile all the dependencies of OpenCV.

Also, I think OpenCV can be a tad inefficient in general. I was running a small test and OpenCV's windows maxed out one of my system's cores (i3)!

I really think converting color spaces, thresholding and some of these relatively simple transformations should be much faster, especially because the formula should be constant.

[faust1706]

Originally Posted by yash101

Also, I think OpenCV can be a tad inefficient in general. I was running a small test and OpenCV's windows maxed out one of my system's cores (i3)!

I really think converting color spaces, thresholding and some of these relatively simple transformations should be much faster, especially because the formula should be constant.

It can be inefficient, but it great at what it does. It is an open source library that people volunteer their time to contribute to. I think I'm going to be able to get away with taking 6 credit hours next semester for doing research in the field of computer science. That means I'll have dedicated time to work on stuff, like optimizing the opencv library (or just the functions I use regularly, such as erode, dilate, find contours, appoxpoly, solvepnp, and optical flow)

I've still not looked at the source code for opencv in depth. I really hope some things in it aren't parallel-ly computed because I love doing parallel computing and it'd give me great practice.

Like you mentioned...2 months ago to me, it spawns threads when it'd be quicker not to do it. Maybe I could simple put a condition that if the image resolution < (x,y), then don't spawn another thread.

[Caboose]

Team 900 this year is currently working with a nVidia Tegra TK1(Quad ARM Cortex-A15 with Kepler GPU with 192 CUDA cores, <$200) to tackle vision processing. So far we are seeing good results with high FPS(>40) with large image resolutions(720p) doing just filtering at the moment all on the GPU with OpenCV. We are working on tracking next.

[Abhishek R]

Answering question 4, the Kinect is a viable means of vision sensing. I'd recommend checking out this paper from Team 987, who used the Kinect very effectively as a camera in 2012's FRC challenge, Rebound Rumble. I believe one of the major advantages of the Kinect is it's depth perception is much better than a standard camera, though I'm not really a vision expert.

[faust1706]

Originally Posted by Abhishek R

I believe one of the major advantages of the Kinect is it's depth perception is much better than a standard camera, though I'm not really a vision expert.

I don't really understand what you mean by depth perception. It has a depth map, if that is what you mean. The kinect is a great piece of hardware because it has an rgb, ir, depth camera, and a built in IR light (though I suggest adding some IR leds if you're going the IR route). It is fairly cheap now, and there has been a lot of work done with it, but if you're using only the rgb camera, the only thing different is how much it distorts the image. All the cameras you will ever think about using basically have the same amount of distortion, so that shouldn't be a concern. If it is, then calibrate your camera:

http://docs.opencv.org/doc/tutorials...libration.html

But I'm going to say that it is a waste of time unless you are doing an camera pose calculation.

The kinect does have a subtle advantage over other cameras, like the axis: It draws attention to it. Little kids love seeing things they see everyday used in a way they never thought possible.

[yash101]

Originally Posted by Abhishek R

Answering question 4, the Kinect is a viable means of vision sensing. I'd recommend checking out this paper from Team 987, who used the Kinect very effectively as a camera in 2012's FRC challenge, Rebound Rumble. I believe one of the major advantages of the Kinect is it's depth perception is much better than a standard camera, though I'm not really a vision expert.

That is quite an old document. OpenKinect has changed significantly and is much harder to use now! The documentation kind of sucks as the examples are all in very difficult (for me) C!

The greatest problem with the Kinect was getting it to work. I have never succeeded in opening a kinect stream from OpenCV!

The depth map of the Kinect is surprisingly accurate and powerful!

As of last year, thresholding was the easy part ! Just create a simple OpenCV program to run on your PC, to connect to the camera and get video! Create sliders for each of the HSV values, and keep messing with one bar until the target starts barely fading! Do this for all three sliders. You want to end with the target as white as possible! It is OK if there are tiny holes or 1-4 pixels in the target not highlighted. Next, perform a GaussianBlur transformation. Play around with the kernel size until the target is crisp and clear!

Last year, I use std::fstream to write configuration files. It is a good idea, unless you get a program that has a much better configuration parser! Just write the HSV values to the file and push it onto your processor! Voilla! You have your perfect HSV inrange values!

Hunter mentioned to me, last year, that when at the competitions, as soon as possible, ask field staff if there will be time where you will be able to calibrate your vision systems! At the Phoenix regional, this was during the first lunch break! USE THAT PERIOD! Take the bot on the field and take a gazillion pictures USING THE VISION PROCESSOR CAMERA, so when you aren't under as much stress, you can go through a couple of them at random locations and find the best values!

As I mentioned before, and will again in caps lock, underline and bold:
SET UP A CONFIGURATION FILE!

This way, you can change your program without actually changing code!

[controls weenie]

Originally Posted by Caboose

Team 900 this year is currently working with a nVidia Tegra TK1(Quad ARM Cortex-A15 with Kepler GPU with 192 CUDA cores, <$200) to tackle vision processing. So far we are seeing good results with high FPS(>40) with large image resolutions(720p) doing just filtering at the moment all on the GPU with OpenCV. We are working on tracking next.

That's great that you are learning the nVidia Tegra TK1. I would like to know more about your experiences with this device. Will you post some example code? Are you using C++ or python?

We used the PCDuino last year and we downloaded code from Billbo911's (above) website. His code is written in python and very easy for the kids to understand. It worked very well last year. We tracked balls, reflective tape and bumper reflectors. We will probably use the PCDuino and Bill's code again this year.

We used a USB web cam but had do down sample to 480x320 to maintain 30Hz with our image processing output. OpenCV and python work very well but you have to be careful because python loops can slow you down.

One thing that I did not see mentioned in this thread is how to enhance the reflective tape image. We used white ring lights last year which is very WRONG! There is so much ambient white light that we had a terrible problem tracking the reflective tape. I recommend using 3 green ring lights. Then pre filter only green pixels. You can buy small diameter, medium and large. The small fits in the medium and the medium fits in the large.

[marshall]

Originally Posted by controls weenie

That's great that you are learning the nVidia Tegra TK1. I would like to know more about your experiences with this device. Will you post some example code? Are you using C++ or python?

We used the PCDuino last year and we downloaded code from Billbo911's (above) website. His code is written in python and very easy for the kids to understand. It worked very well last year. We tracked balls, reflective tape and bumper reflectors. We will probably use the PCDuino and Bill's code again this year.

We used a USB web cam but had do down sample to 480x320 to maintain 30Hz with our image processing output. OpenCV and python work very well but you have to be careful because python loops can slow you down.

One thing that I did not see mentioned in this thread is how to enhance the reflective tape image. We used white ring lights last year which is very WRONG! There is so much ambient white light that we had a terrible problem tracking the reflective tape. I recommend using 3 green ring lights. Then pre filter only green pixels. You can buy small diameter, medium and large. The small fits in the medium and the medium fits in the large.

We're using OpenCV in C++ so example code is plentiful around the web. Our students have just now started to use it so we don't have anything to share just yet. If we make progress to the point where we can share it then we will, probably towards the end of build season.

The big deal with the TK1 is that it has the ability to use the GPU to assist with offloading work. To my knowledge, there is no method to use the GPU assisted functions for OpenCV with Python currently but that might be changing with the 3.x code release around the corner. We're using the 2.4.x code right now.

C++ is what we are using for the GPU integration as of right now because you have to manually manage the memory for the GPU and shuffle images onto it and off of it as you need them. Nvidia has a decent amount of resources out there for the Jetson but it is definitely not a project for those unfamiliar with linux. It's not a Raspberry Pi and not anywhere near as clean as a full laptop. To get it working you have to do a bit of assembly. It's a nice computer, just not as straight forward as a Pi or a PCDuino or any of the others that have larger user bases. There are also problems running X11 on it so you really need to run it headless (Nvidia writes binary blob graphics drivers for linux that are not super stable).

We're aiming for full 1080 but depending on the challenge we will likely have to down sample to 720 to get it to work with the frame rates we need.

Granted, this is all off-season right now and we have a lot of testing to do between now and the events before any of this is guaranteed to go on the robot. For all I know FIRST is going to drop vision entirely... I mean, cameras don't work under water do they?

[controls weenie]

Originally Posted by marshall

We're using OpenCV in C++ so example code is plentiful around the web. Our students have just now started to use it so we don't have anything to share just yet. If we make progress to the point where we can share it then we will, probably towards the end of build season.

The big deal with the TK1 is that it has the ability to use the GPU to assist with offloading work. To my knowledge, there is no method to use the GPU assisted functions for OpenCV with Python currently but that might be changing with the 3.x code release around the corner. We're using the 2.4.x code right now.

C++ is what we are using for the GPU integration as of right now because you have to manually manage the memory for the GPU and shuffle images onto it and off of it as you need them. Nvidia has a decent amount of resources out there for the Jetson but it is definitely not a project for those unfamiliar with linux. It's not a Raspberry Pi and not anywhere near as clean as a full laptop. To get it working you have to do a bit of assembly. It's a nice computer, just not as straight forward as a Pi or a PCDuino or any of the others that have larger user bases. There are also problems running X11 on it so you really need to run it headless (Nvidia writes binary blob graphics drivers for linux that are not super stable).

We're aiming for full 1080 but depending on the challenge we will likely have to down sample to 720 to get it to work with the frame rates we need.

Granted, this is all off-season right now and we have a lot of testing to do between now and the events before any of this is guaranteed to go on the robot. For all I know FIRST is going to drop vision entirely... I mean, cameras don't work under water do they?

Oh yeah...I forgot about the water issue

I see an issue getting a USB camera driver reading the image more than 30Hz. This was an issue with the PCDuino and our web cam last year. The Ubuntu USB driver would not feed the processor more than 30Hz. Dumping the images from RAM to GPU could be a bottle neck because of the huge sizes of the frame buffers.

I used python binding at work to copy data to (and from) the GPU queue. Python might be easier for the kids to use if it is available. I wonder if you can use OpenCL on the TK1 dev kit? OpenCL might give you the OpenCV/python bindings on that OS.

I hope FIRST continues to have image processing during the games. Some of the kids enjoy that more than any other task. Good luck with the TK1.