Vision... what? how?

This will be our teams 3rd year coming up. Our goal is to learn to incorporate new things into our robot each year. Last year we started using Pneumatics and Java. This year I would really like to see us incorporate vision. We know basically NOTHING about how to do this. So I’m going to ask a bunch of questions and if you can answer some of them, that would be great.

  1. As I understand it onboard processing is much preferable to trying to use the classmate. Is this right?

  2. What kind of equipment do you use to do the onboard processing? I haven’t seen anyone with a laptop ziptied to the robot.

  3. How would you communicate with this onboard vision computer? Does it hook up to the router via ethernet?

  4. How do you get the image to the onboard computer from the camera?

  5. Is the stock camera acceptable for this? Do we need separate cameras?

  6. How do we program this onboard computer? Do we install labview on it? Java?

  7. I’ve heard of RoboRealm for vision processing. Would this be installed on the classmate? the onboard computer? do people use this? is it good?

  8. Assuming the onboard computer is getting the image from the camera, and processes it correctly how do we get information (like width, y coordinate, color, etc) from the onboard computer to the java code?

Thanks for all your help. Hopefully by the time the season starts we can have some idea what were doing.

1.There are three ways this can be done. On the cRIO, on the Driver Station laptop, and on an on board processor. The cRIO isn’t great for vision because it doesn’t get a high frame rate with complex processing stuff on it. The laptop is what we used with a lot of success. We used a laptop other than the classmate, and it worked really well. It doesn’t have to be fancy, we used a cheap toshiba one. The onboard processor is the most complicated/difficult, and works the same as the laptop.

  1. Some teams use laptops, some use a board like raspberry pi or beaglebone, others use modified desktop computers, some use a camera board with a processor build in, like CMUcam.

  2. Yes, you can use ethernet. You can also use serial to the cRIO, or I2C.

  3. If you are using an axis camera, you can grab the image from the network. You can also use other cameras like a USB webcam or kinect, and get the image using OpenCV.

  4. Yes, you can use the axis camera. However, if you chose to use the on board processor, you could also use a webcam, but there’s nothing wrong with the axis camera they give you.

  5. You could program it with labview’s vision stuff, but I would recommend using openCV, which can be used with C, Java, and Python. See 341’s example of how they used openCV in 2012 on their driver station laptop.

  6. I’ve never actually used roborealm. It is one way to do vision, but you don’t need it.

  7. You can send the image data to the cRIO through the ethernet connection.

Here’s my advice. If you’re trying out vision, it’s going to be much easier to play around with running the vision processing on the driver station laptop. You can do this with Labview, roborealm, or openCV.

So ultimately we will want to go to an onboard independent system. For the moment though, we have a 4 year old laptop that we may just ziptie to last years bot.

Then we will connect it via ethernet to the router.

How do I grab images from the camera?

I install open CV (and learn how it works) on the vision computer.

How would we then send info to the Crio from the vision computer?

Sorry for all the questions. We are in an area where we really are on our own most of the time.

Most vision systems take a ‘shot’ of their target, calculate what changes need to be made to aim directly at it, and then have the robot perform those actions.

As a result, well written vision code takes almost no processing power, since you only really need one good frame to make all your calculations. That means that the cRIO can easily handle vision processing. Press your trigger to fire, camera takes a shot, turret or drivetrain aims, then your robots fire.

If you’re talking about real-time vision processing, then your best bet is your driverstation laptop. The camera can stream video directly to the laptop, then you can return your targetting data to the cRIO using network tables.

I will tell that after 4 years of writing vision code, we still haven’t managed to write one that is more accurate and faster than an experienced robot operator. Don’t get too bogged down in the coolness factor of vision. It’s nice, but it isn’t really needed for most games and the complexity level is quite high when you try to use it.

You can find frc vision examples written for the cRIO in the Help or Support section of LabVIEW if you want to go that route. They are already configured to detect the targets for the given year’s game.

I agree with what Tom said, realtime vision is best done on the driver station laptop. It’s cheaper, simpler, doesn’t count towards the robot budget, and doesn’t increase the weight of the robot. It’s also way easier to set up for a team who is just starting to work with vision.

The only advantage of an onboard image processing board is the ability to use a different camera, or to interface with sensors on your robot.

From experience, programming in Open CV means you have to handle the (in)stability issues in the program or you’ll lose your vision processing. I tried Open CV this year but had some null pointer issues running some of the libraries. This occured with some particularly noisy images.

That said, there is still an advantage to running vision processing on the robot in terms of reduced bandwidth requirements and reduced lag. The question becomes, does the advantage make up for this complexity? For 2014, I’m considering purchasing a sub $400 windows tablet and installing labview on it to run video processing on. Less than one pound of weight and about 1/4" thick.

Regardless, I recomend purchasing a new laptop if you can afford it. I do not like the classmates from a cost or durability perspective. I replaced our driverstation in 2012 with an i7 laptop for about the same price as a new classmate would cost, but with about an order of magnitude more computing power.

… OpenCV is a great route if you’re going to use a camera like the Kinect or Asus Xtion Pro (which allow you to measure depth). The problem with that is that it’s not easy to set up. You have to rebuild the libraries from the source to get things like OpenNI support, CUDA (gpu core) support, multi core or hyperthreaded core support, and other cool stuff. I have personally been trying for the last week to successfully build it using CMake with little or no success (I’m going to try to find someone to FTP me a working build). There’s quite a few options available if you’re going to go with a small form-factor desktop computer on your robot.