On board micro-controller with camera

[TogetherSword8]

Our programming team at 888 is looking at really developing our vision code. In previous years we have run target recognition through the laptop at the driver station to send back coordinates of targets (frisbees) that our robot would then drive to those coordinates, running over and picking up these objects. However, these involved stopping to recognize the target, and the calculations were not extremely time sensitive.

We are looking at possibly purchasing an on-board processing unit, but we want to know some data from other FIRST teams before we jump into a new purchase.

First, can you run an image on the processor higher than 640 by 480 at a sufficient speed (processing between 7-10 fps), or is this the max due to the limitations of the WPI Library for image processing? (We are a LabVIEW team, and a quick glance gave the highest option as 640 by 480) This would mean that we can have a higher resolution without having to send the high resolution image over the wireless system at competition.

Second, what is the difference between the latency of using a dashboard image processing system compared to an on board processor? If you were to run the processing on the computer, is the time it takes to retrieve the image wirelessly a difference when performing time sensitive applications, or is it offset by the speed of a good laptop's processor? (i.e., knowing the exact distance to the goal in this year's game, while moving towards the goal during the distance calculation)

Lastly, what processing units work well in FIRST applications? We are largely unaware of the benefits compared to drawbacks for the different processors, especially when it comes to power/capability vs price.

[adciv]

Running processing on the RoboRIO at higher than 640x480 is possible, but I would not recommend it. The frame rates drop too low. I'll have to dig out the benchmarking I did last year, but I think it was single digits.

Latency will depend on what you use for processing,what you need, and how big your images are. The variables get a little too complex to estimate, but you can code around them somewhat. In 2012, we did offboard processing and use a sample and move for our turret where we'd pause the turret, get the angle to change to, move to that positiion, stop, and recheck.

We haven't tried onboard in a bit, but next up will be trying to use the Tegra TK1 offered to First. It's a quad core 2+ghz Arm A15 running linux and 192 cuda cores. With OpenCV, this should be sufficient for anything we need. I'm not sure of anything which gives you more bang for your buck.

[RoboNerd01]

Don't forget about the very limited bandwidth FIRST allows you. Most of the time streaming high quality video causes problems. I know this has little to do with video processing, but you should always take the bandwidth into consideration.

[Greg McKaskle]

The image sizes in WPILib were selected to match the initial Axis IP cameras. When later Axis camera models introduced other sizes, the camera VIs were not updated. This wasn't a technical limit, but a practical one.

Other questions you may want to try and answer -- what do you gain by using a higher resolution? The related question is ... What is the minimum resolution you can process in order to make the measurements you are interested in?

To answer your first question directly, you are not technically limited to 640x480, but the processing time is proportional to the number of pixels, and there may be no need to use a higher resolution. It depends on the size of the objects you are trying to identify, the lens, and the distance.

To answer the second question, you can break the processing down into its components and look at the overall time spent for the different approaches. To process an image, it first has to be requested, the camera digitizes, encodes, transmits, there may be more steps here depending on the connection, etc. Then the processing end needs to decode the image and store it. It perform its processing steps, and transmit the results back to the code that cares.

Think about what affects each step and the tradeoffs of higher resolution. Your initial post already captures some of this. A PC has a faster processor than the one on the robot. Do I transmit the data to the remote PC, or do I mount the PC on the robot and simplify the transmission? Or do I simplify the processing? Can I better optimize the request and capture of the image?

It is useful to hear advice from other teams, but keep the steps above in mind as they describe why they made various tradeoffs. Then think about what you really need to use the sensor for and what resolution and framerate is sufficient.

Greg McKaskle