Vision Software comparison

[yash101]

OpenCV vs RoboRealm vs SimpleCV...What's the main difference? Also, what do you teams (with vision) use? I would like to know which is better prformance-wise, feature-wise and simplicity-wise. I currently have OpenCV and RoboRealm.

[Greg McKaskle]

An option that isn't listed is the NI Vision libraries. This is what the example/tutorial in LV uses. It runs on Windows and cRIO, and has LV and C entry points for its functions.

Greg McKaskle

[Joe Ross]

We use NI Vision on a LabVIEW dashboard, communicating to a java program using Network Tables. We've been much more successful with this approach then anything else we've tried (and we've been playing with vision since 2005).

I think much of the success is attributable to the great examples that NI puts together for FIRST, rather then something inherent in the library. In addition, doing vision processing on the dashboard is much easier then trying to do it on the cRIO or on a robot-based co-processor.

[yash101]

An option that isn't listed is the NI Vision libraries. This is what the example/tutorial in LV uses. It runs on Windows and cRIO, and has LV and C entry points for its functions.

Greg McKaskle

Crap. I forgot about that. Is there a way I can edit the poll?

[faust1706]

This is dawned on me:in order for a team to be competitive this year, they are required to do vision processing, and not just sending a feed to the driver station. Reason being that there is absolutely no way to tell unless you are analyzing the hot goal and/or the dynamic target. It will be interesting. A lot of teams get away with not using vision and just indexing on the field, but they can't do that this year, and computer vision has a (very) steep learning curve.

[gluxon]

This is dawned on me:in order for a team to be competitive this year, they are required to do vision processing, and not just sending a feed to the driver station. Reason being that there is absolutely no way to tell unless you are analyzing the hot goal and/or the dynamic target. It will be interesting. A lot of teams get away with not using vision and just indexing on the field, but they can't do that this year, and computer vision has a (very) steep learning curve.

Why wouldn't teams be able to just look in the camera view for illuminated goals?

[bvisness]

Why wouldn't teams be able to just look in the camera view for illuminated goals?

They absolutely can, except for that pesky first 10 seconds. ;)

[Tom Line]

NI vision library on the cRIO.

[omalleyj]

NI library for Java, on the cRIO for the last several years. Debating using OpenCV on a BeagleBone Black coprocessor arrangement this year.

[BradAMiller]

I'll throw in a plug for RoboRealm. In the past I've used NI Vision libraries on the robot and OpenCV on the driver station. Yesterday I decided to try making a program with RoboRealm running on the driver station sending data back to the robot with Network tables.

In about an hour I had a program that was detecting the hot target, and in another half hour had the data back to the robot using the built-in Network Tables implementation in RoboRealm. It's not yet doing distance or anything complicated, but it does make my little desktop robot open and close the claw based on the target being hot. I'll leave the hard parts for the students. I just wanted to see how well it worked.

I was able to test the program using the sample images that are included with the sample programs that come with all the languages. No field construction required, and didn't leave my desk.

Brad

[faust1706]

They absolutely can, except for that pesky first 10 seconds. ;)

That's what I was referring to. In the past, FIRST hasn't made computer vision a requirement to be as competitive as possible, but this year they are.

[yash101]

So, I should ask why your team picked these different technologies. They all work well. I, myself, prefer OpenCV because it is so powerful. It is also well-documented, so learning it can be conquered within weeks!

Also, how do you communicate with the robot? Serial-RS232/UART, i2C, SPI, NetworkTables, Winsocks? By the way, would you guys suggest using winsocks? It seems quite easy enough to use and make a socket server using!

[PaulDavis1968]

Last year we used SimpleCV (python) running on a ODROID-U2 with network tables from here:
https://github.com/robotpy/pynetworktables
So we could have it all Python and it worked fine.


The year before we did OpenCV in C++ using visual studio to compile an app on the driver station that interacted with the Java network tables on the driver side. We got great distances and it ran real fast except we ran c++ on the robot side and we could not get network tables that year working. We were going to convert the robot code to Java but ran out of time. The next year network tables worked great in c++ on the robot.

We played with the code that comes with the WPLIB this year and after adjusting the HSV setting we got decent distances with that. I am still thinking we might stick with off processing. Depends on how much we have to do in Autonomous and if we want it for other uses.

[catacon]

We ran everything on a PandaBoard last year, but we will be going with an ODROID board this year, most likely. We used OpenCV and wrote our own networking libraries. The networking stuff is pretty simple in C++, so it was easy enough to write our own and it gave us more flexibility.

The kids want to use Java this year, so that will require some adjustments in the networking code, but all of that stuff is fairly universal.

[Invictus3593]

Our vision isn't very heavy on any processor except the cRio's, so we use the dashboard. :D

[NotInControl]

OpenCV vs RoboRealm vs SimpleCV...What's the main difference? Also, what do you teams (with vision) use? I would like to know which is better prformance-wise, feature-wise and simplicity-wise. I currently have OpenCV and RoboRealm.

I think you are asking an incomplete question. Most of these libraries contain similar functions which should suffice for FRC. filters, masks, histograms...etc.

What you should be more interested in, is what system a team is using, and the type of performance they are achieving.

i.e on dashboard, achieving 20fps with 500ms lag behind real-time, on crio - 4fps with 2 second lag behind real-time. Co-processor - 10fps real-time.

(Note, I just made these up, do not take the above to be factual, it is purly for illustraing a point.)

In the above scenario, which vision system would you choose? I know exactly which one I would choose. The performace other teams achieve with their Vision setup goes beyond what library they use.

Regards,
Kevin

[gpetilli]

I think you are asking an incomplete question. Most of these libraries contain similar functions which should suffice for FRC. filters, masks, histograms...etc.

What you should be more interested in, is what system a team is using, and the type of performance they are achieving.

i.e on dashboard, achieving 20fps with 500ms lag behind real-time, on crio - 4fps with 2 second lag behind real-time. Co-processor - 10fps real-time.

(Note, I just made these up, do not take the above to be factual, it is purly for illustraing a point.)

In the above scenario, which vision system would you choose? I know exactly which one I would choose. The performace other teams achieve with their Vision setup goes beyond what library they use.

Regards,
Kevin

We are hoping to use a new camera this year called PIXY from Charmedlabs. The specs claim it finds "blobs" and transmits only the x,y coordinates via RS232 - not the entire image. This allows it to process 50FPS with 20ms lag - which is as fast at the cRIO can consume the data anyway. If we get it in time, we hope to have a ball tracking goalie in AUTO.

[Jerry Ballard]

We're using OpenCV with an ODROID U3. Running two USB cameras, where one is for ball tracking and other other one for auton "hot" target detection.

Vision coprocessor communication is two-way between CRIO and ODROID via UDP ports.

[faust1706]

We're using OpenCV with an ODROID U3. Running two USB cameras, where one is for ball tracking and other other one for auton "hot" target detection.

Vision coprocessor communication is two-way between CRIO and ODROID via UDP ports.

I'm glad to see other people using O-Droid products.

Bidirectional udp: what are you sending to the U3? Just out of curiosity.

[NotInControl]

I'm glad to see other people using O-Droid products.

Bidirectional udp: what are you sending to the U3? Just out of curiosity.

We use two-way TCP stream between our CRIO and beaglebone. For this year, the cRIO send a "MatchStart" flag.

We had to do this because, when your setting up, you will be looking at a hot target, and must wait for it to actuate away (indicating other side is hot) or stay up (indication this side is hot). If we didn't send this data, the coprocessor could not differentiate between a hot target seen in disabled mode while setting up, or a hot target seen in autonomous mode (when it really counts).

The co-processor uses this data to then wait some steadystate time(time for the hot goal panel to actuate) and return a "ValidData" bit as well as a bunch of useful information about hot goal or not back to the RIO, which is used to make decisions on how the rest of auton is played out.


You could do this on the cRIO as well, and keep the stream one way, we just opted to do it this way. We choose this way because it makes the cRIO side interface really simple: Just wait for the "ValidData" bit, and "HotOrNot" bit, or timeout.

The commands on the cRIO, have timeouts so that if for some reason we loose the camera or the stream, our auton doesn't hang and still continues, just assuming the closest goal is hot.


Hope this helps,
Kevin

[mwtidd]

We're using a hybrid of different methods.

At week zero our crio based hot detection was perfect, so we figured if it isn't broken why fix it. So that will be staying there.


This year we're spoiled as far as spare weight goes, so we'll also have a D830 on board with a logitech c310, an arduino uno, and radioshack led strip. The beauty of this set up is that it can be placed on any robot, tuned to the sweet spot and will be able to provide an "in range" meter. We use OpenCV for Java on this machine.

We also plan on using the pwm output on the arduino and inputting that into a counter so we can have a PID based on the range outputted by the UNO, but that won't be implemented until our second district event. Essentially it will be a really big "optical" rangefinder.

[ayeckley]

We've found that the NI Vision solution (running on the driver station laptop) is more than adequate for our needs this year. I promise I'm not just saying that because we are an NI Grant recipient :)

[billbo911]

This year we are using both cRio vision libraries and OpenCV on an PCDuino co-processor.

At the beginning of Auton, we will use the Apex camera and cRio to detect the Hot Goal. After detection, the vision processing is turned off.

Ball detection is done via a USB webcam and the PCDuino, this runs continuously. The cRio has a "Socket Request" vi that polls the Polls the PCDuino at 10Hz. The PCDuino responds to the socket request by only returning the latest target "X" coordinate. All code on the PCDuino is scripted in Python.

We are hoping to move ALL vision processing to the PCDuino later this season.

[ykarkason]

This year we are using the cRIO as the only vision processor, using the NIVision libraries in Java.
We use it on-demand, so the cRIO handles it pretty well, making our full goals Report fairly useful in auton.

[faust1706]

We use two-way TCP stream between our CRIO and beaglebone. For this year, the cRIO send a "MatchStart" flag.

We had to do this because, when your setting up, you will be looking at a hot target, and must wait for it to actuate away (indicating other side is hot) or stay up (indication this side is hot). If we didn't send this data, the coprocessor could not differentiate between a hot target seen in disabled mode while setting up, or a hot target seen in autonomous mode (when it really counts).

The co-processor uses this data to then wait some steadystate time(time for the hot goal panel to actuate) and return a "ValidData" bit as well as a bunch of useful information about hot goal or not back to the RIO, which is used to make decisions on how the rest of auton is played out.


You could do this on the cRIO as well, and keep the stream one way, we just opted to do it this way. We choose this way because it makes the cRIO side interface really simple: Just wait for the "ValidData" bit, and "HotOrNot" bit, or timeout.

The commands on the cRIO, have timeouts so that if for some reason we loose the camera or the stream, our auton doesn't hang and still continues, just assuming the closest goal is hot.


Hope this helps,
Kevin

That's really clever. We thought about doing bidirectional communication but decided against it, mostly due to our lack of size on our programming team. How complex is it to do? I'd love to figure this out for my team even though I'm graduating.

We have 3 modes of vision, target tracking to get x,y and heading on the field in relation to the left corner on the side we are scoring on, ball tracking, and robot tracking. All of them currently run at the same time, and it is rather intensive on our XU. It'd be great to hit a button on the DS and switch between the modes when needed. While doing all 3 we get about 10 fps with minimal lag, and maybe 15 when we don't show any images and just let the program spam the udp packets to the cRIO.

[NotInControl]

That's really clever. We thought about doing bidirectional communication but decided against it, mostly due to our lack of size on our programming team. How complex is it to do? I'd love to figure this out for my team even though I'm graduating.

We have 3 modes of vision, target tracking to get x,y and heading on the field in relation to the left corner on the side we are scoring on, ball tracking, and robot tracking. All of them currently run at the same time, and it is rather intensive on our XU. It'd be great to hit a button on the DS and switch between the modes when needed. While doing all 3 we get about 10 fps with minimal lag, and maybe 15 when we don't show any images and just let the program spam the udp packets to the cRIO.

Its not complicated, but it does take some effort to do it well. We program the robot in Java, and the FRC version of the JVM running on the robot does not support UDP, so we are forced to use TCP for all communications. TCP is an open stream and is bi-directional inheriently. It opens a socket which can simply be viewed as just an IO stream so you can write to it or read from it from either end (client or server). Once the stream is up, it stays up until it is closed or lossed.

This is different from UDP where you send data and close the connection, and have no confidence that your message was delivered.

With that said, where TCP does get complicated in the "doing it well part" is multithreading. We have multiple threads running on the robot and the beaglebone each to handle asycronous communications. Three threads run on the robot side, One to listen for incomming connections, one to read from the stream, decompose the strings, and save it in local variables, and another to send data from the robot to the bone. We have mutable looks that allow all of these threads to share data. Right now, each one of these threads run in their own periodic loop just sending and receiving data.

On the beaglebone, which we program in c++, we open 3 threads for TCP as well. One for the client, which wakes up and trys to connect to the robot once every second until a connection is established (this takes care of not having to worrying who turns on first). And the other two do similar functions of sending and receiving messages.

Now you also need to manage these threads in the event the stream is lossed and terminate gracefully. We have code on the crio and the bone, that if the connection is lost, the threads die, and then restart, where the crio goes back to listening, and the bone trys to reconnect once a second. This helps us re-establish comms in the event of power failures or other events.

The bone runs three additional threads, one running FFMPEG grabbing real-time frames from the Axis camera and saving it in a shared, mutable variable, another which runs at a slower frequency and does our image processing, grabbing the latest frame from the FFMPEG thread when needed, and a third which is a counter. The counter thread gets spawned when the matchStart flag is sent from the RIO, and just counts to 5 seconds using the hardware counter on the bone, and changes a shared variable to indicate the hot taget should have changed to the other side, and changes a shared variable again after 10 seconds to indicate auto is over, then it dies. This allows the bone to keep track of what state of autonomous we are in.

Now you do not have to do it this way, but unless you run mutiple threads your program will block when trying to read from the stream (it waits until data is on the pipe, and then pulls it in.).

We choose the robot to be the server, and the bone to be the client so that, when the robot turns on, it just listens for connections. Nothing else needs to be done until a client connects. It doesn't have to wake up and call out to anything, so in the event we loose a connection or test the crio without having the onboard processor, the cRIO is not doing any additional processing.

Right now we send data to the bone 5 times a second from the CRIO. We send data from the bone to the RIO, 15 times a second. We are still working on adjusting our timing loops, but these are working for us at the moment.

Hope this helps,
Kevin

[NotInControl]

While doing all 3 we get about 10 fps with minimal lag, and maybe 15 when we don't show any images and just let the program spam the udp packets to the cRIO.


Just as a reminder. Be carefull when spamming the robot with packets. The VxWorks operating system that all of our cRIOs run on irreguardless of the language used to program it has a hardware buffer that buffers network messages. If the software doesn't wake up in time, or enters a infinite loop for some reason and fails to reads the messages off the buffer, the network buffer will get filled, and new packets will be lossed. This will result in a loss of communication because the driverstation packets will not get through. This was discovered back in 2012.

This is a very real problem and has very devistating results. You loose comms, but more importantly, you can't even reset the cRIO from the driverstation because no packets will get through. Thus rendering your robot motionless for the remainder of the match. Any team doing co-processor related network communications should be aware of this.

Our alliance partner and good friends that year had trouble with this. And they are not only a team we highly respect, but are highly respected amongst the entire FRC community.

This was discussed in the FMS/Enisten Investigation Report Whitepaper released after Championships in 2012.

Hope this helps,
Kevin

[mhaeberli]

We're still figuring it out. We have played a bit with the cRIO based vision sample and with RoboRealm and OpenCV. We do have a working homebrow interface from Visual Studio C++ to NetworkTables on the robot (which is the way we're thinking of going). It appears that the python library implementation of NetworkTables that can be found here also has a complete NetworkTables implementation.

[faust1706]

Just as a reminder. Be carefull when spamming the robot with packets. The VxWorks operating system that all of our cRIOs run on irreguardless of the language used to program it has a hardware buffer that buffers network messages. If the software doesn't wake up in time, or enters a infinite loop for some reason and fails to reads the messages off the buffer, the network buffer will get filled, and new packets will be lossed. This will result in a loss of communication because the driverstation packets will not get through. This was discovered back in 2012.

This is a very real problem and has very devistating results. You loose comms, but more importantly, you can't even reset the cRIO from the driverstation because no packets will get through. Thus rendering your robot motionless for the remainder of the match. Any team doing co-processor related network communications should be aware of this.

Our alliance partner and good friends that year had trouble with this. And they are not only a team we highly respect, but are highly respected amongst the entire FRC community.

This was discussed in the FMS/Enisten Investigation Report Whitepaper released after Championships in 2012.

Hope this helps,
Kevin

I understand the concern. When I say spam the cRIO, I simply mean sending the vision solutions. This isn't even the fastest vision program in the history of the team. Last year was at 27 with an odroid x2, during competition and 33 at school on a much more powerful computer. Sorry for the poor wording.

I remember hearing about that. It would be the absolute worst to make it to Einstein then not be able to compete at full capacity due to network errors. I remember in 2011 a robot never moved in the finals on einstein in both matches. I think FIRST is doing better at addressing this issue with the practice matches, but it is sad that it happens.

[blujackolantern]

We use the NI vision libraries in Java. All we do is separate the particle based on color and size and then wait for a horizontal target. We do that by comparing the top and sides length relative to each other.