How to run vison processing code on classmate during matches?
 

My team is going to be using vision processing for our hybrid mode. From what I understand, rather than running the processing code on the CRIO, it is run on the classmate or other laptop used for driving by my team. How do I make this separate code run on the robot while the robot is running?

Re: How to run vison processing code on classmate during matches?
 

You would have to send the video frames to the computer, run your image processing program, and have it send relevant commands to the cRIO, all over the wifi link. IMO it would be better to have an on board laptop that does the same scheme, but now, it can be connected through Ethernet, which is much faster.

Re: How to run vison processing code on classmate during matches?
 

The forums have various other threads discussing this scheme. Basically, connect your axis camera to your router and grab the stream of of it. But note it is possible to run NIVision on the cRIO (although I think someone mentioned they were attempting to offload it due to it being so CPU intensive).

Re: How to run vison processing code on classmate during matches?
 

Are we allowed to have a laptop right on our robot to do the image processing? I was definitely planning to offload the images from the camera, rather than using the CRIO because there is just so much more processing power on a laptop. What I am asking really is how to execute the processing code that both finds the rectangle, and then figures out where the robot needs to go, on the laptop, rather than any of the other parts of the process, such as how to process the images.

Re: How to run vison processing code on classmate during matches?
 

yes and no:
Kinect sensor->data: yes, but its already built
axis camera processing: no (though you can, as you can see from the rest of this thread)
kinect data-> drive commands: no, on robot

If you are doing insane calculations on images, you should offload it to the classmate. If you are doing more simple image processing (rectangle tracking, etc...), the robot will suffice, as offloading it will take more work than it is worth

Re: How to run vison processing code on classmate during matches?
 

If you plug the camera directly into the DLink switch on the robot, then programs that know the IP address and are on the subnet can request images. This is how the dashboard works. If you wish to change the dashboard, it is written in LV and source is provided, or you can choose one of the other DB tools.

So, getting an image from the camera is as simple as getting the laptop program to ask for it.

As for processing it, you can process it on the cRIO. The target isn't moving, and as long as the robot isn't moving very much, you don't need 30fps. Calculate the speed you need. For processing the image, you have a number of choices. NIVision works on the cRIO and on the desktop and has entry points for C and LV. OpenCV is another pretty accessible choice.

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

Well, thats part of the problem we are having, my team is programming in java, and none of the other programmers on the team are willing to switch to C, which is what I am the most familiar with.

Re: How to run vison processing code on classmate during matches?
 

I'm not sure what to do about that. OpenCV is definitely an option, and if someone is willing to do wrappers, NI-Vision is too.

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

What would have to be done to create the wrappers? None of the programmers on the team are very experienced with writing robotics java code.
Also, what I am really trying to ask in this thread is once the code is written, how do I execute it.

As an aside, the plan my team has is to use the rectangle as a way of positioning the robot, as we plan to have an extremely short range shooter. Would you still suggest processing at less than 30 FPS and on the cRIO?
Today 08:13 PM

Re: How to run vison processing code on classmate during matches?
 

I don't want to hijack this thread, but I have heard usually reliable sources say that sending pictures back to the Classmate (or whatever Win7 Machine you use on your driver station) is a bad idea.

The gist of their argument is that nearby the wifi is up to the task but from across the field with other robots all talking too, the frame rate drops to the low single digits (3-4 fps).

This is the kind of thing that scares me to death because I can't know that it is a problem until we get to an actual competition.

So... ...should I worry about the 10,000 other problems I have to worry about or should I continue to worry about this one (and perhaps decide that I don't really want to send the camera data to the remote Win7 Machine after all)?

Do tell.

Joe J

Re: How to run vison processing code on classmate during matches?
 

Last year we did such a scheme (no data sent back to robot, however) and were running at about ~21 fps

However we did have some small issues with the robot hiccuping. We believe the problem was the classmate's DS commands due to high latency (we graphed some data obtained from wireshark). We have not ruled out the camera directly attached to the wifi with 100% certainty...

Re: How to run vison processing code on classmate during matches?
 

I have heard similar things, however, the counter-argument that I have heard is that running image processing on the cRIO, if ran simultaneously with enough code (no idea how much though) causes noticeable lag in the processing speed. One solution to this that I have heard is to have a second laptop mounted on the robot, and directly wire that to the camera and cRIO, however, I am not sure if the game design rules allow this or not.

Re: How to run vison processing code on classmate during matches?
 

You can't communicate between the robot and the laptop in hybrid/autonomous can you? or are things like the keyboard and joysticks the only things disabled?

Re: How to run vison processing code on classmate during matches?
 

If allowed to put another machine on the robot for image processing, you may as well use the kinect on it instead of the axis! I've heard its very easy to get good data, much more than the axis could ever give you and with a computer with usb on the robot it is very possible and legal to do it. (The kinect anyways, I know nothing of the computer on the robot)

Re: How to run vison processing code on classmate during matches?
 

I can vouch that grabbing the image stream from the Axis camera works fine. Our team got ~29 fps using the smaller resolution. The 3-4 fps I would imagine comes from someone trying to use the cRIO to relay the images.

As for communication during hybrid, you can, the only thing disallowed is human input (well, excepting the kinect).

Re: How to run vison processing code on classmate during matches?
 

Not knowing much about this situation, I can't say for sure what their framerate was on and off the field, but if they used the default dashboard and the M1011, then the framerate would have been single digits to start with.

The default dashboard, along with the cRIO camera communications moved from SW timed requests of individual JPEGs to HW camera timed MJPG stream this year once it was discovered that the M1011 had poor performance with the JPEG route. Both options are still in the LV palette, but the default is now MJPG. The initial decision was somewhat arbitrary, and C and therefore Java, were already using MJPGs. Depending on the setup, other factors could have played a part. It is easy to chalk it up to the "field", but I have never witnessed this and it doesn't feel like the actual culprit.

As mentioned, it is somewhat difficult to simulate a match in your shop, but on the network you have, you can look at utilization and latency. You can look at how/if it changes when a second robot is added. You can also do some back-of-envelope calculations to see how much of the N speed network you are using.

My final advice is to look at the elements being measured and use those requirements to determine the rates and resolutions needed, as well as the appropriate sensor to use.

Due to slow speeds (30 Hz) max, somewhat high latency (>60ms, often 150ms), and variable jitter, cameras are not necessarily a good sensor to close a loop with. It is far better to calculate where the target is and use an encoder or pot to turn the turret. If the robot is to be turned, use a gyro. More CPU does little to improve the numbers. Higher speed cameras exist, but they are not in the kit, their cost is pretty high, and it may be difficult to integrate them.

I think the camera is a very valuable sensor, but it all depends on how it is used.

To the original topic, the laptop allows you to bring more CPU to the table, to process images more thoroughly, at a higher resolution, and perhaps at a faster rate. Once you have an algorithm that demands more CPU, this seems like a good step. Until then, ...

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

All right, so, I have gotten a ton of really helpful information, now what I would like is some help figuring out how I should put it all together.
Here is what I am hoping to do, in its entirety:

Use the camera on the robot to find the vision target,

Export images from the camera to either our driverstation computer or a laptop mounted on the robot,

Use the NI-Vision software to find where the rectangles are,

Take the rectangles found by the NI-Vision software and calculate the distance from and angle off of perpendicular with the backboards,

Using these distances and angles, which will be updated throughout the course of the robots movement, calculate how our joystick would need to be moved in order to move our robot with a short range shooter up to the backboards,

And finally, send these theoretical joystick controls back to the cRIO and use them to move the robot to where it needs to go.

I am starting to get how to get the images from the camera, I can theoretically do the calculations on the angles and movement required, but what I still need to figure out is how to create the virtual joystick and then how to send the virtual joystick controls to the robot, as well as how to make the program I write run on whatever computer I decide it needs to run on.

Re: How to run vison processing code on classmate during matches?
 

So can you do processing on the robot with a laptop as long as you don't control the robot or can't you?

Re: How to run vison processing code on classmate during matches?
 

Currently the rules make it look like you can do the processing on a laptop that is mounted on the robot as long as it doesn't directly control the robot, however, we are not fully sure on this as of right now. Q+A qestion perhaps?

Re: How to run vision processing code on classmate during matches?
 

During the course of the rest of today, I got a fair amount of work done on figuring out how do deal with the positioning calculations, but I am still having a bunch of trouble figuring out how to both actually run the code that gets written, as well as how I will want to send the processed data back to the robot.

Re: How to run vison processing code on classmate during matches?
 

I'm looking for the exact same information. We are using Java on our robot, and are trying to decide which is best - using the cRio to do the image processing, or the drivers station laptop. We had been assuming the laptop, but I'm having a hard time finding the info regarding how exactly you communicate information from Labview back to the Java code running on the robot. Do you customize the driver station software to do the image processing and send it through the packets there? Or do you do a completely standalone Labview application and communicate via TCPIP somehow?
I'm starting to lean back towards doing it all on the cRIO to avoid that whole issue, but then I'm not sure if I can take advantage of the NI Vision Assistant modeling information.....
So many questions....thanks in advance for any additional guidance!

Re: How to run vison processing code on classmate during matches?
 

So, I was running the vision assistant software, and when I told it to acquire an image, it asked me for the NI-IMAQdx driver which I should download from the NI image acquisition software, just something to keep an eye out for when any other teams are working on setting something like this up.

Re: How to run vison processing code on classmate during matches?
 

I believe IMAQdx is used for some of the other camera types, but the bottom Axis camera plugin should be independent of it. They are somewhat comparable, but the bottom one has more config options.

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

Greg, you mentioned "the bottom Axis camera plugin". Is that some plugin that I should be seeing in Vision Assistant to be able to access the Axis camera directly? If so, I'm not seeing it. Any help would be appreciated.
Dale

Re: How to run vison processing code on classmate during matches?
 

I am not seeing a acquire from axis camera button either, my options are acquire image:
standard camera
from IEEE 1394, Gigabit ethernet, USB, or IP camera
smart camera
or RT camera.
all of these options required the IMAQdx driver, which i downloaded along with the NI vision acquisition software

Re: How to run vison processing code on classmate during matches?
 

After the cameras you list, there should be a Simulate Acquisition, and then an Axis IP item. You may need to use the scrollbar. Mine looks like the image I attached, and thin only ones that aren't grayed out are the simulate and Axis.

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

Mine also has the simulate acquisition button, however, there is no acquire axis IP button for some reason. Part of it may have been that when I installed everything from the NI software USB drive for some reason it didn't install the driver station software, which I thought I had told it to install. This issue also happened on one of my teammate's laptop as well.

Re: How to run vison processing code on classmate during matches?
 

That may be. The Axis is a plug-in. Do other people have it?

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

I noticed this (DS missing) as well. but the update took care of that. Perhaps some of the NI Vision stuff also requires the update?

Re: How to run vison processing code on classmate during matches?
 

Which update is this? My software doesn't attempt to update when I run it, and currently, we can't find our team NI software installer USB.

Re: How to run vison processing code on classmate during matches?
 

http://www.usfirst.org/roboticsprogr...control-system
very bottom of the page. Also in the Getting Started with the Control System manual. All languages require the DS and Utils updates.

Re: How to run vison processing code on classmate during matches?
 

i installed the DS and now i have the acquire from axis IP button

Re: How to run vison processing code on classmate during matches?
 

So I want to process my images on the drivers station, but I can not finds out how add Lab View code to the drivers stations. In the CRIO code I have a Dashboard setup screen in Lab View but it does not have a place for the image to be displayed.

Re: How to run vison processing code on classmate during matches?
 

The driver station is not extensible, but the dashboard is. The default dashboard template has a vision display control tabbed with the Kinect display. If you want to process the image, you can do it directly in that loop. You need to choose a way to get your results back to the robot.

Greg McKaskle

Re: How to run vison processing code on classmate during matches?
 

Thank you the reply, I was able to find the details that I was missing, once again thank you.