Tracking Rectangles

[Greg McKaskle]

The framework examples do a bit of this already, but for a limited protocol.

If you drill into the dashboard code, you will find that the camera loop does TCP port 80 communications to the camera. The Kinect loop does UDP from a localhost Kinect Server, and even the other loop gets its data from a UDP port from the robot.

For the robot side, there are C++ classes for building up a LabVIEW binary type and submitting it for low or high priority user data. I'm not that familiar with other portions of the framework which may directly use UDP or TCP.

Greg McKaskle

[mikegrundvig]

The whitepaper is extremely useful but the part I needed help with is actually what's glossed over the most. My understanding is that it's fully possible to determine both angle and distance from the target by the skew of the rectangle and the size. Here is a quote from the whitepaper:

"Shown to the right, the contours are fit with lines, and with some work, it is possible to identify the shared points and reconstruct the quadrilateral and therefore the perspective rectangle"

Except it stops there. Have any other reading or direction you can send us to take this the rest of the way? I'd really like our bot to be able to find it's location on the floor with the vision targets and unless we are straight-on, this is going to require handling the angle. Thanks!

-Mike

[tomy]

But question is still how do you get the robot to track the rectangle like it would with a circle?

[JewishDan18]

Originally Posted by mikegrundvig

The whitepaper is extremely useful but the part I needed help with is actually what's glossed over the most. My understanding is that it's fully possible to determine both angle and distance from the target by the skew of the rectangle and the size. Here is a quote from the whitepaper:

"Shown to the right, the contours are fit with lines, and with some work, it is possible to identify the shared points and reconstruct the quadrilateral and therefore the perspective rectangle"

Except it stops there. Have any other reading or direction you can send us to take this the rest of the way? I'd really like our bot to be able to find it's location on the floor with the vision targets and unless we are straight-on, this is going to require handling the angle. Thanks!

-Mike

This varied depending on which language you use. If you aren't using Java, you should have access to a "convex hull" function, and a "find edge" function, which should do what you want. I haven't tested this yet, as I am using Java and do not have these functions. I'm working on getting them implemented in Java, but I have bigger fish to fry at the moment.

In theory, the bounding rectangle should be enough, if you put your camera as high as possible, and are willing to tolerate a little error. The height would tell you how far away you are, and the width, after accounting for the height, would tell you how far "off center" you are, giving you your position in polar form relative to the target. The error would be greater the further off center you are (since the perspective transformation makes the rectangle taller than it should be), but I would need to test to see if it is a significant amount.

[Greg McKaskle]

Originally Posted by mikegrundvig

The whitepaper is extremely useful ...

Except it stops there. Have any other reading or direction you can send us to take this the rest of the way? I'd really like our bot to be able to find it's location on the floor with the vision targets and unless we are straight-on, this is going to require handling the angle. Thanks!

-Mike

There are a number of approaches, but I'll show the one that I would use -- not in code, but as an example. I'm also making a few simplifications to get things started -- notably, I'm often assuming that the camera and target are in the same vertical plane.

1. I open up the image shown in the paper into Vision Assistant (the one with the perspective distortion).
2. Use the third tool, the Measure tool to determine the lengths of the left and right vertical edges of the reflective strip. I measure 100 pixels and 134 pixels.

First image shows the measurements in red and green.

Since the edges are different pixel sizes, they are clearly different distances from the camera, but in the real-world, both are 18" long. The image is 320x240 pixels in size.

The FOV height where the red and green lines are drawn are found using ...

240 / 100 x 18" -> 43.2" for green,
and
240 / 134 x 18" -> 32.2" for red.

These may seem odd at first, but it is stating that if a tape measure were in the photo where the green line is drawn, taped to the backboard, you would see that from top to bottom in the camera photo, 43.2 inches would be visible on the left/green side, and since the red is closer, only 32.2 inches would be visible.

Next find the distance to the lines using theta of 47 degrees for the M1011...
(43.2 / 2) / tan( theta / 2) -> 49.7"
and
(32.2 / 2 ) / tan( theta / 2) -> 37.0"

This says that if you were to stretch a tape measure from the camera lens to to green line, it would read 49.7 inches, and to the red line would read 37 inches.

These measurements form two edges of a triangle from the camera to the red line and from the camera to the green line, and the third is the width of the retro-reflective rectangle, or 24". Note that this is not typically a right triangle.

I think the next step would depend on how you intend to shoot. One team may want to solve for the center of the hoop, another may want to solve for the center of the rectangle.

If you would like to measure the angles of the rectangle described above, you may want to look up the law of cosines. It will allow you to solve for any of the unknown angles.

I'd encourage you to place yardsticks or tape measures on your backboard and walk to different locations on the field and capture photos through your camera. You can then do similar calculations by hand or with your program. You can then calculate many of the different unknown values and determine which are useful for determining a shooting solution.

As with the white paper, this is not intended to be a final solution, but a starting point. Feel free to ask followup questions or pose other approaches.

Greg McKaskle

[mikegrundvig]

This is incredibly helpful - I have no idea why the idea to use the camera as part of a triangle didn't come to mind but it was the key piece I was missing. Thanks much!

-Mike

[basicxman]

Originally Posted by Greg McKaskle

image shown in the paper into Vision Assistant

Would you be able to provide some raw images of the hoops through the Axis camera?

[Greg McKaskle]

Perhaps, but it is actually pretty easy to get your own.

If you have the camera plugged into the switch and set the camera IP to 10.te.am.11, the dashboard will save an image every second. Connect the ring light and walk around the target. The images will be saved into the user/documents/LabVIEW Data directory as a series of jpgs. You can also do this using the web browser or Vision Assistant, but you'll need to press a button each image and later save them.

Greg McKaskle

[basicxman]

Originally Posted by Greg McKaskle

Perhaps, but it is actually pretty easy to get your own.

If you have the camera plugged into the switch and set the camera IP to 10.te.am.11, the dashboard will save an image every second. Connect the ring light and walk around the target. The images will be saved into the user/documents/LabVIEW Data directory as a series of jpgs. You can also do this using the web browser or Vision Assistant, but you'll need to press a button each image and later save them.

Greg McKaskle

Aye, I was just wondering if you had some already as I don't have a camera and cRio until tomorrow

[shuhao]

Is using OpenCV (JavaCV) more feasible than using NI vision if you're not using LabView then?

Our team is also considering putting a netbook on the robot to do the image processing (gotta figure out 12 -> 18V)... Is that really worth the trouble? I don't know how to get a netbook to communicate with the cRIO yet other than with the driver station...

Any ideas/suggestions?

Thanks

[ProgrammerMatt]

Originally Posted by shuhao

Is using OpenCV (JavaCV) more feasible than using NI vision if you're not using LabView then?

Our team is also considering putting a netbook on the robot to do the image processing (gotta figure out 12 -> 18V)... Is that really worth the trouble? I don't know how to get a netbook to communicate with the cRIO yet other than with the driver station...

Any ideas/suggestions?

Thanks

You can have the netbook on the robot with the battery you dont need 12v to 18v.

and you can talk to the crio over a usb device such as an arduino or serial

[shuhao]

Is anyone else gonna be using OpenCV? (I, hopefully, will be able to use Python)

Also, what about the rule

Robots must be controlled via one programmable National Instruments cRIO (part # cRIO-FRC or cRIO-FRCII), with image version FRC_2012_v43. Other controllers shall not be used.

Does a netbook count as another "controller"?

[JewishDan18]

Originally Posted by shuhao

Is using OpenCV (JavaCV) more feasible than using NI vision if you're not using LabView then?

Our team is also considering putting a netbook on the robot to do the image processing (gotta figure out 12 -> 18V)... Is that really worth the trouble? I don't know how to get a netbook to communicate with the cRIO yet other than with the driver station...

Any ideas/suggestions?

Thanks

Personally, I don't think so, but that is for your team to judge. I have no facts to back this up, but I would bet money that the NI Vision is optimized pretty well for the CRIO and will give you better performance. Just the time required to convert the image to openCV format will be huge, since it is stored as a pointer to a block of memory containing a JPEG image (which you have to decompress to get individual pixel values from, so I'm not sure it would be feasible at all).

As far as the laptop goes, I would run the code on the driverstation, then send the results back rather than attach a laptop to the robot. Might be slightly slower, but a much smaller chance of getting destroyed

[yash101]

I have wanted to design a vision system to work like that and calculate the distance from the target without range sensors or any other sensors. I also wanted to skip the Kinect because of how hard it is to interface to the robot, and it's slow speed. This is exactly the routine that I wanted to do. Now, I know how to implement it. Thank You!
Also, if I am not wrong, does it follow the laws of perspective that explain how an image looks smaller as it is farther away from your eyes, in this case, the camera.

Here's and O: O
Look at it up close. doesn't it look large?
Now look at it five feet away. It should look much smaller now.
If I am not wrong, I think that is how this is supposed to work!

[faust1706]

Originally Posted by yash101

I have wanted to design a vision system to work like that and calculate the distance from the target without range sensors or any other sensors. I also wanted to skip the Kinect because of how hard it is to interface to the robot, and it's slow speed. This is exactly the routine that I wanted to do. Now, I know how to implement it. Thank You!
Also, if I am not wrong, does it follow the laws of perspective that explain how an image looks smaller as it is farther away from your eyes, in this case, the camera.

Here's and O: O
Look at it up close. doesn't it look large?
Now look at it five feet away. It should look much smaller now.
If I am not wrong, I think that is how this is supposed to work!

is it still too late to contribute? I've been teaching those interested in my team during this preseason computer vision. I posted a white paper on here describing my methods of using camera pose estimation for rebound rumble (note, this is very complicated mathematics, I don't recommend it unless you are up for a big challenge) Pose could have been used for this year, too, but that pesky pyramid, so basic trig sufficed. I'm in the process of writing a scholarly (I guess you'd call it) paper describing my program from ultimate ascent (for the purpose of submitting it to our the missouri symposium and have it compete at Intel ISEF and ISWEEEP, the kid who won isef was on daily show last night) You can view our vision system set up from our website if you like. *we used a single board computer, O-Droid X2. It is our toy now essentially. So much fun to play with.

Anyways. Is anyone doing anything with vision this preseason? I've been working with some professors at wash u, Missouri s&t, and harvey mudd to make a camera pose estimation with a known rotation matrix. Turned out to be a lot more math intensive than the 3 of us first thought...the program will be done before build season and I'll have it up online somewhere so whoever is welcome can look at it. I'll make it as....educational as I can with comments, but comments can only do so much (which turns out to be a lot). If it is another game where there is reflective tape that can be used to assist in scoring, which there has been every year I've been in FIRST (starting with logomotion), then I'll put up a working vision code that returns distance on the x-z plane and x-rotation to the center of the target at a reasonable time during build season.