Tracking Rectangles with Perspective Distortion

[wre136]

Hi,

I’m extremely new to vision tacking. I’m taking several pictures of an Ultimate Ascent rectangle goal (It matches the 2 point goal but is 60% the size due to lack of material) from different angles.

I’m trying to use the Vision Assistant and its shape matching function to find the shapes but it will only work if I am look straight at it and not at an angle.

I can use edge detection to detect the left, right, top and bottom edges of the rectangle and use the caliper function to find lines of intersection, angles and lengths which is very effective for giving me the info to determine distance and what not. However, then I run into the issue with having more than 1 rectangle on the screen and now edge detection no longer works.

I am trying to look into the “Vision Targets White Paper” (See attached document) and its section where it talks about Perspective Distortion (page 11). It shows where due to it, we cannot find shapes. But then by using “some work”, we can detect shapes with the quadrilaterals. I’m looking for help on what this work is.

I’ve tried looking into Contour Analysis to try and get something I can use to match what the doc is doing but I am not making any ground on it.

I’m looking for any help on how to deal with this distortion issue or Vision Tracking in general with Vision Assistant. Thanks!

[Greg_McKaskle]

Can you post an image you are trying to process? It would help to know how much distortion we are talking about.

Greg McKaskle

[wre136]

I’ve attached the 5 pictures I am trying to use to do this with. Thanks!

[Alan_Anderson]

I wouldn’t worry about trying to make your code deal with the fifth image. A robot can’t be high enough to get that view.

[Greg_McKaskle]

Assuming that the camera is an M1011, the default code described in the white paper gives a distance range of 12.0 to 9.9

Picture 1:
95, 96, 99, 68, 98 and distance of 11.5

Picture 2:
83, 93, 94, 59, 88 and distance of 10.2

Picture 3:
74, 96, 89, 53, 79 and distance of 9.9

Picture 4:
91, 95, 93, 77, 85 and distance of 12.0

As mentioned, Picture 5 has the camera above the image, but for grins, it is categorized as a center target scoring 86, 97, 85, 96, 57 and distance of 9.5.

This doesn’t seem bad for a quick particle approach. In order to get the mask to work well, I changed to monochrome and chose Luminance as the plane and set the minimum value to about 220. If you intend to use color masking, you should set the exposure much lower so that your colors are more saturated. The tape is practically white.

The question is, do you need better distance calculations than that? You can get fancier with line fits or angle measurements, and let me know if you decide to.

Greg McKaskle

[wre136]

Greg,

I really appreciate the help you are giving me.

So two things. How did you detect the rectangles even though in some of the pictures they were not rectangles? I can get some of the same results as you but I have to use edge detection, points of intersections and then I can measure angles and distance. However, that approach fails when I introduce multiple rectangles into the picture. My goal would be to measure angles and lengths or multiple rectangles even if I’m looking at them at some extreme angle.

Did you use the simple Shape Detection feature or did you use something else to detect the distorted rectangles? If you used something else or somehow tweeked the shape detector feature, it would be great if you could direct me to what settings I can play with to detect distorted rectangles. Thanks again for the quick response!

[Greg_McKaskle]

I used the approach that was used for 2012 and the white paper. The approach is to inspect each particle to see if it is a good hollow rectangle. It does this by calculating a handful of simple measurements and scoring them to how a rectangle would score. The last step is to compare the scores to thresholds to determine how non rectangular you will allow the particles to be.

I’d suggest you compare your approach to the ones in the paper and identify the differences, the shortcuts, and things you’d like to modify.

If you have any questions, ask away.

Greg McKaskle

[JewishDan18]

A quick (and very dirty) way to do it is to fit to parallelograms, and use the difference between the parallelogram and a rectangle with the same height and a fixed width that is dependent on the height (since you know the aspect ratio you are looking for in a target) to solve for additional information.

[wre136]

Greg,

The white paper I’ve been using so far is the one you are talking about. It has the info you were talking about with scoring rectangles.

However, my next question is how can I change those factors? I assume we are using the Shape Detection feature under Machine Vision and it does not give me the ability to change any of the factors I am reading about. If that isn’t the case, please correct me.

[wre136]

@JewishDan18

A quick (and very dirty) way to do it is to fit to parallelograms, and use the difference between the parallelogram and a rectangle with the same height and a fixed width that is dependent on the height (since you know the aspect ratio you are looking for in a target) to solve for additional information.

JewishDan18,

How do you fit the image to parallelograms? Is that some in Shape Detection or something else?

[Greg_McKaskle]

It intentionally doesn’t use high level shape detection so that it is more accessible.
The rectangularity score is based on area/bounding rectangle.

The aspect ratio is based on the width and height of the bounding rect, and to make it a bit more robust to distortion, it also uses something called the equivalent rectangle.
The equivalent rectangle uses the area of the particle and the perimeter of the particle and solves for 2X+2Y=perimeter and X*Y=area.

The hollowness counts the pixels that are on for each vertical column of pixels and again for each horizontal row of pixels and compares those counts to thresholds that expect a strong outer band and weak inner band.

Each of these is scaled so that 100 is a good score and lower is not as good. The initial cutoffs are just based on some initial images and are very easy to change.

Can you think of other simple geometric measures that the code could score on the targets?

Greg McKaskle

[wre136]

@Greg McKaskle

It intentionally doesn’t use high level shape detection so that it is more accessible.
The rectangularity score is based on area/bounding rectangle.

The aspect ratio is based on the width and height of the bounding rect, and to make it a bit more robust to distortion, it also uses something called the equivalent rectangle.
The equivalent rectangle uses the area of the particle and the perimeter of the particle and solves for 2X+2Y=perimeter and X*Y=area.

The hollowness counts the pixels that are on for each vertical column of pixels and again for each horizontal row of pixels and compares those counts to thresholds that expect a strong outer band and weak inner band.

Each of these is scaled so that 100 is a good score and lower is not as good. The initial cutoffs are just based on some initial images and are very easy to change.

Can you think of other simple geometric measures that the code could score on the targets?

Greg McKaskle

Greg,

What I would like to work out is some system that detects all edges on the image and if the edges seem to touch (or are with in a few pixels of each other), then they would be considered a corner. If 4 corners could be found consecutively, then we have a quadrilateral which is the shape we want to analyze.

The only thing that stinks about even trying to start my approach is that the edge finding ability of the vision assistant can only find 1 edge per run and you can only run it in the 4 main directions, left-right, right-left, up-down, down-up. Once each of those finds an edge, I cannot detect anymore edges unless I can find a way to subtract those edges from the image and then keep rerunning the edge detection. If I could at least do that, then I could detect the shapes and do the same measurements and calculations we are now. If you can think of anything about this, that would be helpful.

I do have a question about edge detection though. Does it work off of how many pixels in a column report an edge? For example, if all the pixels in column X say they have found an edge, then is the edge found? If some pixels in columns X, Y, Z (columns that are right next to each other) report an edge, then would that be an edge but be partially diagonal? If that is the case, I wonder if and how we can play with that so we can detect edges that are really skewed. Just thinking out loud.

Again, thanks for the help!

[Greg_McKaskle]

I opened the NI Example Finder, under the Help menu, and searched for Straight Edge. If you set the Number of lines to 2 and drag a rectangle around the image, it will locate both the top and bottom lines. You most likely want to use the mechanism in the white paper to find the bounding box of particles, then use the straight edge to get precise fits to the edges.

Again, this will be a bit tedious, your original image had stronger edges between the board and the background than the tape and board. So without a good bounding rectangle, the edge detection may do what you ask and not what you want.

Contours are another way to go with this. The attached image shows the contour and the graph to the right shows the sharp corner transition. You may want to use the bounding box to set the ROI to where you expect each corner to be and process then one at a time, and then recombine.

Greg McKaskle

[faust1706]

sorry about being so late on this reply:

@Greg McKaskle

.
The aspect ratio is based on the width and height of the bounding rect, and to make it a bit more robust to distortion, it also uses something called the equivalent rectangle.

How I went about this issue was also incorporating an aspect ratio. I took my 4 corners, 0, 1, 2, 3 which correspond to top left, top right, bottom left, and bottom right, respectively. I did (x^2 + y^2)^.5 to get distance between two points. Just the simple distance equation where x and y are pixel coordinates. I did that for the 4 sides. Then I added the top and bottom distances and divided them by the sum of the right and left.

If your program relies on being perpendicular to the target, meaning straight in front of it, then I’d suggest using image moments. Perspective images hardly alter the center of the target with this method.

What I did with this aspect ratio is differentiate between the 2 and 3 point target. The longer and thinner the target, the higher my aspect ratio. I created a simulation program that simulates the field (might I add with pyramid that I will be uploading on here probably tomorrow). I did this to easily decide what a good lower limit of the aspect ration is. I discovered 2.4 was the magic number. I also added an upper limit to the aspect ratio, which turned out to be 4. If the target I find does not have an aspect ration between these two values, I call it a two point then send the driver the image moment’s x rotation so the robot may turn towards the 3 point, even if the camera is unable to detect it.