Need help and suggestions for image comparison method

[jee7s]

What I'm about to say here may complicate your problem. But, I know you are dealing with a difficult problem and I'm assuming you have some very good professional help, so I thought I would add a few comments on what you may be missing. You probably don't want to do this now, but take notes in case you want to do this as a later refinement.

All that is mentioned here regarding the image differencing is quite valid. However, your problem where the image sizes are different is easily solved by padding the images with black pixels so they are the same size. This has some other implications to your math, but it would normalize the images and remove your concerns about area.

However, I think you are focusing on diff-ing the images while missing your real problem: position tracking. While you can use the motion of the floor as an indicator of motion, there's still going to be errors and noise, etc. Theoretically, you could image the entire floor of the pool, then use the image you capture and do a correlation over the whole pool image to find your location. But, that's a huge amount of time, and a huge amount of data to manage, and you probably can't do it on the hardware that's been mentioned.

To Greg's point above and Jared's in the Optical Flow thread, you basically do feature extraction, establish geometries between the features, and determine the difference. That tells you how much the camera moved. The SURF algorithm that Jared mentions is a very robust generalized feature extractor that usually gives useful information. It's just a matter of deciding what of the information that SURF outputs is what you want. Once you do, you'll have the pattern of features. Then, you need to determine how to find the pattern (that's highly application dependent...but could be as simple as a Hough Transform if you are lucky), and determine how far it moved. So long as you don't move too much from frame to frame (like the square-in-the-image concept that Greg mentioned), you can determine the shift of the object in the field of view.

But, you aren't dealing with a camera watching a moving object, you're dealing with a camera ON a moving object. So, you get more information than just the motion of the scene. If you put an accelerometer/gyro combination on the sub, you get an indication of the motion of the craft. Jared suggests this as a means to do dead reckoning, but there's a nuance to it: you have a second sensor. Now, it's not dead reckoning, it's sensor fusion as Gdeaver alludes to in the optical flow thread.

To apply this, you need to use a great but quite advanced tool called a Kalman Filter (yes, I've got other CD posts on this from a couple years back saying it's overkill for FRC...but your problem isn't FRC). It's an algorithm to estimate values in noisy environment. It's used in a lot of things like airplanes, ships, and spacecraft.

Basically, taking your differencing problem, you are going to establish a velocity vector at the end of it. Why? because you know the time between frames, you know the number of pixels the image features moved, you know the size of the pixels from the sensor's datasheet, you know the magnification of the camera's lens, and you know the distance to the floor of the pool. Grind through the math and you'll turn the i,j motion in pixels to an x,y change in position over a known time and bingo...you've got velocity. This is pool-centric velocity, since you are basically measuring the pool.

But, you can take it a step further and do the dead reckoning that Jared describes by getting an acceleration vector from the accelerometer/gyro combo. Taking the vector the accelerometer gives you and rotating it using the gyro rotation vectors you translate craft-centric acceleration to a pool-centric acceleration. Then, if you integrate twice you get position. The problem again is that this is noisy. Well, you've got another measurement: the velocity vector from the pool. So, use that as the "measurement" part of the KF, use the dead reckoning as the "system" part of the KF and now you've got a Kalman Filter tracking estimate of position. It won't be perfect, but it should be less noisy and more accurate than either the dead reckoning or optical flow alone.

Think of it this way: in the short term (on the order of your frame rate plus processing time), the dead reckoning estimates position. But, dead reckoning errors increase as you use them for longer and longer. So, your optical flow/image information "resets" the dead reckoning error back to a small number to compensate. That's not exactly how the KF works, but it's a good conceptual picture of what's going on.

Again, I may have just complicated your problem by bringing this up. But, if you have experienced professionals helping you, then you may get to the point where you want to do something sophisticated like this. Yes, it's pretty advanced knowledge, but it's also a very robust method.

[Gdeaver]

A sub in water is similar to the navigation of a quadcopter. Allot of work has been acomplished in the Quad comunity. http://diydrones.com/ is a good source of info. Go to the DIY DRONES store page look at sensors and you will find this. http://store.3drobotics.com/products/px4flow
Follow the link to the open source code on Github. There you will find all the code needed for the type of optic flow algorithm that you are looking for. The meat is at https://github.com/PX4/Flow/blob/master/src/flow.c
Notice in this code they take the gyro and correct for body rotation. In my other post I mentioned the problem of 2 solutions to a set of x and y's. Look at how they handle this. The algorithm used may not be the best as it is getting a dense feature matrix from the camera in a quad app. As Jared mentioned your pool bottom is most likely going to give a sparse feature Matrix. Have you shared some of your pool bottom video? Could you post a link to some? People on this forum could help if they saw what you have to work with. This is all dealing with a captured digitized frame. To get really good results you may need to clean the image up optically first. A polarizing filter comes to mind but camera people have a whole arsenal of devices to play with optics. I don't know were to get a hold of this knowledge. Today every one wants to do the clean up digitally. It is not always the best way.

All this is not going to work if your sub platform is not stable. I mean rock solid stable. If the the sub is oscillating pitching and rolling the errors will accumulate real fast. How stable is this years sub? When it is accelerated do you get rotation? Physically the sub has to be rock solid and this comes from sound physical design. The mechanical team has to get this right or all your work will only lead to frustration.

Quaternion. You will have to learn how to work with rotations using them and as mention some of the best sensor fusion algorithms use extended state kalman filters. Allot of math to master. This is not easy but if you master it save it for your senior college project. Its at that level or high .