1706 Vision Solution: Tracking totes in depth-map, color, and infrared

[yara92]

hey everyone,

i have some qustion about using the knicet in auto ,but we can't figure if it is supported by the new wpi code (c++)

[Christopher149]

Originally Posted by yara92

hey everyone,

i have some qustion about using the knicet in auto ,but we can't figure if it is supported by the new wpi code (c++)

On the robot, or the driver station? Because if driver station, keep in mind rule G21:

During AUTO, DRIVE TEAMS must not directly or indirectly interact with ROBOTS or OPERATOR CONSOLES.
VIOLATION: FOUL and YELLOW CARD

FIRST salutes the creative and innovative ways in which Teams have interacted with their ROBOTS during AUTO in previous seasons, making the AUTO period more of a hybrid period due to indirect interaction with the OPERATOR CONSOLE. The RECYCLE RUSH AUTO Period, however, is meant to be truly autonomous and ROBOT or OPERATOR CONSOLE interaction (such as through webcam or Kinect™) are prohibited.

emphasis mine

[yara92]

Originally Posted by Christopher149

On the robot, or the driver station? Because if driver station, keep in mind rule G21:

emphasis mine

we mean to use the Kinect as 3D camera

[faust1706]

you have to utilize the libfreenect library in order to interface with the kinect. If you go to our code, cmastudios made grabbing the rgb, ir, and depth map from the kinect into one line of code.

https://github.com/rr1706/vision2015/tree/master/lib the files are free.cpp and free.hpp.

The rgb image is obviously in color (RGB), the ir is in grayscale, and the depth is an interesting type of image, it is grayscale where pixel values are a representation of depth.

[RyanShoff]

Thanks for sharing this. I did get your solution working on a Nvidia Jetson board yesterday.

[faust1706]

I have been stuck on a time consuming project in the lab so I haven't had time to configure it for the jetson board laying on the table next to me. I just stare at it with desire....

How'd it do? On the odroid, we get manageable fps, but it is rather laggy compared to our vision programs in the past that were hitting 30fps.

[RyanShoff]

Adding the profile times in robot.log gives me an average frame rate of 3-4 fps. This is with X running and both color and ir maps displaying on screen. There is a slight lag when putting your hand quickly in front on the camera. It probably is way less than a quarter second. Only two cores are being used.

It is much less laggy that anything I've been able to do with libpcl on the jetson.

With a little work, I think it could work for autonomous navigation.

[RyanShoff]

I didn't understand the profile times in the post above. With image display now turned off, I'm now seeing 20 fps.

[faust1706]

Image display is a rather computationally intensive task, to many people's surprise. I expected a decent jump in fps, but not that much. That is really encouraging to see, actually. Thank you so much for doing this.

[faust1706]

Originally Posted by RyanShoff

It is much less laggy that anything I've been able to do with libpcl on the jetson.

With a little work, I think it could work for autonomous navigation.

I have never written a vision program with a ton of lag. The most I personally witnessed first hand with 1706's vision solution in 2014. It utilized 3 cameras and solved for the robot's position and orientation on the field. It dedicated a single core for the entirety of processing of one of the cameras (so 3 of the 4 were used to process images). It had about a half second of display. I forget the exact amount. We tested lag by placing a stop watching in front of the camera, then took pictures of the vision output along with the stop watch then simply subtracted the times to find the lag.

If you want to, go for it. @cmastudios informed me that they are now using vision in autonomous, which is exciting. I wrote matlab code that is a basic implementation of a* in 2d. cmastudios converted it to c++, then I changed his c++ code to a custom path finding algorithm that takes robot width into consideration. The custom path finding algorithm is being used currently on a robotics team at MST.

There is a step missing between vision output and input to path finding: changing to D.S (data structure) of the vision output to fit that D.S that a * can operate on. Usually it is simply a list of points in a finite, discrete, grid that are deemed in-transversible (obstacles). You cannot simply pass the centers of all detected object to a astar due to the object (in this case totes), having a decent amount of width and length.

A big problem with converting from vision to path finding is precision. Yes, you can return every pixel that is an obstacle, but then your grid is extremely discrete and path finding is O(nlogn) if I remember.

cma utilized the gnu optimizer when we were toying with the idea of a* this past summer and he got a 900x900 grid to be solved in about 1ms, I forget the exact time, on a decent laptop.