M$ Kinect controlling robots!

[basicxman]

So we've all seen Wiimotes and Xbox controllers as alternatives to joysticks, and probably most of us know that Microsoft just released their Kinect system.

Adafruit (probably heard of them too ) offered a $2000 dollar bounty for the developer of the first open source driver for Kinect. Well, although M$ made it difficult, somebody did it! I can't wait to see one of these on an FRC system, even if it's not for a competition.

http://groups.google.com/group/openkinect/?pli=1

Not sure if they've actually been released in the public yet due to M$ being control freaks...time will tell.

[basicxman]

Spoke too soon, there's a git repo:

http://git.marcansoft.com/?p=libfreenect.git

[Vikesrock]

It's difficult to tell from your post if you are aware of how the Kinect system works and what the open source driver can and can't do.

The Kinect hardware provides a 640x480 RGB image and a 320x240 monochrome depth image. It also has a motorized pivot and multi-array microphone. All of the gesture recognition is done in software that runs on the 360.

The open source driver currently in the wild only covers the two video streams so far from what I have read.

Using this device to control a robot would require writing a pretty hefty amount of image processing code to do the gesture recognition. Using it as a sensor for a robot on the other hand.....

[synth3tk]

As Vikesrock mentioned, this would be akin to buying a standard Logitech webcam and writing recognition software for that. There's nothing special per-se about having the driver with the intent of using it in the same way it's used for it's purpose on the 360.

As for using this on the robot, that's an interesting idea...

[Jared Russell]

Originally Posted by synth3tk

As Vikesrock mentioned, this would be akin to buying a standard Logitech webcam and writing recognition software for that. There's nothing special per-se about having the driver with the intent of using it in the same way it's used for it's purpose on the 360.

As for using this on the robot, that's an interesting idea...

The depth measurement (a time-of-flight LIDAR array cleverly built into the CMOS camera) is responsible for much of the capability of the Kinect (using depth to segment you from your surroundings is much faster and more robust than doing it with color/intensity via the RGB camera - otherwise Kinect would fail if you were wearing a white shirt in a white room, for example).

I, personally, am very excited by the commercialization of low-cost depth imagers (Kinect is the most conspicuous example, but several companies and universities around the world are making headway as well). Active depth sensors were the reason why many DARPA Grand and Urban Challenge vehicles were able to succeed - but sensors like the ubiquitous Velodyne HDLs used by many teams run about $75,000:



Sure, Kinect's sensor doesn't offer nearly the range, accuracy, or resolution - but for $150, it certainly would be more than good enough on a FIRST robot!

[JesseK]

So what I gather from these posts is the Kinect lies somewhere between an ultrasonic sensor and a full LIDAR array in terms of price and capability. Fun.

Given that there's a very confined space to drive any VRC, FTC, or FRC robot on a competition field, I don't forsee Kinect revolutionizing the driver's station any time soon.

[biojae]

Originally Posted by JesseK

Given that there's a very confined space to drive any VRC, FTC, or FRC robot on a competition field, I don't forsee
Kinect revolutionizing the driver's station any time soon.

Who says it has to be on the robot?
Why not use it as it was designed, and have your drivers be the controllers?

[basicxman]

Originally Posted by biojae

Who says it has to be on the robot?
Why not use it as it was designed, and have your drivers be the controllers?

That's exactly what I was saying; I realize that it is probably not FIRST-competition reliable but would still be cool in other environments.

[JesseK]

Originally Posted by biojae

Who says it has to be on the robot?
Why not use it as it was designed, and have your drivers be the controllers?

Originally Posted by basicxman

That's exactly what I was saying; I realize that it is probably not FIRST-competition reliable but would still be cool in other environments.

Originally Posted by Wikipedia

The Kinect sense has a practical ranging limit of 1.2–3.5 metres (3.9–11 ft) distance

Standing far enough back from the controls for Kinect to properly sensor a driver means that driver has to stand at the very rear of the driver's station. Additionally, both the Driver Coach and the other Driver have to make sure they do not interfere with Kinect's imaging. Finally, every couple of years the human players are also confined to the driver's station; thus any of the HP's movement would have to keep from interfering with the Kinect as well.

By all means file it under 'neat' and 'innovative' -- so long as it's not within a competition robot driver controls context.

[Mark McLeod]

Using it on the driver station would also seem to be in technical violation of the Autonomous rules, where you would be able to use the controls from behind the autonomous driver line.

[davidthefat]

https://github.com/OpenKinect/libfreenect

Thats the working link.

Now how would we change the motors to legal ones?

[NickE]

Originally Posted by Mark McLeod

Using it on the driver station would also seem to be in technical violation of the Autonomous rules, where you would be able to use the controls from behind the autonomous driver line.

In theory, the coach could also control the robot, which could be an issue.

[Tom Bottiglieri]

Originally Posted by Jared341

The depth measurement (a time-of-flight LIDAR array cleverly built into the CMOS camera) is responsible for much of the capability of the Kinect (using depth to segment you from your surroundings is much faster and more robust than doing it with color/intensity via the RGB camera

I'd like to see how the IR time of flight measuring works in noisy environments (like say, outside or under stage lights). I'll have to get my hands on some hardware pretty soon.

Also, if the frame rate is decent enough, you may be able to spin this thing on a vertical axis for Velodyne type readings. We did this with single plane LIDARs (this project: MIT CSAIL Autonomous Forklift), and the results were pretty good.

The cool thing about the depth sensor here is most CV algorithms (edge detectors, feature finders), should just work.

[Vikesrock]

Originally Posted by Tom Bottiglieri

I'd like to see how the IR time of flight measuring works in noisy environments (like say, outside or under stage lights). I'll have to get my hands on some hardware pretty soon.

Also, if the frame rate is decent enough, you may be able to spin this thing on a vertical axis for Velodyne type readings. We did this with single plane LIDARs (this project: MIT CSAIL Autonomous Forklift), and the results were pretty good.

The cool thing about the depth sensor here is most CV algorithms (edge detectors, feature finders), should just work.

Frame rate is 30 fps according to the technical specs that I have seen floating around.

I'm also very curious as to how it would perform outside. The Microsoft documentation specifically mentions making sure there is no direct sunlight on you or the Kinect when you are using it so there is probably at least some performance degradation.

[jason_zielke]

Originally Posted by Jared341

The depth measurement (a time-of-flight LIDAR array cleverly built into the CMOS camera) is responsible for much of the capability of the Kinect

Did you find an explanation somewhere that lead you to believe the Kinetic uses IR time of flight? That approach is generally very expensive but has been done in the approach you are suggesting in the SwissRanger . . . http://www.acroname.com/robotics/par...0-10M-ETH.html

However, given the price point of the Kinect, I am guessing they are using a triangulation approach similar to that described here: http://en.wikipedia.org/wiki/3D_scanner#Triangulation

This approach is also often referred to as Structured Laser Light Ranging (try a google search - lots of research in this area).

In either case, I believe the FRC rules would have to change to allow the device to be used on the robot because it uses an "exposed laser". This rule is likely in place for eye safety reasons, but I would love the opportunity for our team to be able to use a ranging device like this on the robot. It opens up all kinds of advanced opportunities for robot intelligence and would put students in the thick of developing cutting edge software.