Kinect LabView Drivers

[Ryan Gordon]

Hello all,

Based on the work of the OpenKinect (http://openkinect.org) initiative, I've been able to begin to write drivers in LabView to interface with the Kinect.

For now I've got the following implemented:

Control LED Color
Control Motor Position (between +/- 31 degree)
View Servo Position
View Servo Speed
View Servo Status (Stopped/Reached Limits/Moving)
View Accelerometer data (ux,uy,uz)


Todo:

Turn on Depth & RGB Camera
Retrieve Depth & RGB Data
Retrieve Audio Data


LabView doesn't support isochronous USB transfer so that will make it harder to interface with the camera and audio data but not impossible.

I'll have source code and instructions available as soon as it I get the camera turned on and retrieving data from them. I'm hoping that this can be a learning experience and a research experience for everyone. It would be awesome to see some old FIRST robots lying around from previous competitions become a lot more autonomous with this technology.

I could see robots that could drive themselves around objects given a desired route, map an entire room, use facial recognition and voice commands to carry out tasks by specific individuals. I'm sure there's lots more useful applications for this type of technology.

All of the information and research that has gone into making this possible and more can be found at http://openkinect.org/

Source code soon!

Ryan

[Andrew Schreiber]

How are you connecting the Kinect to the CRIO? I was going to work in a similar project but didn't want to commit funds until I figured out how to connect it.

[Ryan Gordon]

It's actually just USB to my laptop at the moment, sorry I didn't make that clear beforehand. I don't have a cRIO available to me yet at the moment so I haven't had a chance to investigate that.

However, the USB interface is seriously complex, especially with the fact that we have to deal with isochronous transfers for video and audio. It might be possible to strip the USB and power cables and connect the data wires to the DIO board and the power distribution board respectively.

Just to note, the newer cRIO's have a USB interface port.

[Andrew Schreiber]

Just to note, the newer cRIO's have a USB interface port.

They do? The NI 9074 I have at work (purchased less than 3 months ago) does not...

[Ryan Gordon]

Check this one out: http://tinyurl.com/2bjsqx7

Either way we should look for a solution with the DIO board since that will probably be the most natural interface to work with when wiring up the Kinect. It would actually probably be pretty easy to put together a tutorial on creating a male usb to the individual power wire and the I/O wires that can be connected to the Kinect. Could be as simple as buying the Kinect legacy extension from Fry's or the like, and cutting off the ends of the wires, stripping them, and wiring them appropriately.

[Greg McKaskle]

On the USB subject, the cRIO family has lots of variety, and some have a USB port. Last I checked, the USB stack exclusively supported the storage protocol. It was used to provide a file storage interface to a USB stick.

True, drivers for other device types could be written, but doing so for VXWorks and keeping the RT aspects in mind are not so easy, so the other device types will not be supported unless there is a good reason for NI to purchase or build the necessary drivers.

I wouldn't really worry about the USB issue though. Exploring and learning about the kinect using a laptop sounds like an incredibly beneficial journey. Please keep it up. And if you are interested in sharing your results, please consider also posting them to NIWeb as you may get a good number of power users interested.

Greg McKaskle

[Vikesrock]

Check this one out: http://tinyurl.com/2bjsqx7

Either way we should look for a solution with the DIO board since that will probably be the most natural interface to work with when wiring up the Kinect. It would actually probably be pretty easy to put together a tutorial on creating a male usb to the individual power wire and the I/O wires that can be connected to the Kinect. Could be as simple as buying the Kinect legacy extension from Fry's or the like, and cutting off the ends of the wires, stripping them, and wiring them appropriately.

The problem is not in getting the Kinect physically wired to the cRIO, the problem is that the FRC cRIO does not have a USB Host Controller.

I could see someone getting a Kinect running on a robot using a mini-ITX motherboard with a SSD drive. Another option would be to get it running on a microcontroller. I will likely be working on using the Kinect with a Luminary Micro 3000 series chip at work sometime in the next few months, but I'm not certain I will have anything there before kickoff.

[Ryan Gordon]

On the USB subject, the cRIO family has lots of variety, and some have a USB port. Last I checked, the USB stack exclusively supported the storage protocol. It was used to provide a file storage interface to a USB stick.

True, drivers for other device types could be written, but doing so for VXWorks and keeping the RT aspects in mind are not so easy, so the other device types will not be supported unless there is a good reason for NI to purchase or build the necessary drivers.

I wouldn't really worry about the USB issue though. Exploring and learning about the kinect using a laptop sounds like an incredibly beneficial journey. Please keep it up. And if you are interested in sharing your results, please consider also posting them to NIWeb as you may get a good number of power users interested.

Greg McKaskle

Ah, I did not consider that. Yes, it would be difficult to interface with the Kinect without a host USB controller.

I've attached my source code here even though it doesn't have depth or camera data working yet. I've certainly learned a lot from this experience, it's been really rewarding.

I'll probably end up changing it all up to use a dll generated from the https://github.com/OpenKinect/libfreenect/ library. LabView doesn't have isochronous USB support so that will be necessary in the end anyway but I wanted to see how far I could get with the raw VISA interface.

To setup the Kinect to your PC using the labview libraries you'll need to use the VISA Driver Wizard to create a NI-VISA driver for the Xbox NUI Motor and Xbux NUI Camera. Audio isn't necessary since at the moment it hasn't been reverse engineered yet.

Once that's been done it's as simple as open up core.vi and running it. If the Kinect is working and the driver was setup right it will autodetect the usb port it's running on and initialize the motor and accelerometer, etc.

http://ryangordon.net/libfreenect.zip

Uploading isn't working here for some reason.

[ygd]

You could theoretically program the FPGA on the cRIO to act as a USB host, and from there, you could control the Kinect. I haven't actually tried it, but it seams feasable.

[jhersh]

You could theoretically program the FPGA on the cRIO to act as a USB host, and from there, you could control the Kinect. I haven't actually tried it, but it seams feasable.

The FPGA could theoretically implement the host controller features, but you would still need a USB phy (physical layer interface) to host the USB electrically. Also, the host controller is terribly complex and not really a reasonable engineering approach. You would be far better off using a device that already has a USB host controller (such as the cRIO 9022 mentioned earler, a laptop, a TI Beagle Board, or even a Luminary (now TI) Stellaris part with USB OTG support).

[Ryan Gordon]

http://ryangordon.net/kinect_labview_small.png

I've got the LabView drivers fully working but still not completely stable yet. The dlls that run behind the scenes still need a lot of polishing but this is a great start so far. The next big challenge will be to find a reliable/easy way to get the Kinect to run through the cRIO as we wait for the dlls to stabilize.

I'll hopefully have the development source code up in my github repository within the next weeks.

Cheers,
Ryan

[Gdeaver]

Aside from getting the data into the c-rio, there is the raw computational power of doing anything useful with the data. Wouldn't a better strategy be getting the data back to the driver station lap top and do the processing there?
A 400 mhz power pc vs. a dual core x86? The c-rio could be by passed completely. A usb host controller interfaces to the kinect and then sends it by lan to the gaming router back to the laptop. First would have to change the
ruling on no lasers. I believe the Kinect has a true infrared laser. The kinect could have tremendous affect on our robots in the future. You may want to keep an eye on Microsoft's robotic studio. Word has it they may be releasing some drivers.

[Ryan Gordon]

Aside from getting the data into the c-rio, there is the raw computational power of doing anything useful with the data. Wouldn't a better strategy be getting the data back to the driver station lap top and do the processing there?
A 400 mhz power pc vs. a dual core x86? The c-rio could be by passed completely. A usb host controller interfaces to the kinect and then sends it by lan to the gaming router back to the laptop. First would have to change the
ruling on no lasers. I believe the Kinect has a true infrared laser. The kinect could have tremendous affect on our robots in the future. You may want to keep an eye on Microsoft's robotic studio. Word has it they may be releasing some drivers.

This wasn't ever meant to be used during competition in FIRST ;) The laser is a Level 1 Class laser but it's perfectly safe as there is a million safety features that PrimeSense and Microsoft built in to make sure it stays safe.

But the showstopper here is that there is too much interference with multiple Kinect's moving around. I'm just building this so I can make a robot do things by itself autonomously; The idea of true autonomy is very poorly implemented in FIRST, mainly because the complexity and lack of technology to make it more plausible. I quite think this would help others interested in exploring truly autonomous robots. Every year that passes by goes more decommissioned pieces of metal that never get put into competition again and I quite think this would make for some interesting projects with those old robots. That is certainly what I'd like to do off-season with them.

[Brandon_L]

I am EXTREMELY interested in this. Do you plan on releasing it when its done?

[SuperS_5]

Hi, this has grabbed my interest. I believe that LabVIEW does support isochronous connections. Although I could only find documentation on Firewire and isochronous data transfer, I have successfully used every USB webcam that I have tried with LabVIEW. The IMAQ USB drivers have to be installed. They are available to download from NI, but, I am unsure of any licensing. (I think only the NI Vision module has to be activated, not the drivers) If the device can be enumerated via USB devices, it should work via IMAQ USB capture.

[Ryan Gordon]

Hi, this has grabbed my interest. I believe that LabVIEW does support isochronous connections. Although I could only find documentation on Firewire and isochronous data transfer, I have successfully used every USB webcam that I have tried with LabVIEW. The IMAQ USB drivers have to be installed. They are available to download from NI, but, I am unsure of any licensing. (I think only the NI Vision module has to be activated, not the drivers) If the device can be enumerated via USB devices, it should work via IMAQ USB capture.

At this point, I've switched to using the OpenKinect project as a dll (based on cross-platform libusb). Much easier to work with their framework rather then reinvent the wheel :) I previously tried using the VISA interface for USB capture which doesn't support isochronous transfers.

I am EXTREMELY interested in this. Do you plan on releasing it when its done?

I certainly am; I should have source code up in my github fork within the next few days.

[SuperS_5]

The only reason why using LabVIEW for the image acquisition is that IMAQ uses the Vision module. It is all reference based, and very well designed. It is fast, and has a ton of primitives for image manipulation/measurement. For automation, this would surely make life easier. The first time I used the Vision, I built a flag follower module that ran on the cRIO in about 2 hours. I suppose that the image could be sent to the Vision module, but there may be performance issues that would have to be worked out. I would be more then happy to help out with the LabVIEW side.

[Ryan Gordon]

I'll do some timing to see what kind of performance we get out of this; Converting to something that IMAQ can handle isn't too hard.

I'd surely like anyone's help (who wants to) to improve this :)

[SuperS_5]

If you have any modified code from the last time you posted, please re-post or we setup some kind of code repository. Also, which version of LabVIEW do you use? I don't want to up-save, and cause troubles when you try to open the code.

[SuperS_5]

I have setup a Google code repository account/project. This can be changed, I just wanted to try out their repository anyways.

https://code.google.com/p/kinect-labview-api

I have not seriously used their repository before, so I am still learning how to configure it.

[Ryan Gordon]

I already have it up on a github account since the OpenKinect project is located there and forking/syncing is easy.

https://github.com/RyanGordon/libfreenect/ -> Unstable branch -> wrappers/labview/

I'm using LabVIEW 8.6

[TriRulz]

This is the XboX 360 kinect or am i thinking wrong?

[davidthefat]

This is the XboX 360 kinect or am i thinking wrong?

Yes it is the 360 Kinect

[jcabrer]

Hello all.

LabVIEW VISA USB does not support isosynchronous communications. This is stated in documentation describing how to install a RAW USB driver for low-level USB device communication.

The easiest way to use the kinect with LabVIEW is to have it be recognized as a USB camera, and acquire the image data via IMAQdx. I've manged to get this working in the last couple of days, and will be posting a tutorial video on how to get there, as it's a bit involved, especially if you've been playing with all of the other kinect drivers out there from CL, OpenKinect, and PrimeSense.

Of course this limits you to using a PC running Windows OSes, so not really applicable to cRIO integration.

[SuperS_5]

I think I mentioned IMAQ somewhere along the line. I knew that using VISA was going to be difficult at best, and probably totally impractical, if not impossible. (The documentation is only the "official" use case, some people have done amazing stuff to LabVIEW. Take JKI labs for example.)
The Kinect is a USB device, so it can be loaded into virtual machines, if your host is not Windows. I have also had a prelim look at the PrimeSense, OpenKinect, Cl packages, but was stalled due to work responsibilities.

[Simy]

Hi all,
please excuse my possible naiviety on this issue but would it be possible to use the dlls from code laboratories with the call library function to control the device through labview? jcabrer - In relation to using IMAQdx, does this need specialised drivers or are you using existing drivers to do this?

[Ryan Gordon]

Hi all,
please excuse my possible naiviety on this issue but would it be possible to use the dlls from code laboratories with the call library function to control the device through labview? jcabrer - In relation to using IMAQdx, does this need specialised drivers or are you using existing drivers to do this?

I'd say realistically, probably not but it depends on your level of comfort with LabView - Other dll's may return and require datatypes that may not be easily feasible to implement in labview. I wrote an abstraction layer dll on top of the c_synchronous OpenKinect wrapper which sits on top of the OpenKinect asynchronous library just to make it work (easily) with LabView.

[Greg McKaskle]

...would it be possible to use the dlls...
The long answer is, yes. It is possible to wrap anything. Or put another way, I haven't seen anything that couldn't be wrapped, but as the DLL interfaces become more complex, the wrapping does as well. This is one of the more complex things to do in LV. As you develop the wrappers, bugs in your code are pretty unforgiving, and feedback can be pretty obscure. Even when it works, misunderstandings on who is supposed to allocate and destroy buffers, or lifetime of buffers can lead to leaks or instabilities.

That being said, if the dlls are pretty simple, if you are able to connect to external dlls, it shows that you understand many low-level computing concepts. If you attempt this, you should be able to get pretty decent help here.

Greg McKaskle

[Ryan Gordon]

Just a quick update, I've got much improved drivers committed to my GitHub Repository and making its way to the official OpenKinect branch, should anyone be interested in using these :)

More Information: http://ryangordon.net/2011/01/20/kinect-for-labview-part-2/

[borifar]

Hello Ryan,
You are doing excellent work on creating Labview driver. I am programming with Labview for > 15years. Some of my interests are visual inspection in manufacturing environment. I have used a cost effective approach and utilized consumer cameras (Canon, Olympus).Over the years I have created several measurement systems capable of measuring size with ~ 6 microns resolution (system cost <$1000) . To gain accurate measurements, the distance from the object to the camera has to be known. Several tricks can be used to resolve this, but since Kinect has a measurement laser it may have a lot of promise for automation. Do you know how accurately distance can be measured with the Kinect laser?
If you need any help with Labview Image analysis - let me know. I will be glad to help.
Regards
Boris

[Ryan Gordon]

Hello Ryan,
You are doing excellent work on creating Labview driver. I am programming with Labview for > 15years. Some of my interests are visual inspection in manufacturing environment. I have used a cost effective approach and utilized consumer cameras (Canon, Olympus).Over the years I have created several measurement systems capable of measuring size with ~ 6 microns resolution (system cost <$1000) . To gain accurate measurements, the distance from the object to the camera has to be known. Several tricks can be used to resolve this, but since Kinect has a measurement laser it may have a lot of promise for automation. Do you know how accurately distance can be measured with the Kinect laser?
If you need any help with Labview Image analysis - let me know. I will be glad to help.
Regards
Boris

Hi Boris,

Thank you! The accuracy is 11bits of depth and converted into a standard measurement size is approximately 10 cm off at 4 m away, and less than 2 cm off within 2.5 m. However there is a minimum distance also needed for the Kinect, roughly 18 inches. There is lots of information here on it: http://openkinect.org/wiki/Imaging_Information There is talk about Microsoft using a higher resolution IR Projector to achieve better accuracy.

I'll definitely let you know if there's any questions I have with the image analysis :)

Regards,
Ryan Gordon

[borifar]

Hello Ryan,
Thank you for the link- very useful info!
Could you send me your phone # and a good time to call you. I would like to discuss some specifics off line.
My E-mail: bfarber@zircoa.com
Best regards
Boris
------------------------------

Dr. B.Farber
Senior Scientist
Zircoa Inc.
Solon, OH
bfarber@zircoa.com

[AndrewD]

Do you have a download link up for the Kinect Driver vi's?

[Ryan Gordon]

Hi, yes: https://github.com/RyanGordon/libfreenect/tree/unstable/wrappers/labview

Enjoy

[borifar]

Hello Ryan,
Your dirver is working well, I was able to build a visual inspection system, which inspects ceramic crucibles. Resolution is very good - 0.1" depth and ~ 0.5 mm in cross sections. The great advantage that it allows to measure OD/ID, wall thickness, height and bottom support in one snapshort. I am not aware of any other system that provides this flexibility and speed.
The only issue is instability during startup, sometimes it takes several attempts to activate Kinect, turining power on and off and restarting Labview several times. Once Kinect is activated it works well and stable. If somebody is interested, I can share the results.
Are you planing to release new version with improved stability?
Regards
Boris

[Riobotics1]

Hi Ryan and folks who are following this thread,

Just wanted to let you know I was able to take things a little further by adding a 3D point cloud with the help of your code. Thanks for making interfacing to the kinect from LabVIEW a very simple process. Check out my post at http://riobotics.blogspot.com/2011/03/labview-xbox-kinect-and-3d-point-cloud.html. Cheers!

Also check out my youtube video:
http://www.youtube.com/watch?v=7LIL_h8JEpA

To Boris,
Yes I have noticed the same issue with activating the kinect. One time when I was debugging I used highlight execution in LabVIEW to slow things down, the program almost always works this way without error. Now that the kinect is activated, I'll run it at regular speed without any problems. So maybe, add a few delays in the initialization? Just my 2 cents.

[Ryan Gordon]

Riobotics, that is very cool to see! I look forward to seeing what you or other people can do with this

Hello Ryan,
Your dirver is working well, I was able to build a visual inspection system, which inspects ceramic crucibles. Resolution is very good - 0.1" depth and ~ 0.5 mm in cross sections. The great advantage that it allows to measure OD/ID, wall thickness, height and bottom support in one snapshort. I am not aware of any other system that provides this flexibility and speed.
The only issue is instability during startup, sometimes it takes several attempts to activate Kinect, turining power on and off and restarting Labview several times. Once Kinect is activated it works well and stable. If somebody is interested, I can share the results.
Are you planing to release new version with improved stability?
Regards
Boris

Hi Boris,

That's good to hear; Yes I will continue to keep my github synced as fixes become available in the OpenKinect repository. I don't believe that any of the issues you're experiencing could be the result of the LabView wrapper itself but if there's evidence to suggest otherwise I'll be happy to take a closer look at my own code :)

[borifar]

Hello Ryan and John,
Kudos to both of you for advancing Labview interface for Kinect.
One comment regarding stability issues on start up. My experience dealing with this kind of issues on consumer camera’s interface with Labview- having a vi that just connect to Kinect and does nothing else- typically improve stability. I just found one trick to start-up Kinect - I used only get tilt state VI, so far I was able to connect and start Kinect with no problems. I agree with John, time delay on startup is very useful
Regards
Boris

[mvonstein]

Ryan-

Great work. Just got the Kinect sensor yesterday. Working on hooking it up now. My application interests are almost exactly the same as Boris.

I'd also be willing to help you if you need anything. I've been working with Labview and the vision development library for the last 4 - 5 years and have a lot of experience with "machine vision" algorithms.

Thanks,
Mike

[perfectcheerios]

sorry for bumping this but, has there been any advance on this?

[kws4000]

http://www.youtube.com/watch?v=hlgA2nlXUF8

Check'r out, man!

Just got this from my team's lead mentor.