NVIDIA Jetson TK1

[sanelss]

Anyone else trying to use a ZED camera with a tk1? We got it working but it's not looking promising.

First off, you CAN'T use any sort of hub with the zed and usb 3. I've tried two different usb 3 hubs and it's an issue regardless. if you use a hub with other peripherals the image corruption will make it completely unusable. I had to buy a mini pci-e usb module to hook the keyboard and mouse up to so the ZED is the only device on the usb 3 port. Even with that, there is still some occasional image corruption, but at least it's usable.

The other issue the depth viewer example only gets 5-8 fps..... that's abysmally slow. Even other examples without as much heavy lifting don't get much better than that....

[Turing'sEgo]

I know this probably isn't something you want to hear, but...

OpenCV has stereo camera functions in it. They even have GPU support.

One can very easily recreate the fancy ZED camera with 2 $10 dollar webcams. If you want 1080p, then you'll have to spend more, but there is no reason you need 1080p for FRC. 480p is plenty.

[sanelss]

Originally Posted by Turing'sEgo

I know this probably isn't something you want to hear, but...

OpenCV has stereo camera functions in it. They even have GPU support.

One can very easily recreate the fancy ZED camera with 2 $10 dollar webcams. If you want 1080p, then you'll have to spend more, but there is no reason you need 1080p for FRC. 480p is plenty.

At the time wasn't aware of that but even so, the zed does come in a nice easy to mount package, is pre-calibrated, high res, high fps, and comes with examples and support. For some that may not be worth it but we felt it was.

It's just a matter of getting it to work properly. I'm sure even with 2 regular cameras, using stereo vision is still going to bog it down majorily anyway

[KJaget]

Originally Posted by sanelss

Anyone else trying to use a ZED camera with a tk1? We got it working but it's not looking promising.

First off, you CAN'T use any sort of hub with the zed and usb 3. I've tried two different usb 3 hubs and it's an issue regardless. if you use a hub with other peripherals the image corruption will make it completely unusable. I had to buy a mini pci-e usb module to hook the keyboard and mouse up to so the ZED is the only device on the usb 3 port. Even with that, there is still some occasional image corruption, but at least it's usable.

The other issue the depth viewer example only gets 5-8 fps..... that's abysmally slow. Even other examples without as much heavy lifting don't get much better than that....

Have you set the CPU and GPU clocks to max? See http://elinux.org/Jetson/Performance.

Have you reduced the resolution, fps and depth reconstruction quality of the Zed processing? Take a look at the Zed camera constructor and init calls to adjust those parameters.

[marshall]

Originally Posted by Bern1ni

I know this probably isn't something you want to hear, but...

OpenCV has stereo camera functions in it. They even have GPU support.

One can very easily recreate the fancy ZED camera with 2 $10 dollar webcams. If you want 1080p, then you'll have to spend more, but there is no reason you need 1080p for FRC. 480p is plenty.

The Zed is a fancy camera but it was born for this kind of stuff unlike the Kinect, the RealSense Cameras, and even the existing OpenCV libraries, which can't easily sync frames between cameras.

That being said, if you are going to go the OpenCV route then I would suggest looking at the PlayStation Eye cameras. You can also hack them to sync frames. They are very inexpensive and the frame rates can be jacked up to where they'll work great. I do recommend a reliable USB hub for those though into the Jetson.

[RyanShoff]

Our zed just arrived yesterday. We are still waiting on a TK1. I think it has shipped.

[dbbones]

Hello jetsons...

Team4915 is using a TK1 this year (for the first time) and we wondered if anyone has thoughts on best-practices for ensuring that vision software is running during competition.

The goal is to ensure our attached usb camera and associated vision services are running across power-down/restart events.

Currently, we're hoping that installing our software as an init.d service will just work, but we're encountering issues that may be related to intermittent network connectivity (during the reboot sequence). We've also seen some threads about usb power and auto-login may also be contributing to this.

Our current workaround is to ssh in and restart our services, but this seems fragile in a real competition setting.

Any comments are greatly appreciated!

Thanks!

Dana Batali, Spartronics 4915 mentor

[dbbones]

For future jetsonians, here's where we've gotten on this. No claims are made that this is the best approach, but it does seem to work for us (so far).

1. we built and installed a custom mjpg-streamer webserver on the jetson to serve mjpg streams to the driver station... This boots just fine as a standard init.d script.

2. our custom-built mjpg-streamer http server has a trivial extension that allows a remote webbrowser to control parameters of our vision algorithm. This just launches a shell script, which can be modified as desired. We haven't gotten any more fancy than this, but could imagine going further into CGI land or beyond. This would require more webserver hacking and doesn't seem justified at this point.

The result: after powering on the robot, the webserver is available on port 80 on the jetson. (We enabled mDNS services on ubuntu via avahi-set-host-name). Now we can launch our imaging service and it drops occasional images for delivery by mjpg-streamer to the smart dashboard on the driver station.

Issues: if we suffer from brown-out on the robot, the jetson powers off and requires a physical button-press to recover... Anyone have any thoughts on that issue?

Finally: to further protect the jetson, we invested in a power conditioning device. There is some discussion of this issue in the TX1 thread, but here's the one we landed on fwiw:

http://www.amazon.com/gp/product/B011KLQNRG

[lethc]

Originally Posted by dbbones

For future jetsonians, here's where we've gotten on this. No claims are made that this is the best approach, but it does seem to work for us (so far).

1. we built and installed a custom mjpg-streamer webserver on the jetson to serve mjpg streams to the driver station... This boots just fine as a standard init.d script.

2. our custom-built mjpg-streamer http server has a trivial extension that allows a remote webbrowser to control parameters of our vision algorithm. This just launches a shell script, which can be modified as desired. We haven't gotten any more fancy than this, but could imagine going further into CGI land or beyond. This would require more webserver hacking and doesn't seem justified at this point.

The result: after powering on the robot, the webserver is available on port 80 on the jetson. (We enabled mDNS services on ubuntu via avahi-set-host-name). Now we can launch our imaging service and it drops occasional images for delivery by mjpg-streamer to the smart dashboard on the driver station.

Issues: if we suffer from brown-out on the robot, the jetson powers off and requires a physical button-press to recover... Anyone have any thoughts on that issue?

Finally: to further protect the jetson, we invested in a power conditioning device. There is some discussion of this issue in the TX1 thread, but here's the one we landed on fwiw:

http://www.amazon.com/gp/product/B011KLQNRG

We are using the Jetson TK1 as well, and we also had the issue of needing a physical button press to start it at times. We did some research and found this. Essentially, you need to remove the C6D4 capacitor from the board to get it to always power on when it's receiving power. We did this with pliers.

Would you be willing to share your code? Your webserver implementation sounds awesome.

[sparkytwd]

For video streaming this year, we're using gstreamer with NVidia's optimized h264 codecs. On the jetson we run this:

gst-launch -v -e v4l2src device=/dev/video0 -v ! 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! nv_omx_h264enc bitrate=300000 low-latency=true framerate=30 ! 'video/x-h264,width=424,height=240,framerate=30/1' ! rtph264pay pt=96 ! udpsink host=drivestation.local port=5805 -v

That spits out a UdpSrc caps = ... line

Something like this: /GstPipelineipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)\"Z0JAKJWgbH+XQA\\=\\=\\,aM48gA\\=\\= \", payload=(int)96, ssrc=(uint)2314783494, clock-base=(uint)4227592485, seqnum-base=(uint)43060

You need the part after 'caps =', but it seems to not need the sprop-parameter-sets part of it, which gives escaping problems on Windows.

And on the drive station this:

gst-launch-1.0 -vvv udpsrc port=5805 ! $CAPS ! rtph264depay ! avdec_h264 ! d3dvideosink sync=false

Make sure to disable automatic exposure on the camera to keep it a steady 30fps.

It's robust, streaming will come back up even if the tegra reboots, or the stream is closed on the drivestation. Latency is the lowest we've seen on any solution. Quality is good while 300kBps keeps network overhead low

[marshall]

Originally Posted by sparkytwd

For video streaming this year, we're using gstreamer with NVidia's optimized h264 codecs. On the jetson we run this:

gst-launch -v -e v4l2src device=/dev/video0 -v ! 'video/x-raw-yuv,width=320,height=240,framerate=30/1' ! ffmpegcolorspace ! nv_omx_h264enc bitrate=300000 low-latency=true framerate=30 ! 'video/x-h264,width=424,height=240,framerate=30/1' ! rtph264pay pt=96 ! udpsink host=drivestation.local port=5805 -v

That spits out a UdpSrc caps = ... line

Something like this: /GstPipelineipeline0/GstUDPSink:udpsink0.GstPad:sink: caps = application/x-rtp, media=(string)video, clock-rate=(int)90000, encoding-name=(string)H264, sprop-parameter-sets=(string)\"Z0JAKJWgbH+XQA\\=\\=\\,aM48gA\\=\\= \", payload=(int)96, ssrc=(uint)2314783494, clock-base=(uint)4227592485, seqnum-base=(uint)43060

You need the part after 'caps =', but it seems to not need the sprop-parameter-sets part of it, which gives escaping problems on Windows.

And on the drive station this:

gst-launch-1.0 -vvv udpsrc port=5805 ! $CAPS ! rtph264depay ! avdec_h264 ! d3dvideosink sync=false

Make sure to disable automatic exposure on the camera to keep it a steady 30fps.

It's robust, streaming will come back up even if the tegra reboots, or the stream is closed on the drivestation. Latency is the lowest we've seen on any solution. Quality is good while 300kBps keeps network overhead low

Nicely done!

[dusty_nv]

***Attention Silicon Valley FRC and FTC Teams!***
NVIDIA will be hosting its first-ever FIRST Day at our GPU Technology Conference (GTC) on Thursday, April 7th from 10am – 2:30pm at the San Jose Convention Center. Just a few miles from the San Jose State Event Center, GTC is the largest event of the year for developers like you working on the forefront of visual computing and A.I. — you know, the kind of computing that helps robots connect, see, think and learn. April 7th is a practice day at the San Jose Event Center and we are hoping some of you can join us for this FIRST Day at GTC. Students must be accompanied by an adult, and entrance to the event is provided free of charge.

We’ve got some special events lined up for just for FIRST Participants:

A Hands-on Lab that includes Deep Learning and Getting Started on NVIDIA Jetson
This will be a practical hands-on session on deep learning and Caffe with GPU acceleration. Each station will include a Jetson for detecting and classifying objects in realtime from a live camera using neural networks.

To make sure you leave GTC with everything you need to take your robotics to the next level, each hands-on lab station will include a free Jetson TX1 to take home, compliments of NVIDIA.

Tour of the NVIDIA GTC Exposition Hall
• Come see our intelligent, autonomous machine and robot demos that includes everything from submersibles to drones that help with search and rescue to ruggedized, autonomous vehicles that use deep learning.
• Experience the VR Village where you can explore the latest advances in VR technologies and learn all about the visualization power they demand. You’ll see VR demos from 3D gaming, to product design, to cinematic experiences and beyond
• See demos on artificial intelligence, graphics virtualization, accelerated computing, product design, self-driving cars, media & entertainment and more!

There will also be a presentation on NVIDIA technology and how GPUs have changed the game in everything from gaming to movies to the new frontier in artificial intelligence.

Space is limited, so please sign up here to reserve your spot: http://goo.gl/forms/wU4MFGqcFI
Hope to see you there!

[snekiam]

Originally Posted by dusty_nv

***Attention Silicon Valley FRC and FTC Teams!***

To make sure you leave GTC with everything you need to take your robotics to the next level, each hands-on lab station will include a free Jetson TX1 to take home, compliments of NVIDIA.

Its unfortunate this is only available to California teams, but I appreciate your ongoing support of the FIRST community!

[marshall]

Originally Posted by snekiam

Its unfortunate this is only available to California teams, but I appreciate your ongoing support of the FIRST community!

In fairness to Nvidia, they invited us out for it (and we are a fair bit away from CA) but we've got a competition to attend. Wish we could be there though. What an amazing partner for FIRST to have and I can tell you that they are an awesome sponsor for our team.

[dusty_nv]

Originally Posted by snekiam

Its unfortunate this is only available to California teams, but I appreciate your ongoing support of the FIRST community!

If you're in the area, RVSP and stop by!