Hi everyone, as many of you know this years FIRST challenge is one that vision tracking may be useful for.
My team is 100% planning on using vision tracking. Here is what we have for it. If I need more parts for this or need to replace anything I would love to hear your suggestions
NVIDIA Jetson TK1
Microsoft Lifecam 3000
I think this is all we need for vision tracking, but I’m not quite sure.
Aside from that, I am clueless as to where I should start. We do not have any programming mentors, and I’m the only programmer on our team (8 people). Is it something as simple as plugging it into the roborio, and accessing it through the FIRST API? Can I even use Java? I’m just very frustrated right now, and am in dire need of help.
Something just to start me off, and understand how I can program and utilize it for vision tracking. One of my main worries is not being able to use Java to program it, but I’m more than willing to use another language if I have to.
Also, if it’s not as easy as programming it in the same project as the rest of the robot code, how would I go about using the other code in the main robot project?
Thank you very much for reading and any responses you leave!
Team 1619 has used and likely will continue to use a Jetson TK1 with a USB webcam for vision. We have written our vision code in Python, utilizing the Python interface to OpenCV, an open source computer vision software library. I would recommend using Python, as it is very easy to pick up and use for vision processing. It is also very easy to run on the Jetson.
In the past, we have run a socket network in order to communicate between the Jetson and roboRIO (through the router). However, a simpler (and possibly better) alternative is to use pynetworktables, a “pure python implementation of the NetworkTables protocol”. This will allow you to communicate calculated values from your vision code to your robot code running on the roboRIO. In order to stream the unprocessed/processed video feed from the Jetson to the driver station, a software such as MJPG-streamer can be used.
I would recommend getting OpenCV running on a regular computer first. This way, you can figure out how to interface with OpenCV using Python and do some basic image manipulations and calculations, without the hassle of developing on the Jetson. You will want OpenCV 2.x as this is what will run on the Jetson. Here are some great tutorials on image processing with OpenCV: http://docs.opencv.org/master/d2/d96/tutorial_py_table_of_contents_imgproc.html. You can also play around with pynetworktables (https://github.com/robotpy/pynetworktables) to communicate between an external device (the computer) and the roboRIO.
From here, you can install OpenCV on the Jetson: http://elinux.org/Jetson/Installing_OpenCV.
Once the Jetson is installed on the robot, you will have to use SSH to operate it (http://elinux.org/Jetson/Remote_Access). For our setup, we assigned the Jetson a static IP, however this is not necessarily required. A warning here: SSH’s default port of 22 is blocked by the FMS, so if you need to SSH to the Jetson pre-game from the driver station, you must change the SSH port to one allowed by the FMS. Reference this: http://wpilib.screenstepslive.com/s/4485/m/24193/l/291972-fms-whitepaper.
I hope this gives you a good idea of where to start!
I would make a suggestion, if you are the only one programming the robot, and there is a good chance you will get pulled into doing other things (since there are only 8 people on your team) stay away from doing vision on the Jetson, it is not a small undertaking, and the resources are limited. It is a great offseason project. I mentor a team with multiple programmers and programming mentors, and we just got a TX1, currently we are leaning away from using it. It is 100% definitely possible to do this on your own, but it will be a lot of work. I don’t want to stop you from learning something new, but we did a Cost/Benefit analysis for our own team and are having trouble committing to the project.
I am nearly 100% certain at this point that a PiXY cam see here will be capable of doing the vision tracking you want to do. Interfacing with it can be as simple as using analog IO on the rio. The cost is low, $60ish and it stops you from having to do most of the programming. Some teams used this to great success last year. The majority of the top teams used an older version of this as far back as 2006 to build auto aiming shooters. I strongly recommend it to a team with limited programming resources.
Correct me if I am wrong, but all of the documentation I have found says that OpenCV does not have the Python bindings for the GPU. So if you use Python then you are only able to use the regular CPU functions. I think the main advantage of the Jetson is having the Tegra Accelerated OpenCV functions. If a team is committed to using Python, then I would recommend other boards that are cheaper and better supported/documented.
The Jetson really shines when you use Cuda and offload operations to the GPU. This can only be done using C++ and for simple target tracking is likely going to be overkill.
Speaking from past experience don’t get in the mindset that you need the highest frame rate possible. After using vision for automatically aligning to the goal last year 2481 learned that in most cases you only need 1 or 2 frames. The key to making this work is to use the camera to determine the angle you need to rotate and the distance from the goal on one frame. Then use a gyro or encoders to perform the rotation. Optionally once your control loop indicates you are on target you can snap another frame and verify to make sure that you are aiming at the target. For a simple algorithm like this you can easily get away with using OpenCV running on the RoboRio and save yourself some complexity. All that said we ran 8 fps and that was plenty adequate.
I would also recommend you checkout GRIP. They just added code generation for C++, Java, and Python.
Something I’d add to the materials list is a green ring light to mount on the LifeCam. It’s important for thresholding purposes.
As for advice, if you do decide to do vision, I’d suggest you toy around with GRIP and perhaps some basic python/opencv code in your own computer first, to get the basics down. It’s important to decide what exactly you’re trying to accomplish with vision, whether you’re going goal alignment or gear alignment. I’d advise you look at other teams’ well documented vision code for inspiration.
Start with GRIP and develop a plan for what you want to do. Then worry about the technology for how to do it on the robot. Why use a TK1 and make things hard when you might be able to do it on the driver station or the RoboRIO?