Best way to do vision?

Technical Programming
21

We also used the Orbitty carrier this year. The actual Jetson is a small module that connects to the dev board:

The carrier board for the TX1 / TX2 is essentially just a breakout board for IO. The Orbitty board does the same thing but with a much smaller footprint and with some nice features (screw terminals for power) that the larger board doesn’t have.

1 Like
22

I have a couple of questions regarding how to get the vision work with CheesyVision.

First of all, what is the application you use on your android phone for processing the vision and running the code? Does the android application do the processing all by itself, or did you get a kangaroo/ Arduino to process the vision info.

Finally, how do you connect your phone to the roborio (I am assuming USB since it would be the most typical thing to do…)?

Thank you
Team 3647 Programming Team

23

We used pynetworktables; it was relatively painless.

24

Last year we went down the path of vision on a BeagleBone black as co-processor, using the MS Lifecam 3000, but ended up with a horrible 2s lag and about 1-2 fps. Over the weekend before champs, I learned a ton, recompiled a lot of packages, and bought a Logitech C920. This got it to about 20fps with a 100-200ms lag.

Fast forward to this year, and we used a RPi 3 with the Logitech C920, with a stack that let us run OpenCV code as a filter for mjpg-streamer. This let us hook into the streamer and process an image, draw on it, and send it back to the dashboard very easily.

The code and tool setup we have running has been set up and run on a BeagleBone Black, RPi 2 & 3, and the Jetson TX1, so we can easily pick and choose the platform we want. This year with the Pi3, we found we got a few extra frames per second over the BBB. We didn’t use the TX1 for a few reasons, mostly space, and we didn’t have a spare.

To get values back to the RIO, we are simply using a socket to send data as a string over TCP, to a socket handler thread in our robot code. When a client connects, we send the HSV values to the coprocessor for it to use, so it is all configured through the robot software (and our config file). We spin the client socket off on another thread so it doesn’t hold up image capture and processing. In all, we were running at about 15-20fps, with about a 500ms delay, which we want to eliminate once we figure out where it is introduced.

As drive coach, the benefit of having the code as a filter was being able to draw key points on the image, as well as an indicator for if the coprocessor was not able to connect to the RIO (really useful to know before a match starts!), as opposed to the RIO thinking it is connected, and being able to do all that without having to write images to be streamed back, or just the raw camera stream.

Our instructions for setting this up are written up, but I’ll likely clean them up and post it up in the next week or so and get it on our blog.

25

So the Cheezdroid app is a custom built app for vision processing. Its role is to do the vision processing and send over relevant info (distance, angle etc) through a USB cable to the rio, which interprets this data and you would use to turn your robot, etc. You dont need another coprocessor like an Arduino to process the data,as the trio can doi it pretty fast, with little CPU usafe.

26

Best way to do vision?

For most teams, the best way to do vision would be finding a mechanical solution to the problem.

Vision is a challenge for most FRC teams and I’ve seen it ruin multiple teams’ seasons because they spent all their time working on vision and just never got it working. However, you can obviously learn a ton from implementing a vision system on your robot, and when it works, it’s a huge advantage for your team.

4607 used a TX1 and we have a very solid foundation for vision processing for future years.

27

So CheezDroid is the app which runs on the android phone and you do not need a kangaroo/arduino because the phone acts as a co-processor. It sends to the robo-rio vis USB using ADB. I think CheezDroid send data in a JSON format. I would suggest just putting it into Android Studio and uploading onto an android app to check it out. Then, mess around with it and go from there. I have not found 254’s actual processing file but if you can find it, adapt your script and use their’s as an example.

There are really lots of ways to vision process. The most cost-effective would be running it straight on the rio. Running it on the rio however can take up lots of RAM and too much CPU Usage. A rio would be a good solution for low-resource teams.

Another cost-effective method would be running GRIP on the DriverStation and sending values over to the rio via Network tables. This could possibly fix your solution of power issues.

A more costly method is using a co-processor. A Pi is one of the most common ones because they are cheap. Other co-processors are the BeagleBone and Jetson. The hardest thing about co-processors is networking and integration. If you guys can spend the money on a Jetson Tx1/TX2 or get it from FIRST Choice I would say go for it. Its processing power is amazing. If using a Jetson, I would highly recommend utilizing the GPU Acceleration. It is not necessary, but it can save some lag and resources. The Jetson might be your best bet because processing power will not be an issue with the Jetson. If you guys need any future help with OpenCV, networking, integration, or anything at all feel free to ask here on ChiefDelphi or email us at [email protected]

PS: I realized after I typed rio solution and pi solution that you guys have already tried that. However, I’ll leave it in there if someone else was wondering the “best” way to vision processing. When you say power problem you mean processing power for the rio and the pi right, not electricity problem? I just want to make sure lol. :slight_smile:

28

in debate if a new thread should be started but I been following and commenting in this one about he cheezdroid.

I have android studio installed. The Vision app downloaded and open in Android studio. This step is done.

Download http://sourceforge.net/projects/opencvlibrary/files/opencv-android/3.1.0/OpenCV-3.1.0-android-sdk.zip/download
Unzip OpenCV-3.1.0-android-sdk.zip (OpenCV-android-sdk)
Create directory app/src/main/jniLibs
Copy OpenCV-android-sdk/sdk/native/libs/* to app/src/main/jniLibs/

I keep getting this error when trying to send it to my Galaxy S7



FAILURE: Build failed with an exception.

* What went wrong:
Execution failed for task ':app:ndkBuild'.
> A problem occurred starting process 'command 'ndk-build.cmd''

* Try:
Run with --stacktrace option to get the stack trace. Run with --info or --debug option to get more log output.

BUILD FAILED
29

Did you install the NDK and did you configure the path to the NDK?

30

Well that is probably it. I Now installed the NDK but configuring the path is un clear.

UPDATE:
I had to change the build.gradle


// call regular ndk-build(.cmd) script from app directory
        task ndkBuild(type: Exec) {
            if (Os.isFamily(Os.FAMILY_WINDOWS)) {
                commandLine 'C:\\Users\\Applefamily\\AppData\\Local\\Android\\sdk\
dk-bundle\
dk-build.cmd', '-C', file('src/main').absolutePath
            } else {
                def ndkDir = android.ndkDirectory.getAbsolutePath()
                commandLine ndkDir + "/ndk-build", '-C', file('src/main').absolutePath
            }
        }
31

For the teams starting down the vision road installing GRIP can really help to get started.

32

I did not see anyone mention using a Kangaroo PC which has an FRC legal integrated battery and as a PC can run x86 64bit kernels of Microsoft Windows or Linux.

I did see someone mention the low frame rate of the Microsoft LifeCams and by the way that bug is because you can not disable the white balance controls correctly for those camera except in Windows - we tried to work with the Linux package maintainer but could not get needed details from Microsoft to finish.

I like the PS3 Eye for high frame rate needs.
Logitech C920 for high resolution needs.

The Kangaroo means you could develop on any x86 laptop and deliver directly to your vision system on likely the same OS removing whole layers of complexity to target ARM, NVidia’s chips or Android which each are valuable skills to develop but, you do have a deadline in FRC.

Also embedded platforms sometimes change enough between revisions that consistent documented direction needs to be continuously produced, where a Linux or Windows PC is generally expected to be compatible with software written within the previous decade.

Finally Packt Publishing has a pile of books on OpenCV and sometimes they give them away for free:

I have hundreds of free computer related books from that link. Just go there once a day every day for a few years ;).

Unlike Safari Books online you get the whole PDF or eBook not just chapters, but all of those books are in Safari like most O’Reilly books.

Maybe later this year I will follow this up with some detailed instructions. It has come up when I CSA before.

33

Team 5638 used a kangaroo pc running grip to send values to the roborio.

you can see the rio side of the code https://github.com/brandonlund/Final-Code-2017-5638

34

I second the Kangaroo and GRIP as a great solution. Our team has used this the past 2 seasons with varying success. Mostly positive. We are now looking at the other more complicated solutions like android.