This year we have been experimenting to utilize vision in our robot for next year. Our programmers are really good with JAVA and have experimented with GRIP before. What is the best camera/ Processor to use? We have seen Raspberry Pi’s being used, NVIDIA Jetsons, JeVois Smart Cameras etc. Since we are beginners with Vision what is the best easiest way to implement this, and what materials are the best to purchase?
From my experience, Raspberry Pi’s are too slow for any advanced visual workloads. This year our team will be using a Jetson TX1 running some exported GRIP code. I think that it would be easiest to just run GRIP on a coprocessor like a Jetson without exporting code. If you want good vision processing, I would go the Jetson route. I don’t have experience with any of the other technologies though, so take this advice with a grain of salt.
Sent from my iPhone using Tapatalk
I’m about as far away as you can be from a vision expert, so take this with a grain of salt.
The easiest way, bar none at the moment, is Limelight. If it’s as good as everybody is saying it’s plug and play. Next would probably be the JeVois camera. I can’t comment on using the raspberry pi or other boards as I have no experience in either of those.
The easiest thing IMO is the Limelight. But if you want better performance and a lower price, I’ve heard that a Kangaroo Camera running Linux and GRIP with a Logitech C920 is extremely easy to set up and tune. Team 3991 has been working on making a tutorial and I’m trying to get my hands on a Kangaroo as soon as possible. It has an integrated battery as well so wiring issues are not going to kill vision. The old Playstation Eye camera does 120fps 240p video so it’ll be somewhat faster than the limelight.
I’ve worked with the JeVois and while it is easier than an entirely custom solution, you do need to know what you’re doing somewhat to get it working.
I looked on the Limelight website and they are sold out and on backorder. Do you feel that the kangaroo camera is comparable?
Nothing that exists today is comparable to limelight IMO.
+1. Limelight is it’s own thing. But a Kangaroo is very, very good from what I’ve seen.
I would have to say limelight. We got a beta unit to mess with and had it fully working within two hours. It was as good or better than the poof camera/vision setup we used in 2017. Would highly recommend.
Most of the time we spent getting it working was flashing it, running an Ethernet cable, and making a mount for it.
More units will be available here shortly.
We’ve done vision with a Raspberry-Pi running OpenCV in Python for the last two years.
Here’s the vision section from our code training manual:
OpenCV
OpenCV is an open-source image processing library. It can be installed for using apt-get or pip, depending on the Linux distribution.
sudo apt-get install python-opencvReading an Image
OpenCV can read images from a USB camera, such as the Microsoft LifeCam. Once a capture object is created, parameters such as image resolution can be set. Images can be captured using the read() function.
cam = cv2.VideoCapture(0) cam.set(cv2.CAP_PROP_FRAME_WIDTH, 320) cam.set(cv2.CAP_PROP_FRAME_HEIGHT, 240) ret, frame = cam.read() # frame is the imageHSV Filtering
Filtering by hue, saturation, and value (HSV) allows us to select the pixels in an image that are a certain color, in most cases the color of the reflected LED light. It is the first step in finding a target. The image is first converted from the RGB colorspace to HSV. Then lower and upper values for hue, saturation, and value are specified. Those values are then used to create a binary image with white pixels wherever the specified color was in the original.
# Convert RGB to HSV hsv = cv2.cvtColor(frame, cv2.COLOR_RGB2HSV) # define range of blue color in HSV lower_blue = np.array([110,50,50]) upper_blue = np.array([130,255,255]) # Threshold the HSV image to get only blue colors mask = cv2.inRange(hsv, lower_blue, upper_blue)Contours
Once an image has been filtered, a contour must be found in order to extract information such as the height, width, and position of the target. Contours are contiguous regions of pixels. The findContours function returns a list of all the contours in an image. There may be contours other than the target, so you must then loop through the list to find the best match (for example the largest contour). The contour’s centroid can be found using moments.
contours, hierarchy = cv2.findContours(mask, cv2.RETR_TREE, cv2.CHAIN_APPROX_SIMPLE) target = contours[0] for c in contours: if cv2.contourArea(c) > cv2.contourArea(target): target = c m = cv2.moments(target) cx = int(m'm10'] / m'm00']) cy = int(m'm01'] / m'm00'])Once the centroid has been found, the angle between the robot’s heading and the target can be calculated using the field of view of the camera.
angle = (cx / IMAGE_WIDTH) * (FIELD_OF_VIEW / 2)Other Camera Settings
To make sure the reflection appears green (and not overexposed as white), the camera must be underexposed. Exposure can be changed using libwebcam’s uvcdynctrl tool. Use the -c flag to list available controls for your camera, -g to view a control’s current value, and -s to set a value. Turning off auto-exposure and set manual exposure to its lowest setting is usually good.
We used networktables for communication in 2016 and 2017. Here’s the section on that:
NetworkTables is the software used to communicate between different devices on the robot’s network. One example of a NetworkTable is SmartDashboard, which allows the roboRIO and the driver station laptop to communicate. Other NetworkTables can be created for communication between the roboRIO and a co-processor, such as a Raspberry-Pi. There are libraries for both Java and Python. NetworkTables should not be used for time-sensitive applications, such as vision processing. If network communication is needed, use UDP packets instead.
The Python library can be installed with pip install pynetworktables. You can load a table with the getTable() function, and send and retrieve data using the putNumber() and getNumber() functions.
from networktables import NetworkTable # Connect to roboRIO NetworkTable.setIPAddress(10.51.15.2) NetworkTable.setClientMode() NetworkTable.initialize() # Load a table named 'myTable' nt = NetworkTable.getTable('myTable') # Send data to the table nt.putNumber('key', value) # Get data from the table value = nt.getNumber('key')The Java library must be imported from edu.wpi.first.wpilibj.networktables. It uses the same names for functions. The first three statements (connecting to the roboRIO) aren’t necessary when writing code for the roboRIO.
You also might need to set your Raspberry Pi to a static IP. I’m not sure if it’s absolutely necessary, but we find that using static IPs for everything makes it all run smoother.
Setting yourself a static IP:
open a terminal window
type:cd /etctype:
sudo nano dhcpcd.confwaaay at the bottom, type:
static ip_address=10.51.15.30/24 static routers=10.51.15.1Then press:
Ctrl+x
Y
enter
Make sure the changes took hold by restarting the pi. Then check your ip address by opening a terminal window and typing:
ifconfig
You should see “10.51.15.30” somewhere in there.
Btw, any time you see 5115 or 51.15 in here, replace it with your team number.
Good luck this season!
“Best” method is very subjective. Best will depend what you want to achieve. May I suggest you think more along the line of long term of direction for your team, rather than an immediate need.
Correct, the easiest is Limelight. It is as close to plug ‘n’ play as you are going to find. That said, it will not, in it’s current incarnation, be a system that will allow your students to learn a lot about vision processing and tracking software. Again, this is not a bad thing as long as it meets your needs for “best”.
I strongly disagree with this comment. While the Pi3 is not the fastest processor available, it is more than capable of processing over 100fps at 320X240 when running multi-threaded. In all honestly, a vision system will only generate unique tracking data at a rate that is equal to the fastest rate that the sensor is capable of acquiring them. If your processor if faster than that, you will be receiving duplicate frames. For example, if the HD-3000 USB camera can only acquire at 30fps in 320X240, you will never get more than 30 unique sets of targeting data per second.
This is an excellent approach as well. A similar approach can be taken with JeVois, but the code needs to be exported to it from a separate host. There is at least one whitepaper on this approach already available. Search for it.
Now for my $.02.
If you’ve been on CD much in the last month or so, you may have noticed a lot of traffic from others and myself on JeVois. So, I may be a bit biased.
We (2073) are shifting to JeVois this year for multiple reasons. Price, size, weight, and performance. We have used the RPi3 with an HD-3000 for a couple years now with great success, and I believe it is still a good choice.
I will not tell you JeVois is the “Best”. It will require you to learn a bit about programming, serial communication, and system integration to get the most out of it.
There are at least two white papers already available on how to integrate your code into it. One is basically for Python based OpenCV and the other is for GRIP.
Lastly, Limelight is a majorly phenomenal product. It has excellent support. It is a bit expensive, but you get what you pay for.
A few quick notes:
- If you have the money, and are just starting out with vision, and are not too afraid of their delivery schedule – Limelight looks hard to beat
- Limelight uses a pi compute, which has the same processor as a Raspberry Pi, so it should be fast enough. That said, we had more trouble with frame capture speed than the actual vision algorithms when using a Pi – the Limelight guys obviously have a good solution for this.
- JeVois is a really nice package, very affordable, support so far has been spot on. But will require more effort to integrate than the Limelight. It also exposes more of the process to your developers so they will have more opportunity to learn about computer vision and to personalize it for your needs.
- We had very good luck with an Android camera solution based on 254’s code, but the frame rate & latency was not as
- Until we know this years game, everything is going to be guessing, but I suspect either the Limelight or the JeVois will be “best.” It is likely canned solutions will be available shortly after the game is released for both systems, that will be fast enough to allow closing the loop on vision only.
Will they be in stock before build season starts?
I don’t know when they will be in stock, but I do know you can sign up on the limelight website to get updated when they are.
My thoughts:
(1) The Limelight seems like the easiest solution right now bar none. The fact that the first set sold out so quickly and the tight schedule is a bit scary though.
(2) Raspberry Pi is a very viable and easy set up to take advantage of. It’s cheap, somewhat reliable, and not too difficult to implement, with a lot of guides online to help. If you need help with any of it feel free to let me know.
(3) Some people recommended the Jetson; I would not for you right now. The Jetson is not something you should be taking on during build season, especially if your programmers don’t have a lot of C and Linux knowledge.
(4) I’d start ASAP. Your biggest obstacle right now will be time, and getting vision to work reliably before the end of build season is going to take a few dedicated programmers and lots of time on the internet.
Best of luck, if you have any issues feel free to message/email me 
If you are planning on working with OpenCV on a co-processor board, I would look for better processors than the RaspberryPi. The RPi is too slow in my opinion.
My team is working with the ODROID-XU4, which is 3 times faster than the RPi (which is significant), but much cheaper than a Jetson. That speed means we can easily keep up with processing a 640x480 image at 30 fps.
It might be too slow for 640x480 images, but you don’t need images near that quality for the popular color segmentation approach. 240p is more than enough, and a Raspberry Pi can handle that decently, especially the newer models. ODROID boards look like good solutions, but I don’t think they’re necessarily needed in a lot of cases.
Yes, that is true. OTOH, we like to feed the same image to the driver’s station, and 240p is pretty small for that. Down sampling a 640x480 before processing the image just adds to the time. But, yes, a RPi is workable.
I don’t think downsampling is a very real time concern. I tested this out in Python by creating a random RGB image at 640x480 and downsizing it to 360x240 and got a result to the order of a few hundred microseconds on my laptop. Of course this would be much slower on a coprocessor such as a Raspberry Pi, but I doubt it’s incredibly significant on the scale of the rest of the pipeline at 30 Hz. Here are my results and the script to test if you have a coprocessor handy:
C:\Users\DanWaxman\Documents>py downsampletest.py
Total time over 10000 iterations: 1.9498084102566757
Average time: 0.00019498084102566757
To be clear, I’m not disagreeing that a more expensive and powerful coprocessor is a bad idea; just that it’s not needed for a team just getting into vision, and that downsampling isn’t a very real concern.
Edit: Whoops, in my original post I was testing grayscale. I changed it to RGB which made it slower, but still reasonable.
My 2 Cents from my experiences.
1 Kangaroo running GRIP. We used a Axis-IP Camera and USB. Very Easy to use and configure. A little more work connecting all the parts. A plus or minus maybe you can[must] create you own vision pipeline. The USB to Ethernet adapter gave us some trouble with radio communications, just be sure keep it away from the radio and everything will be fine.
Hardware:
($180)Kangaroo
($30-100)Camera
($10)USB 2 Ethernet adapter
($10)12vdc regulator
($10)cooling fan for Kangaroo & 3D printed case
($10)LED ring Light
($?)3D Printed mounts
Software:
everything mostly Windows
Grip [fairly flexible]
VNC [to remote into Kangaroo]
Data into robot:
Network Tables [easy]
2 Lime Light, Yes I have one [Beta]. Very Simple to tune the very well developed vision pipeline.
Hardware:
($400)Lime Light
Software:
web browser [Chrome]
firmware tool to transfer updates to device.
Data into robot:
Network Tables [easy]
3 Jevios, I only just started trying to get this working. FRC guides are only just beginning to get written. I haven’t been fully successful yet. Must have to write the vision pipeline from scratch or currently manually copy GRIP pipeline. Can be very customizable if your familiar with OpenCV
Hardware:
($60)Jevois
($10)LED Ring Light
($?)3D printed mounts
($10)USB Power >800mA
Software:
GRIP [Tune Pipeline]
Python editor [Copy pipeline to Jevios]
Arduino IDE [for serial monitor]
AMCap [video monitor]
Etcher [burn jevios images]
Data into robot:
TTL Serial [unstructured more complicated]
USB-serial
USB-video
JeVois does not do RS232 without additionsl hardware.
It can do serial over USB, or TTL serial.