Go to Post I've learned never to say never when it comes to FIRST teams. - Karthik [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rating: Thread Rating: 8 votes, 5.00 average. Display Modes
  #1   Spotlight this post!  
Unread 08-01-2014, 13:23
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Raspberry Pi + Camera Module for Vision Processing

In the past when my team (172 NorthernForce) has tried to do vision processing, we've not had much luck. 2-3 years ago, we tried doing it on the cRIO, which had horrendous framerates, so last year we added a raspberry pi with a USB camera. This worked better, but was still pretty slow. This year, we have some of the raspberry pi camera modules - and so far this looks pretty promising. Interfacing with the camera is a real pain however, so I thought I'd share what I'm working on, in case anyone else is interested in using the camera module.

I've built a class called PiCam for interfacing with the camera. All you have to do is construct a PiCam instance, giving it a requested size for images, and a callback, which processes the frames. The callback has one parameter which is an OpenCV matrix/image of the current frame to be processed.

The code is on github at http://github.com/NorthernForce/2014-Vision, and there is Doxygen documentation hosted at http://northernforce.github.io/2014-Vision, which I will try to keep up to date as well.

Please keep in mind that this is in a pretty early state, and that there are many unfinished things - for instance I plan on adding more features for controlling the camera parameters (especially turning off that pesky auto exposure!).
  #2   Spotlight this post!  
Unread 08-01-2014, 17:07
wesleyac's Avatar
wesleyac wesleyac is offline
Registered User
AKA: Wesley Aptekar-Cassels
FRC #1678 (Citrus Circuits)
Team Role: Programmer
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Davis, CA
Posts: 61
wesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to beholdwesleyac is a splendid one to behold
Re: Raspberry Pi + Camera Module for Vision Processing

Out of curiosity, why did you choose the Pi?

Our team decided to go with doing all the computation on the driver station with Octave (A free MatLab clone), but we are doing some fairly heavy calculations. I am a bit worried about the bandwidth limit, but my other team members seem to think it'll be fine.

Also, I can't seem to load the GitHub page, but that's probably GitHub's fault
Sometimes I get a "Unicorn!" error, although once I got a Error 500.
  #3   Spotlight this post!  
Unread 08-01-2014, 17:56
yash101 yash101 is offline
Curiosity | I have too much of it!
AKA: null
no team
 
Join Date: Oct 2012
Rookie Year: 2012
Location: devnull
Posts: 1,191
yash101 is an unknown quantity at this point
Re: Raspberry Pi + Camera Module for Vision Processing

Use an Intel NUC or an ODROID, etc. instead. The RPI is one of the slower chips you could get. Also, I think there are already tons of libraries to do this!
  #4   Spotlight this post!  
Unread 08-01-2014, 21:13
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Re: Raspberry Pi + Camera Module for Vision Processing

Mainly, I'm using the pi right now because we already have several, as well as the camera modules. It would be nice to have something a bit faster though. Thanks for pointing out alternatives though, I had done some looking but didn't get very far. I may just have to get my hands on an ODROID.

To clarify I'm using OpenCV for the image processing, but getting data from the raspberry pi camera module (over the csi-2 port) isn't straight-forward. I did look around for some library to do this, but didn't have much luck - the closest I really found was the raspicam app from userland (https://github.com/raspberrypi/userl.../apps/raspicam) - which is what this code is based on.

I don't have any trouble with the links .
  #5   Spotlight this post!  
Unread 08-01-2014, 22:16
Jeanne Boyarsky Jeanne Boyarsky is offline
Java Mentor
FRC #0694 (StuyPulse)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2010
Location: New York
Posts: 100
Jeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud ofJeanne Boyarsky has much to be proud of
Re: Raspberry Pi + Camera Module for Vision Processing

Nice! We tried to use the Pi last year but the Java libraries couldn't process images fast enough. (Ironically another Java library was fine, but not compatible with the vision code and then we ran out of time.) This year, we are trying in C++. We are definitely going to look at your code and see if it can help things along.
__________________
Team 694 mentor 2010-present, FIRST Volunteer and Co-organizer of FIRST World Maker Faire Tent
2012 NYC Woodie Flowers Finalist
2015 NYC Volunteer of the Year
  #6   Spotlight this post!  
Unread 09-01-2014, 08:40
Foster Foster is offline
Engineering Program Management
VRC #8081 (STEMRobotics)
Team Role: Mentor
 
Join Date: Jul 2007
Rookie Year: 2005
Location: Delaware
Posts: 1,394
Foster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond reputeFoster has a reputation beyond repute
Re: Raspberry Pi + Camera Module for Vision Processing

If you are using a Raspberry PI and Java: Go install the latest Java 7 SDK from Oracle for the PI. Raspberry PI announcement

This version has been compiled for the PI with hardware float set and some other optimizations. I measured a huge decrease in processing time (transactions that were taking 4-5 second are now subsecond) with the new version.

As always, Your Mileage Will Vary, but it's well worth taking the 10 minutes out to download and set up the new version.
__________________
Foster - VEX Delaware - 17 teams -- Chief Roboteer STEMRobotics.org
2010 - Mentor of the Year - VEX Clean Sweep World Championship
2006-2016, a decade of doing VEX, time really flies while having fun
Downingtown Area Robotics Web site and VEXMen Team Site come see what we can do for you.
  #7   Spotlight this post!  
Unread 09-01-2014, 14:03
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Re: Raspberry Pi + Camera Module for Vision Processing

I've done some more testing, and the framerates so far are much much better than what we got last year. I'm doing a conversion from RGB to HSV, doing a simple threshold on each channel, and then finding the centroid and this is running at 17fps for 320x240 video and 30fps for 160x120 video. One of the main things I want to do is track the balls - which I think these resolutions would be fine for, since the ball is so big. We might need higher resolution for the vision targets, but I'm not sure.
  #8   Spotlight this post!  
Unread 10-01-2014, 07:01
AverageGuy AverageGuy is offline
Registered User
AKA: Jim Lynch
FRC #4080 (Reboot)
Team Role: Mentor
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Georgia
Posts: 4
AverageGuy is an unknown quantity at this point
Re: Raspberry Pi + Camera Module for Vision Processing

Just in case you've not seen this package, picamera is a Python library for accessing the camera on a Pi. I haven't used it but the author seems to like it.

http://picamera.readthedocs.org/en/release-0.8/

I'm just curious why you would run Java on a Pi? It seems awfully resource heavy. C or even Python would appear to be much better manager of resources. Just wondering.

Jim.

PS we're using C++ and raspistill in signal mode to grab parts of the cameras FOV for pattern matching. FYI.

Last edited by AverageGuy : 10-01-2014 at 07:04.
  #9   Spotlight this post!  
Unread 10-01-2014, 09:57
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Re: Raspberry Pi + Camera Module for Vision Processing

Using the signal mode of raspistill is something I hadn't thought of. That must be pretty slow though, since you have to wait for the image to be written out and then read it back in. I'm hoping to track the balls so I need a pretty high framerate (and as I mentioned, it looks like I'll have it), but for identifying when the goal is hot in autonomous mode, fetching images with raspistill is probably fast enough and easier to accomplish.
  #10   Spotlight this post!  
Unread 10-01-2014, 10:41
AverageGuy AverageGuy is offline
Registered User
AKA: Jim Lynch
FRC #4080 (Reboot)
Team Role: Mentor
 
Join Date: Jan 2014
Rookie Year: 2013
Location: Georgia
Posts: 4
AverageGuy is an unknown quantity at this point
Re: Raspberry Pi + Camera Module for Vision Processing

Quote:
Originally Posted by jmartin View Post
Using the signal mode of raspistill is something I hadn't thought of. That must be pretty slow though, since you have to wait for the image to be written out and then read it back in. I'm hoping to track the balls so I need a pretty high framerate (and as I mentioned, it looks like I'll have it), but for identifying when the goal is hot in autonomous mode, fetching images with raspistill is probably fast enough and easier to accomplish.
I don't know about writing but reading and processing a 552x91 image (10K file) took .4 seconds real time (.3 user). 552x91 is the approximate size of the rectangle of the goals. We haven't been able to measure the camera write time accurately, but I've been told it's less than 750 ms. Obviously for ball tracking that's not quick enough but as you say detecting hot mode is easily done.

Just curious, how are communicating back to the cRio/LabView?

Jim.
  #11   Spotlight this post!  
Unread 10-01-2014, 11:22
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Re: Raspberry Pi + Camera Module for Vision Processing

I haven't been worrying about that yet. We did it last year over ethernet, so I can always fall back to that. We were thinking though that either using the GPIO pins on the pi to digital in pins on the cRIO, or an I2C or SPI connection might be better.

Last edited by jmartin : 10-01-2014 at 12:55.
  #12   Spotlight this post!  
Unread 10-01-2014, 14:52
merscwog merscwog is offline
Registered User
FRC #0116
 
Join Date: Feb 2013
Location: Herndon, VA
Posts: 11
merscwog is an unknown quantity at this point
Re: Raspberry Pi + Camera Module for Vision Processing

Last year we (Team 116 Epsilon Delta) used 2 USB cameras connected to a RaspberryPi (definitely stretching the power limits).

One camera was to stream video using mjpg_streamer, which barely taxed the CPU (less than 14%) and could easily stream up to 30fps with 720p images (since all the mjpg encoding work was already being done within the USB camera), but we dialed it down to 416x240@10fps since that was sufficient for the human driver and limited the amount of network bandwidth used by the camera to less than 2Mbps.

The second camera was controlled by OpenCV code which processed only a few frames per second at best. With the dedicated Pi camera, there should be significantly increased data rates, but the biggest constraint will still be the OpenCV code itself. Now maybe just trying to find the center of a single object and at most two colors for the balls versus the multiple goals like last year might be faster (we haven't done this yet).

The only real disadvantage I see to the Pi camera is the shorter length of the ribbon cable, so it might be more difficult to position the camera where it really needs to be on the robot.

Recognition for autonomous mode should be easy. If 160x120 can be processed at 30 fps, then assisted catching should be possible. That definitely wasn't the case with our USB-attached cameras and the Pi.

I'm wondering if anyone has attempted to use the BeagleBone Black to do OpenCV processing and see how it compares to the Raspberry Pi. I now think I'm going to have to get a couple of the dedicated cameras.

-Spencer
  #13   Spotlight this post!  
Unread 10-01-2014, 15:12
jmartin jmartin is offline
Registered User
AKA: Joseph Martin
FRC #0172 (Northernforce)
Team Role: Programmer
 
Join Date: Jan 2013
Rookie Year: 2011
Location: Gorham
Posts: 30
jmartin is on a distinguished road
Re: Raspberry Pi + Camera Module for Vision Processing

I've also been wondering about performance on the beaglebone vs raspberry pi. I've already have a beaglebone, but haven't had time to get it set up.

And I have been able to process 160x120 at 30fps. The processing is just an rgb->hsv conversion, performing a threshold on each channel, and then centroiding the resulting binary image, but it seems to work pretty well. Haven't been able to test on an actual competition ball yet though.

The fact that we only ever have to track one ball at a time makes the processing much easier, since we don't have to do any blob separation (though we might want to depending on how good the threshold works)

My hope is to be able to do assisted catching, though another concern with that is the very limited field of view that the camera module has.
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 03:30.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi