2013 Lessons Learned for a Vision Co-Processor

[Jerry Ballard]

On the eve of Kick-Off, we'd thought it would be a good time to post our lessons learned from last year's competition.

Team 456 (Siege Robotics)

Hardware: Pandaboard with Linux Ubuntu, Logitech C110 Webcam
Software: OpenCV, yavta, iniparser, mongoose
Code: C

1) Control your exposure: For consistent results, avoid over saturation of bright targets, set and control the exposure (integration time) setting on the webcam. We found that a 8ms integration time worked best in most settings. Camera exposure settings were set using yavta: (https://github.com/fastr/yavta). Remember, many webcams have auto-exposure set as a default. The auto-exposure setting won't allow you to consistently identify targets in changing lighting conditions (see #2).

2) Do calibrate at competition: Each competition arena will have different lighting. Take advantage of the time provided (before competition begins) and calibrate camera to those settings. We setup a simple executable script that recorded movies at different camera exposures that we then ran against our target tracking systems off the field. I've attached frame shots of the same field at different camera exposures. Corresponding movies are available, let me know if you would like a copy of them.

3) Use a configuration file: We found that using a configuration file allowed us to modify target tracking settings during competition without having to recompile code. Recompiling code during competition is rarely a good thing.

4) OpenCV isn't always optimized for FRC Competition: In the pursuit of faster frame processing, we found that going to the traditional OpenCV RGB->HSV->Threshold(Value)&Threshold(Hue) pipeline was a major time-eater. We found a significant gain in performance from 15 fps to 22 fps by doing the thresholding on Value and Hue while converting from RGB. This allowed a reduction in computation load by eliminating the Saturation calculation and skipping obvious non-target pixels. If requested, we can post the algorithm here or it is also available on our github site. (GitHub is awesome!)

5) Avoid double precision: If our input source is a 640x480 image, most if not all calculations can be done with integer and single precision arithmetic. Using doubles adds more processing time = reduced fps processing.

6) Data transfer via HTTP was questionable: We used the Mongoose webserver code to transfer target angular coordinates to the control system (https://github.com/cesanta/mongoose). Coding-wise, the Mongoose code was easy to compile, simple to integrate, and multi-threaded (that is another discussion). During competition, sometimes it appeared that we would not get target lock as quick as was tested and we suspect that it was related to the HTTP communication between the PandaBoard and the CRIO. We are still trying to decide the best way to communicate between the vision co-processor and the CRIO (looking at UDP next).

[billylo]

thanks for sharing... #priceless

[mechanical_robot]

Thankyou for sharing this. Very interesting as I am considering doing a OpenCV project with my new Raspberry Pi

[yash101]

Thanks. Can you post a link to the algorithm you used? Did you use the web server to communicate with the cRIO or just the DS?

[JesseK]

UDP would work fine. We use raw network programming like that for a complex data structure between the robot and driver's station. We could use something fancy like Protobufs, CORBA, or other WSDL-type middleware - but hardcoding the encode/decode order of 4-byte values also works and is what we do.

As for vision itself, I have no comments

[Jerry Ballard]

Originally Posted by yash101

Thanks. Can you post a link to the algorithm you used? Did you use the web server to communicate with the cRIO or just the DS?

Communication was just with the cRIO LabView controller code.

Here's the algorithm we used for the color conversion:

/*
**  Local MACRO defines
*/
#define MIN3(x,y,z)  ((y) <= (z) ? \
                         ((x) <= (y) ? (x) : (y)) \
                     : \
                         ((x) <= (z) ? (x) : (z)))

#define MAX3(x,y,z)  ((y) >= (z) ? \
                         ((x) >= (y) ? (x) : (y)) \
                     : \
                         ((x) >= (z) ? (x) : (z)))

/* 
** prototypes of functions
*/

void T456_change_RGB_to_binary( IplImage *, CvMat *, int, int, int);
void T456_filter_image( unsigned char , unsigned char , unsigned char , 
                 unsigned char *, int, int, int);

/*
**  Filter RGB image, HSV convert, threshold, etc...
*/

void T456_change_RGB_to_binary( IplImage *rgb, CvMat *binary,
                                int val_thresh, int hue_mid_thresh, 
                                int hue_mid_span )
{
  register int y;
  register unsigned char r,g,b;
  register char *data;
  register uchar *bin_data;

  register int total_vals;

  /* 
  **  Point the data pointer into the beginning of the image data
  */
  data = (char*)rgb->imageData;

  /*
  **  Point the output binary image pointer to the beginning of the image
  */
  bin_data = (uchar*)binary->data.ptr;
  
  total_vals = rgb->height * rgb->width;

  for ( y = 0; y < total_vals; y++ )  /* rows */
  {
     /* grab the bgr values */
     b = data[0];
     g = data[1];
     r = data[2];
     data += 3;

     T456_filter_image( r,g,b, bin_data, val_thresh, 
                        hue_mid_thresh, hue_mid_span );

     /* increment output pointer */
     bin_data++;
  }

}


void T456_filter_image( unsigned char r, unsigned char g, unsigned char b, 
                 unsigned char *binary ,
                 int val_thresh, int hue_mid_thresh, int hue_mid_span)
{
   unsigned char rgb_min, rgb_max, rgb_diff;
   unsigned char hue = 0; 
   unsigned char val = 0; 

   /*
   **  set the default return value to zero 
   */
   *binary = 0;

   /*
   **  get the min and max values of the RGB
   **   pixel
   */
   rgb_min = MIN3( r, g, b );
   rgb_max = MAX3( r, g, b );

   rgb_diff = rgb_max - rgb_min;

   val = rgb_max;

   /* 
   **  This is the trivial case:
   **    zero pixels or value is less than VAL_THRESH
   */
   if ( (val == 0) || (val < val_thresh) ) {
      return;   /* binary = 0 */
   }

   /*
   **  Zero out white pixels 
   **   WARNING (use only if camera is not oversaturated)
   */
   if ( (val >= val_thresh) && (rgb_diff == 0 ) ) 
   {
      *binary = 0;
      return;
   }

   /* 
   ** Compute hue 
   */
   if (rgb_max == r) {
       hue = 0 + 43 * (g - b)/(rgb_diff);
   } else if (rgb_max == g) {
       hue = 85 + 43*(b - r)/(rgb_diff);
   } else /* rgb_max == b */ {
       hue = 171 + 43*(r - g)/(rgb_diff);
   }

   /* 
   **  to get to this point, val > val_thresh
   */
   if (    (hue >= ( hue_mid_thresh - hue_mid_span)) 
       && ( hue <= ( hue_mid_thresh + hue_mid_span) ) )
   {
       *binary = 255;
   }
 
   return;
}

[Joe Ross]

Your tips are useful for vision processing in general. Do you have any tips specific to using a Co-Processor, ie how to power it, how to shut it down cleanly, how to get a development environment up and running quickly, etc?

You can enclose your code in [ code] tags to keep it formatted.

[Jerry Ballard]

Thanks Joe for the formatting tip.

Here are a few more lessons learned related to your questions:

7) Stable and reliable power is required: During competition, the system voltage can vary significantly (below 10V) due to the varying demands of the other powered components. If input voltage dropped that much the PandaBoard would drop power and reboot (not good). So we used the MiniBox DC-DC converter (http://www.mini-box.com/DCDC-USB?sc=8&category=981) to solve this problem. The only issue (we believe) we had in competition with power was probably due to a loose power connection into the PandaBoard.

8) Hard Stop and Rebuild during competition: During the build season, we made the decision not to find a graceful method to shutdown the system during competition. Instead we let the system hard stop when the power was shutdown and then did a reboot and fsck'ed (file system checked) between matches. Also, we maintained multiple copies of the complete OS on memory cards and often did card swaps between short match times. Swaping OS memory cards allowed for some simple diagnostics of target tracking and provided redundancy in case of severe system crash.

9) Diagnostic photos are good: During the matches, we captured and saved frames from the camera once a second (sometimes less) to help us determine how the targeting system was doing. It turned out that these images were very useful to the pilot/copilot of the robot to quickly replay the previous match. Visual cueing from these images helped the students recall better what happened during the match (from the robot's point of view). I've attached an example image below. The blue circle is the aim point of the shooter, cross-hairs identify targets in range, the red dot is the predicted frisbee hit, and the green circle is the target selected.

10) Raspberry Pi didn't work for us: We spent a lot of time trying to get the RPi to run the targeting code. But as written, our code wasn't able to achieve the fps speed needed. We had set a goal of 15 fps as a minimum. The most we could squeeze out of the RPi was 10 fps. We all love the RPi but in this case we weren't able to get the required speed.

[SoftwareBug2.0]

Originally Posted by Jerry Ballard

10) Raspberry Pi didn't work for us: We spent a lot of time trying to get the RPi to run the targeting code. But as written, our code wasn't able to achieve the fps speed needed. We had set a goal of 15 fps as a minimum. The most we could squeeze out of the RPi was 10 fps. We all love the RPi but in this case we weren't able to get the required speed.

How did you determine what speed you needed? Was 10 Hz too slow just because it wasn't meeting your goal of 15 or did you try it and not like the results?

[MikeE]

Originally Posted by Jerry Ballard

8) Hard Stop and Rebuild during competition: During the build season, we made the decision not to find a graceful method to shutdown the system during competition. Instead we let the system hard stop when the power was shutdown and then did a reboot and fsck'ed (file system checked) between matches. Also, we maintained multiple copies of the complete OS on memory cards and often did card swaps between short match times. Swaping OS memory cards allowed for some simple diagnostics of target tracking and provided redundancy in case of severe system crash.

9) Diagnostic photos are good: During the matches, we captured and saved frames from the camera once a second (sometimes less) to help us determine how the targeting system was doing. It turned out that these images were very useful to the pilot/copilot of the robot to quickly replay the previous match. Visual cueing from these images helped the students recall better what happened during the match (from the robot's point of view). I've attached an example image below. The blue circle is the aim point of the shooter, cross-hairs identify targets in range, the red dot is the predicted frisbee hit, and the green circle is the target selected.

Thanks for sharing your experiences.
Your success with hard stop is particularly helpful as clean shutdown was a significant concern. I like the pragmatic approach of having spare memory cards to swap between matches.

Great tip about the diagnostic images.

[virtuald]

Originally Posted by Jerry Ballard

Thanks Joe for the formatting tip.

9) Diagnostic photos are good: During the matches, we captured and saved frames from the camera once a second (sometimes less) to help us determine how the targeting system was doing.

We did this as well. It was great for tuning purposes.

[faust1706]

Originally Posted by virtuald

We did this as well. It was great for tuning purposes.

Also, save off all your variable that you need to a logfile. I've done this the past 2 years. You can write an octave script that will make graphs out of the solutions. Also, you can REPRINT your solutions onto the corresponding image. This was really cool. We went out with a cart with a computer and camera during calibration and saved images 10 times a second, then created a "video" that we could show in the pits to curious students and mentors and show the judges it working on the field.

Also, if you can, go to another regional you are not competing at and take pictures so you can get an idea of what to expect at your regional. We have also done this for the past 2 years.

[Greg McKaskle]

Since we are talking about logging data for review, has anyone played with the dashboard template that includes the ability to record and playback a match? It logs an avi and another file that includes the contents of the network table, joystick data, I/O values, etc. It would also work to test out changes to dashboard-based vision processing. Your code under development would be processing the AVI instead of the live version.

Greg McKaskle

[virtuald]

Originally Posted by Greg McKaskle

Since we are talking about logging data for review, has anyone played with the dashboard template that includes the ability to record and playback a match? It logs an avi and another file that includes the contents of the network table, joystick data, I/O values, etc. It would also work to test out changes to dashboard-based vision processing. Your code under development would be processing the AVI instead of the live version.

Greg McKaskle

That sounds like a nice capability. However, not useful for anyone not using the LabVIEW dashboard.

I have been thinking about logging all of the data and doing something with it for awhile now, but I've never actually gotten around to doing it.

[billbo911]

Originally Posted by Joe Ross

Your tips are useful for vision processing in general. Do you have any tips specific to using a Co-Processor, ie how to power it, how to shut it down cleanly....

Our approach to powering the co-processor was to use the same +5 vdc source that is provided on the PDB for powering the Axis camera. We used a USB webcam, so the output was free.
To power down gracefully, we created a vi that only ran during "Disabled mode" and was activated by pressing a designated button on one of the joysticks. The vi initiated a socket connection to our co-processor. On the co-processor, we had a routine that ran and constantly listened for a socket request. When it received a request, it ran a "shutdown" command. This was done at the end of every match.