Issues Doing Vision Processing On Pi

So I have written a Vision Script to run on the Pi but I have been having issues with incredibly low frame rate and displaying the processed image. As of now it can process one frame a second and publish data to the Networktable at the same rate. The FRC PC Dashboard sees the stream but when viewed it shows up as a black image running at 20 fps. Am I using CvSink and CvSource incorrectly to display the image and is there a way to speed up the rate the Pi Processes images or am I over estimating the Pi’s capabilities? This is also running on a Raspberry Pi 2 Model B
V1.1 on the Latest FRCVision image

Here Is the Code

import json
import time
import sys
import numpy as np
import cv2

from cscore import CameraServer, VideoSource, CvSource, VideoMode, CvSink, UsbCamera
from networktables import NetworkTablesInstance

def TrackTheBall(frame, sd):
    BallLower= (0,253,225)
    BallUpper = (15,255,255)
    #if no frame arrives, the vid is over or camera is unavalible
    if frame is None:
        sd.putNumber('GettingFrameData',False)
    else:
         sd.putNumber('GettingFrameData',True)


    frame = cv2.flip(frame, 1)

    #Blur out the Image
    blurred = cv2.GaussianBlur(frame, (11,11), 0)
    hsv = cv2.cvtColor(blurred, cv2.COLOR_BGR2HSV)
    
    #Make a mask for the pixals that meet yhe HSV filter 
    #then run a bunch of dolations and
    #erosions to remove any small blobs still in the mask
    mask = cv2.inRange(hsv, BallLower, BallUpper)
    mask = cv2.erode(mask, None, iterations = 2)
    mask= cv2.dilate(mask, None, iterations = 2)
    
    #find the Contours in the mask and initialize the
    #current (x,y) center of the ball
    a, cnts , b= cv2.findContours(mask.copy(), cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_NONE)
    center = None
    #only do stuff if a single contor was found
    if len(cnts) > 0:
        #find the largest contour in the mask, then use it
        #to compute the minimum enclosing circle and centroid
        c = max(cnts, key=cv2.contourArea)
        ((x,y), radius) = cv2.minEnclosingCircle(c)
        M = cv2.moments(c)
        center = (int(M["m10"] / M["m00"]), int (M["m01"] / M["m00"]))

        #if the dectected contour has a radius big enough, we will send it
        if radius > 10:
            #draw a circle around the target and publish values to smart dashboard
            cv2.circle(frame, (int(x), int(y)), int(radius), (255,255,8), 2)
            cv2.circle(frame, center, 3, (0,0,225), -1)
            sd.putNumber('X',x)
            sd.putNumber('Y',y)
            sd.putNumber('R', radius)
        else:
            #let the RoboRio Know no target has been detected with -1
            sd.putNumber('X', -x)
            sd.putNumber('Y', -y)
            sd.putNumber('R', -radius)
            
    print("Sent processed frame")
    return frame


if __name__ == "__main__":
    if len(sys.argv) >= 2:
        configFile = sys.argv[1]

    # read configuration
    if not readConfig():
        sys.exit(1)

    # start NetworkTables to send to smartDashboard
    ntinst = NetworkTablesInstance.getDefault()

    print("Setting up NetworkTables client for team {}".format(team))
    ntinst.startClientTeam(team)

    SmartDashBoardValues = ntinst.getTable('SmartDashBoard')
    
    #Start up camera stuff
    print("Connecting to camera")
    cs = CameraServer.getInstance()
    cs.enableLogging()
    Camera = UsbCamera('RPi Camero 0', 0)
    Camera.setResolution(160,120)
    cs.addCamera(Camera)
    
    print("connected")

    #This Is the object we pull the imgs for OpenCV magic
    CvSink = cs.getVideo()
    
    #This will send the process frames to the Driver station
    #allowing the us to see what OpenCV sees
    outputStream = CvSource('Processed Frames', VideoMode.PixelFormat.kBGR, 160, 120, 28)

    #buffer to store img data
    img = np.zeros(shape=(360,240,3), dtype=np.uint8)
    # loop forever
    while True:
        #Quick little FYI, This will throw a Unicode Decode Error first time around
        #Something about a invalid start byte. This is fine, the Program will continue
        # and after a few loops and should start grabing frames from the camera
        GotFrame, img = CvSink.grabFrame(img)
        if GotFrame  == 0:
            outputStream.notifyError(CvSink.getError())
            continue
        img = TrackTheBall(img, SmartDashBoardValues)
        #print(img)
        outputStream.putFrame(img)

FYI, your buffer size doesn’t seem to match the resolution you’re requesting from the camera…

I Tested it Earlier with the camera at 360,240 resolution and it processed frames every two seconds, I lowered the resolution but forgot to lower buffer lol. I just changed it and still have the same issues

Is there a particular reason why you’re copying the mask array? That’d be a very large array to copy.

Also, are you sure you have the dimensions of the array correct? Frames are in row-major order.

The Raspberry Pi 2 is quite a bit slower than a Raspberry 3 B or B+. A Pi 3 is between 50-100% faster on synthetic benchmarks, and a B+ is faster still. You might consider upgrading since you’re doing image processing, not just streaming.

2 Likes

With Copying the array, At the time I was reading on how to use findContours correctly and saw that people used a copy of the mask as the source image for finding contours. Im pretty sure I have the Array dimensions correct, I am declaring the image as
img = np.zeros(shape=(160,120,3), dtype=np.uint8)
since the Image from the camera is 160 pixels wide and 120 pixels tall

We finally raised the funds to buy some raspberry pi 3’s. we get a majority of our funds from nasa but since the gov shutdown we haven’t been able to receive our check so budget’s been tight this build season. We are using a Raspberry Pi 2 just because I already had one.

I was also thinking about having the Pi’s stream video to a jetson tx1 for vision processing since it’s much more powerful and we have one laying around from last year.

You also aren’t using CvSource correctly. You’re creating the source, but never hooking it up to a mjpeg server. I’d recommend you use CameraServer.putVideo() (which returns a CvSource) to both create a source and a server so you can actually connect to it with a dashboard or web browser.

2 Likes

I did that and was able to view the process stream through frcvision.local:1181 on my browser with the Pi wired into my computer. I have wait till tomorrow to test it on my team’s bot. I also removed the Image flip and Blur since I don’t technically need them for my ball following code to work and now the frame rate is much higher.

In your robot code there is driverstation folder in that folder there are codes called Ball_Tracking_DriverStation_V1 and etc. What are they? How do you use it?

We use RaspberryPi like you. But you know that there are a lot of operating systems for raspberry pi. We use raspbian classic but thare are raspian full and raspbian lite as you know. Which one do you do you prefer. Thank you!

Most teams are probably using the FRCVision image as documented on the FIRST Screensteps website.