What happen to AI and Axon for FRC? Removed from wpilib docs 2023

FIRST General Forum
1

Our team invested in Axon/AI last year for experiementation, object detection. Curious to why it’s removed this year?

1 Like
2

It has not been updated in a few years, and we don’t have any active developers to continue supporting it. We didn’t want new users trying to get started only to find that there’s not really any support for it anymore.

Last year’s docs are still available here.

2 Likes
3

Hum, sad, it’s a shame since FTC has support & documentation for use of AI seems like continuing the exposure through FRC would be important. Maybe not so much for this years game where game pieces are easily recognizeable with OpenCV because they don’t move so fast. Any thoughts on future AI support in general?

4

WPILib is a community effort. So if someone steps forward and volunteers to create/update/maintain AI tooling, it will happen… if someone doesn’t, it won’t. There are a small handful of teams that are actively exploring AI methods, but in general it seems like traditional vision processing methods are sufficient for most team needs, so there’s not a lot of community momentum behind maintaining a “standard” AI offering.

7 Likes
5

Got it, tnx for your feedback.

6

We are collecting data from teams around the world here to train an object detection model for Limelight + Coral.

We also support Teachable Machine classification models. Detailed here: Introducing Limelight OS 2023.0

3 Likes
7

Thank god someone is!

Are you going to make this dataset open?

9 Likes
8

Hi Brandon,
Thank you for your efforts and your response! We will gladly donate images, do you want them tagged?

Also, if you are making the models available, we will be able to run them on our Limelight 2+?

Thank you, I am so happy to have help with a path forward.

1 Like
9

@marshall This will not be open. I think the potential downsides with respect to privacy and other concerns outweigh the benefits at the moment.

@nabgilby You will be able to run the models on your Limelight 2+! No tagging is required. You can upload images straight from your phone.

1 Like
10

@marshall I should mention that you will be able to use the model on whatever hardware you want.

4 Likes
11

I’m sad it isn’t completely open and leveraging one of the various ML dataset sites out there (I do understand the reasoning) BUT I’m excited others will be able to use it. You and I should chat and see if the 900 dataset with things like tape corners and stack lights would be of interest.

4 Likes
12

Is it possible to use the exported model from Teachable Machine on WPILibPi? If so any general direction would be helpful. The export model points to Tensor Flow.

1 Like
13

Yes, I just got this working yesterday. It was a lot easier than I expected.

You have to install the Edge TPU runtime and the pycoral API, then drop a dozen lines of Python into a vision application.

I am working on a write-up of this, and will share my Python script if anyone is interested.

14

Yes I would love to see that. We have a Coral TPU so it would be very helpful to learn how to implement the easy to use Teachable Machine models.

1 Like
15

Is the model already available for testing?

16

I trained a very simple model using the Teachable Machines site by holding game pieces in front of my webcam. (Better training data is needed - this was a 30-minute experiment.) As far as I am aware, the LimeLight model has not been released yet, as they are still crowdsourcing training data.

Here’s my model if you want to try it, but it’s not very good. It has outputs for “upright cone”, “tipped cone” and “cube.”

converted_edgetpu.zip (900.9 KB)

(EDIT Sorry, meant to reply to @Qworg )

1 Like
17

These are the steps to run Tensorflow on WPILibPi:

1. Install the Edge TPU Runtime and pycoral API:

  • Start with a fresh WPILibPi image.

  • Connect a keyboard and monitor to the Pi, and connect it to the internet over either wifi or using an ethernet cable.

  • Log into the pi: username “pi” password: “raspberry”

  • Run the following command sequence: (NOTE: I got some errors reported during the “apt-get update” but everything worked, so…?)

    echo “deb Index of /apt// coral-edgetpu-stable main” | sudo tee /etc/apt/sources.list.d/coral-edgetpu.list
    curl https://packages.cloud.google.com/apt/doc/apt-key.gpg | sudo apt-key add -
    sudo apt-get update
    sudo apt-get install python3-pycoral

There is one more driver to install, but you need to decide if you will run the Coral at normal clock rate, or overclock it at double speed.

  • For normal clock rate:

    sudo apt-get install libedgetpu1-std

  • For 2X clock rate:

    sudo apt-get install libedgetpu1-max

The WPILibPi image is now ready to run your models. Make sure your Coral TPU is plugged in.

2. Train an Image model at Teachable Machines.

  • Define labels, provide training images, train and test your model
  • When you are satisfied, export your model as Tensorflow Lite with a target of Edge TPU. This will download a .zip file containing your .tfilte model and a text file containing the label definitions.
  • Upload the .zip file onto the Pi and unzip it into the /home/pi folder. (You can do this using the WPILibPi configuration webpage Application tab’s File Upload feature.)

3. Upload the Vision Application

On the WPILibPi configuration webpage’s Application tab, set the Vision Application Configuration to use an “Uploaded python file” and then upload the attached python script:

(NOTE: This is a very basic script that runs the model, but does not publish anything to the network tables. Classifications are printed to the console output.)

from cscore import CameraServer
from ntcore import NetworkTableInstance, EventFlags

import cv2
import json
import numpy as np
import time

# Python code to run the classifier:
import re
import os
from pycoral.utils.dataset import read_label_file
from pycoral.utils.edgetpu import make_interpreter
from pycoral.adapters import common
from pycoral.adapters import classify

# the TFLite converted to be used with edgetpu
modelPath = "./model_edgetpu.tflite"
# The path to labels.txt that was downloaded with your model
labelPath = "./labels.txt"

team = None
server = False

# This function takes in a TFLite Interpter and Image, and returns classifications
def classifyImage(interpreter, image):
    size = common.input_size(interpreter)
    common.set_input(interpreter, cv2.resize(image, size, fx=0, fy=0,
                                             interpolation=cv2.INTER_CUBIC))
    interpreter.invoke()
    return classify.get_classes(interpreter)


def main():
    with open('/boot/frc.json') as f:
        config = json.load(f)
    camera = config['cameras'][0]

    width = camera['width']
    height = camera['height']

    CameraServer.startAutomaticCapture()

    input_stream = CameraServer.getVideo()
    output_stream = CameraServer.putVideo('Processed', width, height)
    img = np.zeros(shape=(height, width, 3), dtype=np.uint8)

    # Table for vision output information
    # start NetworkTables
    ntinst = NetworkTableInstance.getDefault()
    if server:
        print("Setting up NetworkTables server")
        ntinst.startServer()
    else:
        print("Setting up NetworkTables client for team {}".format(team))
        ntinst.startClient4("wpilibpi")
        #ntinst.setServerTeam(team)
        ntinst.startDSClient()
    #vision_nt = NetworkTables.getTable('Vision')
    vision_nt = ntinst.getTable('Vision')

    # Load your model onto the TF Lite Interpreter
    interpreter = make_interpreter(modelPath)
    interpreter.allocate_tensors()
    labels = read_label_file(labelPath)

    # Wait for NetworkTables to start
    time.sleep(0.5)

    prev_time = time.time()
    while True:
        start_time = time.time()

        frame_time, input_img = input_stream.grabFrame(img)
        output_img = np.copy(input_img)

        # Notify output of error and skip iteration
        if frame_time == 0:
            output_stream.notifyError(input_stream.getError())
            continue

        # Flip image so it matches the training input
        ##frame = cv2.flip(frame, 1)
        # Classify and display image
        results = classifyImage(interpreter, input_img)
        confidence = round(100*results[0].score, 3)
        tagline = f"{labels[results[0].id]} ({confidence} %)"
        print(f'Label: {labels[results[0].id]}, Score: {results[0].score}')
        cv2.putText(output_img, tagline, (0, height-20), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255))

        # TODO: Output results to the Network Table

        processing_time = start_time - prev_time
        prev_time = start_time

        fps = 1 / processing_time
        cv2.putText(output_img, str(round(fps, 1)), (0, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255))
        output_stream.putFrame(output_img)

main()
18

We are also experimenting with multiple possible options for machine learning on rPi for 2023. @veg what fps are you getting with the latest wpilibpi? Are you using an rPi 4b?

19

I am using an RPi 4B with 2GB RAM and a Coral TPU. When running the model I shared above I get about 7 - 15 fps, sometimes 30.

This is with a Microsoft LifeCam running 160 x 120 @ 30 fps.

I should also mention that I am running the Coral at standard speed, not overclocked.

20

At this point I will say the Teachable Machines models are pretty cool. The big difference I see between them and Axon is that TM models don’t provide a bounding box around recognized objects. (This makes sense, since the training data does not include bounding box information.)

So a TM model can tell you what it sees, but cant tell you where it is in the frame. ie: “Somewhere in this picture is a tipped cone.”

This has me wondering if it might be worth the (considerable) extra effort to create a database of tagged images.