Using AprilTag on Raspberry Pi

I’m trying to use the new april tag features on a raspberry pi 4 (with the wpipi image), using java. We downloaded the provided Java multi-camera server to use as a base, but cannot figure out how to import the april tag libraries in VS Code. The VS code add dependencies feature tool is not active.

We triedmodifying the settings.json file to point to the apriltag jar file in the edu.wpi folder, and also tried copying the jar file in the project directory. But no luck.

I made a new java application and added all the needed jar files and export a final jar file, but then it doesn’t run on the raspberry pi. We get the error message:

java.lang.UnsupportedClassVersionError: App has been compiled by a more recent version of the Java Runtime (class file version 61.0), this version of the Java Runtime only recognizes class file versions up to 55.0

Any suggestions on how to get this to run in Java?

We are also experimenting with PhotonVision. It works great on a laptop, but when we try to run it on the raspberry pi with the photonvision image, we get no camera feed and it hangs. We are using a LifeCam, but will try a PiCam next week. Anyone else have issues with photonvision on the raspberry pi?

1 Like

I was seeing this same thing yesterday. PhotonVision on a RPi 4 with MS LifeCam was running and recognizing tags, but I could not see either the raw or processed video feeds in the browser.

What was your networking setup at the time? If it wasn’t a network switch connected to the first port of the radio you’ll likely have trouble viewing the streams.

No robot or radio involved. I had the pi and my mac plugged into a switch.

We had the raspberry pi connected directly into the windows laptop

Yep unfortunately no surprise there it didn’t work. Link local connections just generally sorta suck. See our docs:

3 Likes

That fixed the issue with our photonvision issue- thank you ! I had read
through the camera trouble shooting doc but not the networking one.

1 Like

We’ve not yet released a WPILibPi image for 2023, and 2023 things won’t work on a 2022 or older version. I’ve been working on it, should release in the next couple days.

3 Likes

Thank you - I’m looking forward to it!
We have a couple of students interested in writing their own custom code for the pi, and the wpi image makes things so much easier.

WPILibPi 2023.1.1 has been released! Please give it a go.

3 Likes

Thank you! Please treat yourself to some sleep before you bother yourself with this, but… :stuck_out_tongue:

This runs with no obvious problems in (superficial) bench testing on an RPi 4 with MS LifeCam. Looks good so far!

So I am trying to script up an AprilTag detector just “for fun” and am choking at the first step. I naively try to “import apriltag” and get the following:

  File "/home/pi/uploaded.py", line 7, in <module>
    import apriltag
ModuleNotFoundError: No module named 'apriltag'

The release notes lead me to believe the package should be included in this release. I’m sure I am making some simple mistake.

We include robotpy_apriltag, but not any other apriltag library.

1 Like

Wonderful - going to try it out tonight. Thank you so much - we have been very impressed at how much easier the raspberry pi image makes everything

1 Like

It would be really cool if someone could take the new WPILib apriltag examples (Java and C++) and merge them into the WPILibPi examples and PR it. I just haven’t had the time to do it.

I can give that a shot tomorrow

Thanks for the guidance! I was able to cobble together a Python sample that locates April Tags on the latest WPILibPi. It is based on an old vision sample that I attempted to update to 2023 standards.

I’m getting steady detection at about 15 fps on a Pi 4 with MS LifeCam at 160 x 120. Not bad for a crude science experiment…

PythonAprilTagDetection

I will looking at the C++ and Java samples next, if just because it seems like something I ought to be able to do.

Here’s the working python script. Gentle feedback is welcome!

#!/usr/bin/env python3

from cscore import CameraServer
from ntcore import NetworkTableInstance, EventFlags

import cv2
import json
import numpy as np
import time

import robotpy_apriltag
from wpimath.geometry import Transform3d
import math

team = None
server = False


def main():
    with open('/boot/frc.json') as f:
        config = json.load(f)
    camera = config['cameras'][0]

    width = camera['width']
    height = camera['height']

    CameraServer.startAutomaticCapture()

    input_stream = CameraServer.getVideo()
    output_stream = CameraServer.putVideo('Processed', width, height)
    img = np.zeros(shape=(height, width, 3), dtype=np.uint8)

    # start NetworkTables
    ntinst = NetworkTableInstance.getDefault()
    if server:
        print("Setting up NetworkTables server")
        ntinst.startServer()
    else:
        print("Setting up NetworkTables client for team {}".format(team))
        ntinst.startClient4("wpilibpi")
        #ntinst.setServerTeam(team)
        ntinst.startDSClient()

    # Table for vision output information
    vision_nt = ntinst.getTable('Vision')

    # Wait for NetworkTables to start
    time.sleep(0.5)

    prev_time = time.time()
    while True:
        start_time = time.time()

        frame_time, input_img = input_stream.grabFrame(img)
        output_img = np.copy(input_img)

        # Coordinates of found targets, for NT output:
        x_list = []
        y_list = []
        id_list = []

        # Notify output of error and skip iteration
        if frame_time == 0:
            output_stream.notifyError(input_stream.getError())
            continue

        # April Tag detection:
        detector = robotpy_apriltag.AprilTagDetector()
        detector.addFamily("tag16h5")
        #detector.addFamily("tag36h11")
        estimator = robotpy_apriltag.AprilTagPoseEstimator(
            robotpy_apriltag.AprilTagPoseEstimator.Config(
                0.2, 500, 500, width / 2.0, height / 2.0
            )
        )

        # Detect apriltag
        DETECTION_MARGIN_THRESHOLD = 100
        DETECTION_ITERATIONS = 50

        gray = cv2.cvtColor(input_img, cv2.COLOR_BGR2GRAY)
        tag_info = detector.detect(gray)
        filter_tags = [tag for tag in tag_info if tag.getDecisionMargin() > DETECTION_MARGIN_THRESHOLD]

        # OPTIONAL: Ignore any tags not in the set used on the 2023 FRC field:
        #filter_tags = [tag for tag in filter_tags if ((tag.getId() > 0) & (tag.getId() < 9))]

        for tag in filter_tags:

            est = estimator.estimateOrthogonalIteration(tag, DETECTION_ITERATIONS)
            pose = est.pose1

            tag_id = tag.getId()
            center = tag.getCenter()
            #hamming = tag.getHamming()
            #decision_margin = tag.getDecisionMargin()
            print(f"{tag_id}: {pose}")

            # Highlight the edges of all recognized tags and label them with their IDs:

            if ((tag_id > 0) & (tag_id < 9)):
                col_box = (0,255,0)
                col_txt = (255,255,255)
            else:
                col_box = (0,0,255)
                col_txt = (0,255,255)

            # Draw a frame around the tag:
            corner0 = (int(tag.getCorner(0).x), int(tag.getCorner(0).y))
            corner1 = (int(tag.getCorner(1).x), int(tag.getCorner(1).y))
            corner2 = (int(tag.getCorner(2).x), int(tag.getCorner(2).y))
            corner3 = (int(tag.getCorner(3).x), int(tag.getCorner(3).y))
            cv2.line(output_img, corner0, corner1, color = col_box, thickness = 2)
            cv2.line(output_img, corner1, corner2, color = col_box, thickness = 2)
            cv2.line(output_img, corner2, corner3, color = col_box, thickness = 2)
            cv2.line(output_img, corner3, corner0, color = col_box, thickness = 2)

            # Label the tag with the ID:
            cv2.putText(output_img, f"{tag_id}", (int(center.x), int(center.y)), cv2.FONT_HERSHEY_SIMPLEX, 1, col_txt, 2)

            x_list.append((center.x - width / 2) / (width / 2))
            y_list.append((center.y - width / 2) / (width / 2))
            id_list.append(tag_id)
        
        vision_nt.putNumberArray('target_x', x_list)
        vision_nt.putNumberArray('target_y', y_list)
        vision_nt.putNumberArray('target_id', id_list)

        processing_time = start_time - prev_time
        prev_time = start_time

        fps = 1 / processing_time
        cv2.putText(output_img, str(round(fps, 1)), (0, 40), cv2.FONT_HERSHEY_SIMPLEX, 1, (255, 255, 255))
        output_stream.putFrame(output_img)

main()
1 Like

I tried merging the apriltag detecor example into the java-multicamera-server but it gives an error when I try to run it on the Pi. I confirmed that java-multicamera-server runs on my pi (model 4), but adding april tag detector causes it to crash with a “could not load class edu/wpi/first/math/geometry/Rotation3d” error.

I traced it to creating an instance of AprilTagDetector. I did include the import for Rotation3d. The complete error message is below also.

import edu.wpi.first.apriltag.AprilTagDetector;
import edu.wpi.first.math.geometry.Rotation3d;
import edu.wpi.first.math.geometry.Rotation2d;

    if (cameras.size() >= 1) {

      AprilTagDetector detector = new AprilTagDetector();

      VisionThread visionThread = new VisionThread(cameras.get(0), 
              new MyPipeline(), pipeline -> {
                
      });

Here’s the complete error trace from the pi.

Waiting 5 seconds...

Setting up NetworkTables client for team 294

NT: starting network client

Starting camera 'rPi Camera 0' on /dev/video0

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: SetConfigJson: setting video mode to pixelFormat 1, width 160, height 120, fps 30

Waiting 5 seconds...

Setting up NetworkTables client for team 294

NT: starting network client

Starting camera 'rPi Camera 0' on /dev/video0

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: SetConfigJson: setting video mode to pixelFormat 1, width 160, height 120, fps 30

could not load class edu/wpi/first/math/geometry/Rotation3d

Exception in thread "main" java.lang.NoClassDefFoundError: org/ejml/data/Matrix

	at java.base/jdk.internal.loader.NativeLibraries.load(Native Method)

	at java.base/jdk.internal.loader.NativeLibraries$NativeLibraryImpl.open(NativeLibraries.java:388)
	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:232)

	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:174)

	at java.base/jdk.internal.loader.NativeLibraries.findFromPaths(NativeLibraries.java:315)

	at java.base/jdk.internal.loader.NativeLibraries.loadLibrary(NativeLibraries.java:287)
	at java.base/java.lang.ClassLoader.loadLibrary(ClassLoader.java:2422)
	at java.base/java.lang.Runtime.loadLibrary0(Runtime.java:818)
	at java.base/java.lang.System.loadLibrary(System.java:1989)
	at edu.wpi.first.util.RuntimeLoader.loadLibrary(RuntimeLoader.java:86)
	at edu.wpi.first.apriltag.jni.AprilTagJNI.<clinit>(AprilTagJNI.java:38)
	at edu.wpi.first.apriltag.AprilTagDetector.<init>(AprilTagDetector.java:186)
	at Main.main(Main.java:339)
Caused by: java.lang.ClassNotFoundException: org.ejml.data.Matrix

	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)

	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:520)

	... 13 more

Waiting 5 seconds...

It looks like ejml isn’t getting included the way it needs to be, will look at it this evening.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.