This year in FRC, we want to detect the game piece. We have a Raspberry Pi 4 and a Microsoft HD 3000 camera, and we are using the PhotonVision library. Can we achieve this setup to detect the game piece? If we can’t directly use PhotonVision for this task, how else can we achieve it? What else can we do/use to make it work with PhotonVision?
If we can’t use PhotonVision for this task, how can we send data from the Raspberry Pi to NetworkTables?
PhotonVision neural network note detection is currently only for the Orange Pi 5 ( or 5 Plus/Pro etc). You could maybe use colored shape detection depending on what the game piece is next year.
If PhotonVision’s neural network note detection is only supported on the Orange Pi 5 series, how can we achieve this task using a Raspberry Pi? Are there any alternative methods or libraries that can be used for game piece detection with our Raspberry Pi 4 setup?
You can definitely get game piece detection with that set up. I was able to get note detection working on the same camera and an orange pi in 2024 and cube/cone detection on a limelight 2 in 2023 both using the colored shape settings in photon vision
Thank you for the helpful information!
The color and shape processing in PhotonVision has a good chance of being able to detect the currently unknown 2025 game pieces.
Before PhotonVision, for the Raspberry Pi, there was WPILIbPi. It’s still around and usable if you want to have more control over your vision solution. It still does a lot of the heavy lifting in terms of providing a vision framework, but you write code to implement your specific solution.
For a large majority of teams, PhotonVision will be the way to go. However, WPILibPi is also an option.
I don’t see “boat bumpers” as a shape to detect so writing custom code for a shape isn’t too hard using the WPILibPi setup (my team used to do that until the students wanted no effort simple and got the LL).
(I did have to fix the OpenCV match shapes routine as it didn’t have proper convergence criteria and I doubt my fix is OpenCV yet but I didn’t check.)
Here’s “blue circle” detection in PV (there was a lot of jitter but it worked)
We’re currently working on a Python script to enable running PyTorch models on x86 mini PCs. The script communicates to PhotonVision via gRPC, and would work just like the existing object detection pipeline for Orange Pis.
Technically I’ve gotten ML based game piece detection working, years ago, on a pi 3B, but it was 5 seconds per frame, so not useful on the robot. But I had no idea how to build models properly (still don’t, but I was clueless then) and was mostly doing it because we had a student that was off-the-freaking-chain smart in everything and in doing this he got to play with some GCP VMs with monster GPUs at the time. Fun!
Anyhoo, the PhotonVison docs hit upon what the main problem is early on: About Object Detection - PhotonVision Docs
They build their models to run on the NPU in the Orange Pi 5 series which is a chip optimized to do the math behind AI/ML quick and efficiently. It’s like a GPU except whereas a GPU usually has hundreds or thousands of cores, and dedicated memory, the NPU is just a single core and sharing the main RAM. But the NPU can do linear algebra wicked fast compared to a CPU, much like a GPU.
Which makes an NPU really nice for inference of a model at runtime. Especially on a robot because NPUs are energy efficient.
But, what you can do, in response to “If we can’t use PhotonVision for this task, how can we send data from the Raspberry Pi to NetworkTables?” is put the pi4 in the robot, on the network, use the pynetworktables library to publish data to network tables, and drop a Python script on the pi4 setup so that systemd starts it up as a daemon of sorts and let it handle processing the camera. You can run ML inference on the pi4 CPU if you have a proper model.
You could probably convert the rknn format to onnx. I don’t recall doing this but recall trying to go the other way and getting frustrated. Though I was also trying to quantize the model at the same time. And it was not robot related.
But that also brings me to quantizing the model. If you really want to have a go at ML on a pi4 you’ll want to be going this route. A model is a series of weights which are really just matrices of numbers. And those numbers are usually 32 bit or 16 bit floating point numbers. And NPUs and GPUs love floating point numbers! Wee hoo! Though they’re even faster with integers.
CPUs aren’t as fond of floats. It’s hilarious how many PCs just din’t do float math. And the ARM series in the pi’s aren’t strangers to just leaving out some math operations on the chip like division. Instead they optimize for common operations, like integer math and multiplication.
Anyhoo quantization is when you take those fp32 or fp16 values and scale them into 0-255 integer values. You lose resolution on the inference of what is or isn’t an object but it goes faster. And you rebuild the model so it uses those new weights and the CPU has a much easier time with it all. But it still isn’t great.
All that said I think you’d be better with every other option suggested. What I outlined above are things I’ve learned from mistakes.
