PhotonVision v2024.2.0
What’s Changed
First big feature update of 2024! Brings (still incubating!) object detection using RKNN with a hard-coded YOLOv5 model trained on notes, and an (also new) LimeLight 3 image. Now that features work we’ll be changing gears to focus on documenting new things. As always please reach out on Discord with journalctl logs if you encounter any issues!
- Add RKNN / Object Detection Pipeline by @mdurrani808 in Add RKNN / Object Detection Pipeline by mdurrani808 · Pull Request #1144 · PhotonVision/photonvision · GitHub
- Add LL3 image by @BytingBulldogs3539 in LL3 by BytingBulldogs3539 · Pull Request #1166 · PhotonVision/photonvision · GitHub
- And other changes included since v2024.1.4:
- Load libquadmath on Windows by @mcm001 in Load libquadmath on Windows by mcm001 · Pull Request #1163 · PhotonVision/photonvision · GitHub (Mrcal calibration fixes for Windows)
- Bind-mount repo in image builder by @mcm001 in Bind-mount repo in image builder by mcm001 · Pull Request #1157 · PhotonVision/photonvision · GitHub (reduces total image size again)
- Update spotless by @rzblue in Update spotless by rzblue · Pull Request #1162 · PhotonVision/photonvision · GitHub
- [photon-lib java] Make targeting classes extend ProtobufSerializable by @ArchB1W in [photon-lib java] Fix classes with protobuf support not "announcing it" and as a result causing crashes with AdvantageKit by ArchB1W · Pull Request #1156 · PhotonVision/photonvision · GitHub
The nitty gritty
RKNN is the neural network accelerator hardware built into rockchip CPUs using on SBCs like the Orange Pi 5. We’re able to use this hardware to massively accelerate certain math operations like those needed for running ML-based object detection. The code for that lives here, and publishes to Maven. Our pre-trained note model lives here, and NeuralNetworkModelManager deals with extracting that to disk. If a file named “note-640-640-yolov5s.rknn” and labels.txt does not exist at photonvision_config/models/, it’ll be extracted from the JAR – otherwise, the one already on disk will be used. This technically allows power users to replace the model and label files with new ones without rebuilding photon, though this feature is still largely untested, too.
I’ve seen latencies of 20-25ms with our current model, although further performance gains seem possible with model updates. Note that the note detector runs at 640x640 pixels – images larger or smaller are “letterboxed” up/down to this resolution. Picking a camera resolution with a width of 640px seems reasonable.
Full Changelog: Comparing v2024.1.5...v2024.2.0 · PhotonVision/photonvision · GitHub