YOLOv8 in FRC

We saw this video and it gave us some questions:
Are there any groups here that managed to use YOLOv8 for image processing?
Did you use OrangePi for this, or did you use another component?
What are the advantages and disadvantages of this in relation to PhotonVision?

And if you use it, do you have a tip on where to start or in general a tip for getting started?

1 Like

We have a custom fork of PhotonVisiom with ML running on the RKNN accelerator (only for an OrangePi).
We used it this year to detect notes and it has worked amazing. We’ve been using autonomous collection almost always and it worked without any problems.
To get started, you need an OrangePi, and to put the fork image / pv image on it.
After that, find or create a good training dataset. There are a lot we found / created.
From there, that’s pretty much it. You can play around with confidence threshold / normal vision settings, and access the data through the normal PV library using camera.getLatestResult().getBestTarget()
This is our implementation of the camera in code. (note this is with advantage kit style code, so it may be a bit different from what you’re used to.)

Could you elaborate on what about this video caused you to ask these questions? I’m somehow missing the connection (but still think the topic is super cool)

1 Like

I’ve never seen YOLOv8 used in FRC, so I wanted to check how hard it is to learn and use it

I think his question was what about the video sparked that curiosity

1 Like

I think you’re asking more of how teams are using/doing machine learning, many teams are now using ML on their robots using a custom setup, LimeLight with a Google Coral, or PhotonVision Object Detection on the Orange Pi 5.

This year our team used PhotonVision’s object detection. All you need to do is install PhotonVision onto a Orange Pi 5, plug in a camera and change the camera type to object detection.

Running a detection model is the easy part, how you utilize it is the hard part. My team took the target from PV and put that into a homography, basically a homography takes camera points and maps them to real life distances. Using that we took our robots position and the note’s position and turned that into a field relative position. After that we plugged the notes position into our pathfinder and autonomously picked up the note.

Here’s our note detection code: https://github.com/FRCTeam3044/2024SwerveBase/blob/main/src/main/java/frc/robot/subsystems/NoteDetection.java

Feel free to ask any questions about our setup or code.

2 Likes

I’d like to point out that this fork is in violation of the Photonvision license (specifically the JNI modules) and the creator is banned from the PV Discord so I’d err on the side of caution if you decide to use it.

900 used a YoloV8 based model they fine tuned on the robot. I used it for various off robot tasks. We didn’t end up running note tracking during the season so it wasn’t a thing I can say we did.

We’ve been running YOLO models (including v8) for not long now in the offseason on our mini PC, which has a AMD Ryzen 6900HX. We get great performance.

As far as I know, the project does not violate its GPLv3 license. This claim was mentioned a few times very vaguely. If there’s a licensing issue, please file a detailed issue on the repo. Until such a claim is verified, please refrain from spreading misinformed rumors.

This has been talked about by the PV maintainers in their Discord. In my eyes it’s far from a rumor.

Please enable issues on the repository if you prefer this dispute to be held there. Otherwise, the relevant DMCA notice filed with GitHub explains the issue.

1 Like

After a brief read of the repo’s readme:

What does it mean if there’s a notice posted here? It only means that we received the notice on the indicated date. It does not mean that the content was unlawful or wrong. It does not mean that the user identified in the notice has done anything wrong. We don’t make or imply any judgment about the merit of the claims they make. We post these notices and requests only for informational purposes.

It seems that the project still being public is a sign that the violation is “fixed”?

If the repository owner makes the necessary changes, then GitHub will not disable the content. In some cases, that means the repositories are currently available despite being listed in the notice.

Now, looking at the commit history of that project, this commit from the day that noticed was filed added contact details for source code for a library. So from what I’ve pieced together I’m assuming GitHub notified them of the violation, they added the disclaimer, and GitHub has accepted it as a solution?

1 Like

I think the only definite response we’re going to get is from the PV maintainers themselves. According to a comment made by a PV maintainer in the Discord GitHub never responded to their DMCA request.

Didn’t try YOLOv8 but tried YOLOv5 with CUDA on Jetson Nano 4GB in 640 x 640 and it’s somehow choppy. Less than 30FPS I’d say.

1 Like