High FPS object detection with PhotonVision fork - v1.0!

Technical PhotonVision
,
1

Get it here!

What is this, and is it relevant to me?

This is a fork of PhotonVision. It uses the NPU on the RK3588 processor to run object detection models.
This is relevant to you if you have a board with an RK3588 processor, such as the Orange Pi 5/5+.

Why use this over the stock implementation?

Pros:

  • Faster than the stock implementation. I’m getting 60 fps for 3 cameras running at the same time. Here are the results that team 1493 got: Google Docs
  • More mature - tested by over 8 teams
  • Allows you to use custom models very easily. Use one of the 10 models that we’ve trained, or train your own using Kaggle or Google Colab. See the RKNN readme for more details.
  • Allows you to use YOLOv8, which is faster and more accurate than YOLOv5
  • Allows you to use the detection in code. The stock implementation doesn’t give you the class and confidence of the target.
  • Includes the stock implementation in case you want to switch back. And of course all of the other pipelines such as reflective and AprilTags
  • Dedicated Discord server. link removed by moderator
Image from testing by team Galaxia 5987

Cons:

  • This is a fork, which is not supported by the PhotonVision team. It will get the latest features of PhotonVision in a delay, which I will try to minimize and keep under 24 hours, although I expect it to be within an hour.

Special thanks to the following teams, that helped with testing

  • 67
  • 1493
  • 5987
  • 7421
  • 8847
  • 9312
8 Likes
3

I’m kind of new to all of this PhotonVision stuff, but I was wondering if it is possible to use a Raspberry Pi instead of an Orange Pi for note detection, or if there is another way to detect notes efficiently rather than using the colored object detector (which is the only other way I could find).

4

Unfortunately, this is only available for devices with the rockchip (that has an RKNN accelerator).
You could always opt out for a custom machine learning solution, though it won’t really be as fast.
I know Limelight has a built-in option for object detection using the Coral, so stuff like that are also an option

5

Out of curiosity, have you talked with the photonvision developers? Is there a reason to fork as opposed to adding this work to photonvision?

3 Likes
6

The creator is banned from the PV discord iirc.

11 Likes
7

Thanks. Just spent a few minutes catching up on some of those channels. That explains it

Backing Up Homer Simpson GIF - Find & Share on GIPHY

8

How is this different from the RKNN functionality added in PhotonVision 2024.2.0?

You mentioned performance increases- is this because of a different model or because of optimizations in the implementation?

2 Likes
9

At what resolution? What camera are you using? What camera settings? What model?

Does PhotonVision not have its own dedicated discord server? Also, I clicked on the discord invite attached to this post, and it says “LaviRZ” invited me, who happens to have been banned from the FRC Discord server as well as the PhotonVision server.

Why not open a PR to merge these changes in to PhotonVision if they’re so much better than what PV currently has to offer?

3 Likes
10

Pretty sure suspended or banned from CD too… at least, I havent heard anything from said user lately (and their username doesnt come up for me while searching users here).

Perhaps this is an alt account of theirs?

Levyishai addresses these concerns a couple posts below.

1 Like
11

They are banned/suspended.

1 Like
12

It appears that Yishai lent their account to Lavi in order to make this post, based on the number of links with his name attached to them and not Yishai’s. The linked github repo was also created by Lavi.

4 Likes
13

The main differences are laid out in the original post, but I’ll try to sum it up here in a clearer way.
Functionality wise, the main difference currently is that we support uploading custom models (that can be trained using the notebooks we made), of both YOLOv5 and YOLOv8, while the stock implementation supports only YOLOv5. Models can be easily changed using a dropdown:

image.png780×644 32 KB

Also, the stock implementation does not give you access to the confidence and class values from the robot code, while ours does.

The answer is both.
On the model side, we include a pretty good YOLOv5n model, in addition to the stock model that is now included with PhotonVision, and a lot of other models:
The model that comes with the stock implementation is note-640-640-yolov5s. Our YOLOv5n model is smaller and thus can run faster.
On the implementation side, when running the same model, the performance of our implementation seems to land at around twice that of the stock implementation, without sacrificing accuracy of course. If the PhotonVision team is interested in merging this, they can reach out via the Discord server, we’d love to have it baked in instead of maintaining a fork.

1280x720, with our notes-yolov5n-640 model. That test was using black and white Arducam cameras, because that’s what we had on hand, but that shouldn’t make a difference, as the frame is converted to rgb anyway. Camera settings don’t affect performance.

It certainly does, but this is a project that’s not affiliated with the PhotonVision team, but is a fork of their work. If it makes you more comfortable, the invite link above is my own.

At this time, it does not look like the PhotonVision team is interested in merging this into their codebase. That’s the reason this fork exists. If they are interested, they’re welcome to reach out either here or through the Discord server.

I’m really sorry if this gave the wrong impression. I’m the new programming lead of 5990.
We asked the CD mods if we could post Lavi’s fork from my account since we wanted other people to have it available. And yes, this is Lavi’s fork with his work.
I really don’t want to get into any controversy, I only posted this to maks the fork accessible for all teams :slight_smile:

5 Likes
14

This is awesome! It will enable teams with lower resources that do not have a Limelight 3 or a Google Coral to achieve object detection. Have you compared the performance between the limelight and photonvision?

1 Like
15

Apologies for my last post. Seeing Lavi’s name attached to some of the stuff set off some red flags/alarms given their history on this forum and from what I’ve heard about on other platforms.

But with all that said/addressed, welcome to ChiefDelphi!

3 Likes
16

This isn’t exactly true, your speed up seems to be in part based on using the smaller model. This inherently means you’re going to be less accurate. Though this may not bear out in testing. It’s possible that model size is adequate for what you’re doing. But it’s not without loss in accuracy. It’s maybe without an observed loss in accuracy.

This isn’t really how open source works. If you are interested in merging it then you submit a PR/CL and discuss it with project maintainers. The other way lies madness.

Note here, the model operates on 640x640, you may actually get some speed up if you had a camera closer to that…

I mean, the reason for that seems to have been the author not aligning with the code of conduct of the project. (Which, note, if you maintain open source anything - have a code of conduct. Or if you have a group of any sort, have a code of conduct, it’s gonna save you a lot of hassle eventually)

I mean, so who is this posting? Lavi or? It’s really not clear and based on context. Obviously forum mods are aware of this but still feels weird.

While I’m pro making things available to teams, you stepped in the controversy already by posting the work of someone who is currently banned from multiple communities.

That all being said, I’m happy to see the colab notebooks for converting to RKNN, that was on my list of things to figure out.

8 Likes
17

The discord is also owned by lavi with 0 rules in said server. It is currently unmoderated and the owner will dm minors and adults alike.

Promoting the discord seems like a violation to CD rules regarding YPP safety

10 Likes
18

The way you said this, I think you must be sort of new to how this all works in terms of open-source collaboration. If there is a desire to have this baked in to PV mainline, then it should be submitted as a Pull Request to the PV mainline repository.

I’m not a PV developer, just a user, so I have no idea what the licensing on the code they’ve released but there’s a non-zero chance that you distributing this fork as it’s own thing may be a violation of the license statements. This also could be me on my boomer rocking chair, but if this is the case, that there is a license violation on the distribution of the PV dev team’s prior art, this is just stealing someone else’s code. Having the attitude of “If they want these improvements, they can just steal them back if they want so we don’t need a fork” isn’t how this type of collaboration works. If you’re building off of someone else’s work, please have the respect and decency to properly credit and respect the effort that created the thing you’re building from.

3 Likes
19
20

As far as I know, distributing this fork is not a violation of the license. What is a violation is keeping the JNI binaries he modified for his fork closed-source.

9 Likes
21

Standard I Am Not A Lawyer disclaimer, but this is the sort of thing I was talking about spiritually. As sort-of software developer myself, often building cool solutions on top of open-source stuff…It just bugs me to see how much theft there is out there without crediting the original work.