Tracking Notes with Vision (Demo, Model, Tools)

Technical Technical Discussion
1

With the release of the new game I’m sure a lot of folks are interested in tracking the game pieces. I’m sure some of y’all have made progress and that’s great. But for folks who are struggling or don’t know where to start, or just want something that works… let’s talk.

Above is running a fine tuned YoloV8 on one of the field tour videos (I needed a video that contained Notes and that was clearly ok for public display…) As you can see it’s not perfect, lots of false positives. This was trained on ~1 minute of footage of the quarter scale Notes I printed on Kickoff. (More on training later)

I wanted to share it with teams (here). This is the exact model and all annotated training data. Base work on it, use it as is, whatever. Just don’t be a jerk.

But there’s more, I wanted to improve the model as well as show more how models are trained without the fancy Roboflow training stuff (though if all you care about is getting something that works, Roboflow is great, let me get a proof of concept in under an hour, highly recommend) so I started working on doing it more manually (I say more because I’m still using a fair bit of Roboflow’s tooling, it makes life a lot easier) but let’s you run on your own infrastructure with a fair bit more flexibility.

The biggest hurdle to ML models is getting good training data - for real world tasks there exist data sets like COCO but obviously none of those will contain FRC specific game pieces. A very modern approach is to use a larger slower model like FB’s SegmentAnythingModel to assist in generating labeled data, you still have to do some clean up later but it’s a lot less work. So, I set out to do that… and now I’m handing it to you.

I will be using this to annotate training data and then manually refining it to train a new model. I’m still working on finding the right place to host the data set and weights (likely a zip file in google drive) but wanted to share it in case you want to do this on your own.

I know it’s likely a lot of students are interested in this domain and may not have mentors able to support their interests, I hope that this helps.

36 Likes
2

Looks awesome!
Can’t wait to see how accurate the AI models will be at the end of the season :hushed:

1 Like
3

@dh28567 @AidenK

4

I’m still very new to ML, but will this YoloV8 model run on a Google Coral, or will you have to take the training data and train a TensorFlowLite model for a Coral?

1 Like
5

Someone finally explained why everyone kept asking about the Coral… I don’t have one, but exporting to tflite should work for this as best I can find. I’ll try to put together a notebook over the weekend (probably not before Sunday though tbh) on downloading and converting it to TFLite for use on a Coral.

2 Likes
6

Ok, good to know. Alternatively, Limelight docs links to Teachable Machine for training new models, but I’m not sure what set of steps will give you the best model.

7

So, one of the reasons I’ve avoided teachable machine is that it does not support object detection only classification. It can tell you that you have an object in the image but not where in the image it is. Though if you were using camera to detect if you were holding a game piece inside your robot this would be useful but for locating and acquiring it’s less useful.

4 Likes
8

One last thing, the Coral has some requirements for any model that runs on it, found here: TensorFlow models on the Edge TPU | Coral. If you do try putting something together for exporting to a TFlite model, this might help. Alternatively, I can download as many images as I can/take more pictures and follow corals docs to compile a TFlite image myself and then share it.

2 Likes
9

Darn… this looks pretty good.

Thanks for providing this service!

1 Like
10

So I’ve went around and found any existing datasets for NOTE detection, and compiled them into a single dataset here. I’ll be updating it constantly as new images get added to other datasets, along with (hopefully) be providing a .TFLite model soon. I’m going to add more images myself next to robots with both red and blue bumpers, along with potentially robot bumper detection.

Credit to Andrew Schreiber, Botbusters FRC, teams 4143, 4169, 4201, and anyone who contributed images to any of the original datasets.

5 Likes
11

Looks like the “try this model” on that website doesn’t work with the pictures I’ve given it. It just says “something went wrong with the detection”. Not sure if it’s supposed to work, if my images are too high-res, etc. Regardless, great progress!

12

I got that this morning too, but I simply thought it was the bad Wi-Fi at my teams build space. When I trained the model last night on my machine at home, I tested it on the web page and it had the least number of false positives and false negatives than any of the other datasets I pulled from.

13

What image format did you give the webpage to test it?

14

I gave it both a .jpeg video I had downloaded locally, and a youtube link to the same video that was used in the original post.

15

Trying with the youtube video, looks pretty good, definitely struggles with notes that are seen edge-on. Hopefully the Coral with lower-resolution images has better latency than the website with the youtube video…

16

Another thing is, this is the heatmap of the notes in the images:
image

As you can see there’s a lot clustered to the center and not a lot near the edges/corners. I also need to get more images with high/low lighting, but I think I can do that within the RoboFLOW tools.

17

I’m working on setting up a public GitHub repository where people can add images to easily, as I don’t see a way to easily do that with RoboFLOW, and I want to make sure that people can get the images and dataset public to use however they want.

3 Likes
18

This is great, thank you! :clap:

If you’re just looking for an easy way to collect images (non-annotated), I’m happy to donate a SmugMug account that you can just enable public uploads into from anyone. Alternatively, Flickr is free and people could just upload images with a common tag (FRC2024Note or similar).

If you want annotated images, though, those platforms won’t help. I don’t know if you’re annotating yourself or collecting annotated data from other teams?

1 Like
19

Also looks like Limelight is collecting data as well. Might be a way to collaborate with them? Limelight Tools

20

As for collecting images, I do have a GitHub repository set up, and I can setup a script to update any new images in the repository into my RoboFLOW project, which is where I’ve been annotating images, both manually and with their auto-annotation tool. With RoboFLOW I can community source annotations/annotation review, but not have other users add images (as far as I can tell). I might look into both the programs you mentioned though.

As for collaberating with Limelight, I’ll send them an email asking if I can, but seeing as there’s no way to download the collected images from that link I’m not entirely sure if they’ll let me do that. I will add all the images I have downloaded at regular intervals however, so even if I can use their data, anyone using Limelights stuff can use the data I’ve collected.

3 Likes