Paper: Real-Time Vision Processing with Convolutional Neural Networks using OpenCV’s AI Kit

I’ve been working on developing system that integrates computer-vision based machine learning using the OpenCV AI Kit for FRC applications. I’m excited to finally share the results of my research publicly.

The full paper describing my research can be found here.

The code can be found on my GitHub repo.

The dataset I used for training can be found on Supervisely.

Additional demonstration videos can be found on my YouTube playlist.

Huge thanks to @GabeHart17, @marshall, and the rest of team 900 for making their data public.
Also thanks to @AustinSchuh for sharing their implementation of SIFT.


Thanks for sharing. Fun to see those few teams who put their efforts to raise the bar of vision processing in FRC.

Very freaking cool! Love it.

This is a really great paper. Very thought provoking, and I must go back and read it again. It really does seem like machine learning is the new swerve drive, and it is thanks to efforts like this.

1 Like

Although there are some ML posts on CD lately, I would be careful before crowning vision ML as the new swerve.
Remember that swerve gained it’s popularity thanks to success on the field + being a COTS item.

At least we aren’t using reinforcement learning for control of robot mechanisms yet.


The next comment should be @marshall suggesting fully autonomous robot.
But there’s lack of data to model the environment (full scores breakdown from FMS please!; and some other stuff like proper robot model and game pieces/robots locations on the field).

Probably getting full scores breakdown is the easiest step (and actually would be useful for teams in many other areas). Do you know what prevents it from happening?

What I’m referring to is something like this: Reinforcement Learning and Control. You use repeated trials with carrot-and-stick methods and variations on gradient descent to converge on an optimal control policy. It was kind of tongue-in-cheek because for a lot of constrained optimization problems, you can get better results quicker for a lot less work with dynamic programming (if your system isn’t super high-dimensional, and you can model it well). That Quikplan trajectory optimization thing FRC 604 posted a while ago is an example.

The reasons I’ve heard are pretty common in longer-lived software companies in general: technical debt, and company politics/culture preventing cleanup of the former.

Yeah I got you, I was referring to fully autonomous robot. But I think we got a bit off the topic.

1 Like

I wouldn’t worry about that. One nice thing about ML models is they can be packaged and reused. It’s much less of an investment to use than swerve which requires large amounts of infrastructure and tuning.

That’s not to say it’s easy. But a plug and play system akin to the limelight is viable in this space.

OP - TSIMFD that is all.


Some people seem to be interpreting my post as a negative comment, so allow me to clarify. Far from it! The comparison to swerve was meant as a compliment. This is high-end technology that will trickle down to raise the floor for a lot of teams.