OpenSight Post-Kickoff Update - Now with H.264, Contour Filtering, and more!

Hello everyone,

We have released OpenSight v0.2.0. A lot has happened since the last version we announced, v0.1.1.
New to OpenSight? Learn more here!
We’ve added many ways to filter contours, line and circle detection, beginnings of logic modules, and… wait, is this right?

H.264 Support

No, you’re not dreaming. OpenSight is proud to announce that, by our reckoning, we are officially the first FRC vision suite to integrate H.264 streaming.

In 2019, to accommodate the inevitable USB camera on every single robot, FIRST cut the bandwidth from 7Mbits/sec/team to 4Mbits/sec/team - nearly in half! And just as our team of elite fortune-tellers predicted, that cut remains in full force for the 2020 season at least. So we searched for a solution. And the manual delivered.
Some more delving into dark arts later and we had it! With The Quadrangles’ potential-engine and official examples as guides, we developed the perhaps single best H.264 RTSP server targeted at FRC today, and we named it…

rusty-engine

Yeah, not the best name, looking back.
“But,” I hear you ask, “what makes rusty-engine so cool?”

The first thing rusty-engine does is make your drivers very, very happy. H.264 can encode nearly twice the resolution as the roboRIO CameraServer in the same or even less bandwidth. It accomplishes this by encoding most of the frames in the video streams as (essentially) transformations of the previous frame, while only sending complete images when absolutely necessary or on a reasonably low interval. (In contrast, every single frame from a roboRIO CameraServer stream is a complete JPEG image.) Because of this, you can have a drastically better-quality stream for your drivers without breaking the FMS.

All you need to do enable H.264 on your OpenSight installation is change the “backend” setting of a CameraServer node from “MJPEG” to “H.264”:

Yes, really. That’s it.

But wait! There's more (esoteric technical features)!

Extensibility

rusty-engine is written in (mostly) well-done Rust. Adding new input modes such as zmq is as simple as adding a new member to an enum, and seeing where the compiler asks for you to fix things. The same is true for new encoders.
In general, maintaining rusty-engine is far easier than potential-engine or a modification of test-launch. And we would know, because rusty-engine came from the same guy as potential-engine.

Shared Memory

Perhaps rusty-engine's true “killer feature”, rusty-engine can read frames from a shared memory socket and encode them in a nice H.264 stream just like they came from a normal webcam. So you can process an image, mark it up as much as you want, and send the annotated frame to your drivers and whatever target information you’ve extracted to e.g. NetworkTables. And all of this can be done in a normal OpenSight pipeline.

You don’t even need to run it on an OpenSight image, we have deb packages for amd64, aarch64, and armhf.
You want a library to write to shared memory too? OK, here you go.

“But,” you continue to ask, “how will I show the stream on my driver station? There’s nothing for H.264 in SmartDashboard or Shuffleboard!”
You’re in luck! Team 4639 has created a Shuffleboard plugin to view H.264 video streams in Shuffleboard. Just run the installer, open up Shuffleboard, and drag over your shiny new H.264 stream!

New Modules

OK, OK, enough about H.264 support. What about those new modules? We’ve got support for GPIO LEDs, so you can turn a light on and off as you target things. We added a slew of new processing modules, for finding circles, lines, and filtering contours. You can now fetch values from NetworkTables to use in your pipeline, or pair it with the GPIO module to do LED control through NT.

Example Pipelines

Find the center of the POWER PORT


Note: FindPowerPortCenter is a modification of the FindCenter stock function.

Contour Filter gallery

Display the color of a given pixel in terms of red, yellow, green or blue

Sample Output


Taken on an in-development version. The Y in the top-left is indicating that the detected color is indeed yellow.

Where do I get it?

Good question.

First Time Install

If you’re installing OpenSight on a new system, follow the installation instructions.

Upgrading to 0.2.0

If you currently have OpenSight, and the version installed (as indicated in the bottom right of the settings page) is older than version 0.1.2, you must either upgrade to 0.1.2 and then 0.2.0, or start over with a fresh installation of verion 0.2.0.
If you already have version 0.1.2 or later, you can simply follow the standard upgrade process.

Want more info, help, or just want to talk?

Check out our Discord server!

Full Changelog

Changelog
  • GPIO LED support through gpiozero: LEDs and PWM LEDs can now be turned on/off through the Raspberry Pi’s GPIO interface (added in v0.1.3)
  • Logic module: (added in v0.1.3)
    • Freeze node: enable “freeze” to stop the nodes connected to the Freeze node’s output from running
    • A framework has been built for conditional modules in the future
  • Circle detection: Detect and find circles using the new AbsoluteDifference functions (HSV and RGB), Clamp Min/Max, FindCircles, and Draw Circles functions (docs WIP) (added in v0.1.3)
  • Line detection: Detect lines using the Canny, FindLines, and DrawLines functions (docs WIP) (added in v0.1.3)
  • GetNT: Get NetworkTables values using the GetNT function. Useful for setting LEDs and other parameters. (added in v0.1.3)
  • (new in 0.2.0) Contour Filtering: You can filter contours by…
    • Area
    • Rectangular area (both bounding and minimum area)
    • Speckle removal (filters out contours that are much smaller than the largest contour)
    • Aspect ratio
    • Orientation
    • Angle
    • Top-most, Bottom-most, Left-most, Right-most, Most central, largest, and smallest contours (e.g. only pass the 2 biggest contours)
  • (new in 0.2.0) Color sampling and detection:
    • ColorSampler: New function that provides provides the color of a given pixel in an image
    • ColorDetector: New function that returns which of four colors a given color is closest to by hue. Useful for the CONTROL PANEL when coupled with the ColorSampler.
  • (new in 0.2.0) You can now choose between DHCP and static IP in the settings tab.
  • (new in 0.2.0) H.264 Video Streaming via rusty-engine
    • Brand-new rusty-engine RTSP server with support for Video4Linux2, Raspberry Pi Camera Module, and shared memory socket input.
    • upgraded-engineer Python package to easily write frames from an OpenCV Python application to rusty-engine.

Credits

  • @Jackflef: author of all the contour filters, circle and line detection, color sampling and detection, and probably just an ordinary toaster. Team 3641. :bread:
  • Caleb X. Berger, @tkdberger: author of rusty-engine and upgraded-engineer, H.264 madman and possibly a crab. Formerly team 3494. :crab:
  • Quy Nguyen, @carbotaniuman: author of gstreamer-shuffleboard, drinks green tea. Team 4639. :tea:
  • Steven Spanger, @StevenSpangler: author of some (excruciating) H.264 misc bits. Lots of other things. Team 1189. :snowman:
14 Likes

Does this support NeoPixel LED arrays?

Also, this update looks amazing! Kinda regret having bought a Limelight now, seeing as we could get all of these features for the price of an RPi + camera module

2 Likes

We don’t support NeoPixels currently however it would be a trivial addition if people are interested in it. If you want to add it yourself you could probably make a module for it in a few minutes, see our module dev docs.

1 Like

Have you benchmarked the latency with H.264?

Not formally, and it will depend on your pipeline/image quality/framerate/etc. However, rusty-engine alone should sit nicely at or below 100ms for a 240p stream.

This will also depend on your coprocessor. Hardware acceleration is done through the OpenMAX framework on all platforms when available, so a Jetson will naturally outperform a Raspberry Pi 3, for example.

Personally speaking, I would encourage you to have your team’s drivers (assuming they’re already chosen) try driving off of it to get their opinions on it before making any final decisions.

1 Like

See @tkdberger’s response, and for what it’s worth I tested it with one of our robots and at 480p30, according to our drivers there was no discernible latency.

2 Likes

[Using 0.2.1, or at least I think I am with a fresh 0.2.1 image plus Updating to 0.2.1, although Settings still shows 0.2.0-rc2]

I recreate this pipeline, get a green check, and when I go to view the CameraServer output, and It is a forever loading white page. If I back up to the node building page, the check is still green. Am I missing something?

Try pressing CTRL+Shift+R on the settings page. Sounds like you have a rogue cache.

Try adding ?compression=90 to the end of the URL, or try using Shuffleboard to use the stream.

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.