Hello everyone,
We have released OpenSight v0.2.0. A lot has happened since the last version we announced, v0.1.1.
New to OpenSight? Learn more here!
We’ve added many ways to filter contours, line and circle detection, beginnings of logic modules, and… wait, is this right?
H.264 Support
No, you’re not dreaming. OpenSight is proud to announce that, by our reckoning, we are officially the first FRC vision suite to integrate H.264 streaming.
In 2019, to accommodate the inevitable USB camera on every single robot, FIRST cut the bandwidth from 7Mbits/sec/team to 4Mbits/sec/team - nearly in half! And just as our team of elite fortune-tellers predicted, that cut remains in full force for the 2020 season at least. So we searched for a solution. And the manual delivered.
Some more delving into dark arts later and we had it! With The Quadrangles’ potential-engine and official examples as guides, we developed the perhaps single best H.264 RTSP server targeted at FRC today, and we named it…
rusty-engine
Yeah, not the best name, looking back.
“But,” I hear you ask, “what makes rusty-engine so cool?”
The first thing rusty-engine does is make your drivers very, very happy. H.264 can encode nearly twice the resolution as the roboRIO CameraServer in the same or even less bandwidth. It accomplishes this by encoding most of the frames in the video streams as (essentially) transformations of the previous frame, while only sending complete images when absolutely necessary or on a reasonably low interval. (In contrast, every single frame from a roboRIO CameraServer stream is a complete JPEG image.) Because of this, you can have a drastically better-quality stream for your drivers without breaking the FMS.
All you need to do enable H.264 on your OpenSight installation is change the “backend” setting of a CameraServer node from “MJPEG” to “H.264”:
Yes, really. That’s it.
But wait! There's more (esoteric technical features)!
Extensibility
rusty-engine is written in (mostly) well-done Rust. Adding new input modes such as zmq is as simple as adding a new member to an enum, and seeing where the compiler asks for you to fix things. The same is true for new encoders.
In general, maintaining rusty-engine is far easier than potential-engine or a modification of test-launch. And we would know, because rusty-engine came from the same guy as potential-engine.
Shared Memory
Perhaps rusty-engine’s true “killer feature”, rusty-engine can read frames from a shared memory socket and encode them in a nice H.264 stream just like they came from a normal webcam. So you can process an image, mark it up as much as you want, and send the annotated frame to your drivers and whatever target information you’ve extracted to e.g. NetworkTables. And all of this can be done in a normal OpenSight pipeline.
You don’t even need to run it on an OpenSight image, we have deb packages for amd64, aarch64, and armhf.
You want a library to write to shared memory too? OK, here you go.
“But,” you continue to ask, “how will I show the stream on my driver station? There’s nothing for H.264 in SmartDashboard or Shuffleboard!”
You’re in luck! Team 4639 has created a Shuffleboard plugin to view H.264 video streams in Shuffleboard. Just run the installer, open up Shuffleboard, and drag over your shiny new H.264 stream!
New Modules
OK, OK, enough about H.264 support. What about those new modules? We’ve got support for GPIO LEDs, so you can turn a light on and off as you target things. We added a slew of new processing modules, for finding circles, lines, and filtering contours. You can now fetch values from NetworkTables to use in your pipeline, or pair it with the GPIO module to do LED control through NT.
Example Pipelines
Find the center of the POWER PORT
Note: FindPowerPortCenter is a modification of the FindCenter stock function.
Contour Filter gallery
Display the color of a given pixel in terms of red, yellow, green or blue
Sample Output
Taken on an in-development version. The Y in the top-left is indicating that the detected color is indeed yellow.
Where do I get it?
Good question.
First Time Install
If you’re installing OpenSight on a new system, follow the installation instructions.
Upgrading to 0.2.0
If you currently have OpenSight, and the version installed (as indicated in the bottom right of the settings page) is older than version 0.1.2, you must either upgrade to 0.1.2 and then 0.2.0, or start over with a fresh installation of verion 0.2.0.
If you already have version 0.1.2 or later, you can simply follow the standard upgrade process.
Want more info, help, or just want to talk?
Check out our Discord server!
Full Changelog
Changelog
- GPIO LED support through
gpiozero: LEDs and PWM LEDs can now be turned on/off through the Raspberry Pi’s GPIO interface (added in v0.1.3) - Logic module: (added in v0.1.3)
- Freeze node: enable “freeze” to stop the nodes connected to the Freeze node’s output from running
- A framework has been built for conditional modules in the future
- Circle detection: Detect and find circles using the new AbsoluteDifference functions (HSV and RGB), Clamp Min/Max, FindCircles, and Draw Circles functions (docs WIP) (added in v0.1.3)
- Line detection: Detect lines using the Canny, FindLines, and DrawLines functions (docs WIP) (added in v0.1.3)
- GetNT: Get NetworkTables values using the GetNT function. Useful for setting LEDs and other parameters. (added in v0.1.3)
- (new in 0.2.0) Contour Filtering: You can filter contours by…
- Area
- Rectangular area (both bounding and minimum area)
- Speckle removal (filters out contours that are much smaller than the largest contour)
- Aspect ratio
- Orientation
- Angle
- Top-most, Bottom-most, Left-most, Right-most, Most central, largest, and smallest contours (e.g. only pass the 2 biggest contours)
- (new in 0.2.0) Color sampling and detection:
- ColorSampler: New function that provides provides the color of a given pixel in an image
- ColorDetector: New function that returns which of four colors a given color is closest to by hue. Useful for the CONTROL PANEL when coupled with the ColorSampler.
- (new in 0.2.0) You can now choose between DHCP and static IP in the settings tab.
- (new in 0.2.0) H.264 Video Streaming via
rusty-engine- Brand-new
rusty-engineRTSP server with support for Video4Linux2, Raspberry Pi Camera Module, and shared memory socket input. upgraded-engineerPython package to easily write frames from an OpenCV Python application torusty-engine.
- Brand-new
Credits
- @Jackflef: author of all the contour filters, circle and line detection, color sampling and detection, and probably just an ordinary toaster. Team 3641.
- Caleb X. Berger, @tkdberger: author of
rusty-engineandupgraded-engineer, H.264 madman and possibly a crab. Formerly team 3494.
- Quy Nguyen, @anon12642235: author of
gstreamer-shuffleboard, drinks green tea. Team 4639.
- Steven Spanger, @StevenSpangler: author of some (excruciating) H.264 misc bits. Lots of other things. Team 1189.
