Limelight Offseason + Control System Updates

FIRST General Forum
1

Limelight Offseason Updates

The New Control System

Communications about the new control system will primarily come from FIRST. What we can say is that the first iteration of the ecosystem was working early January, so it has already seen a huge amount of iteration.

From the perspectives of everyone involved, the only ‘feature’ that actually matters is reliability. If you have it, little else matters. If you don’t have it, nothing else matters.
This is a violently binary characteristic of any control system, and it has driven a focus on hardware simulation, design reviews including the 60+ incredible engineers at Raspberry Pi, rapid iteration, and destructive testing.

The overwhelming majority of our our development time over the past 12+ months has been spent on electrical protection implementation and testing.

We have a comprehensive specifications document that serves as our “North Star”, but we have also been writing “Scenario Explorations” that translate technically complex protection implementations into easy-to-understand “what-if” scenarios.

You will see a few more of these over time, but here’s an early set of three scenario explorations that have already been validated on real, working hardware:

All exposed interfaces, even USB 3.0 and Gigabit Ethernet, are designed beyond the RFP’s specifications with ±30KV air and contact ESD protection. This is critical for reliability, especially on FTC fields.

Hardware

Limelight 3A

Limelight 3A is a $189 USB smart camera for FTC and FRC. With CPU neural network inference and MegaTag2 AprilTag localization, FTC teams and FRC teams now have access to a low-cost, fully-integrated vision solution. Limelight 3A features ±30KV ESD protection on exposed interfaces to survive FTC and FRC environments.

The FTC community’s rapid adoption of Python SnapScripts, neural networks, and MegaTag2 has blown us away. The next batch of 3A units is coming soon.

Limelight 3 and 3G Refresh

Limelight 3 and 3G have been redesigned with a brand new power supply, and they’re already in the wild. With 4.5V - 30V input and external short protection, Limelights are more robust than ever. The new units have red configuration buttons.
If you’ve purchased a camera during the offseason, you likely have refreshed hardware. Additional LL3s will come online in early December.

Team

Our growing team now includes a senior aerospace EE, and a 17-year automotive power EE. We have a growing, AAA team working on new products for FIRST teams.

Software Updates (Misc)

Ask-A-Question

Our LLM-based ask-a-question feature is simultaneously reducing our support load and providing better answers to teams. In several instances, it has provided such amazing answers that we have changed our documentation to match its output. Our cloud infrastructure scrapes our documentation and depersonalized support interaction database periodically to stay up-to-date with the latest information.

Text-A-Question (Beta)

You can now text +1 (833) 705 1817 to receive instant Limelight assistance.

Internationalization

The limelight docs now include Spanish, Portuguese, French, Hebrew, Arabic, Hindi, Turkish, Japanese, and Simplified Chinese translations.

Note - some changes to the docs were made today, so you will see a few pages with incomplete translations. Our translated changes tend to take < 6 hours to fully propagate.

AprilTag Field Maps

The map generator and on-camera visualizer now support FTC field maps. The online generator will accept and auto-convert WPILib apriltag configurations including those generated by cowlibration-field.

New Website

While primarly an visual update for now, the new website gives us a few new capabilities that will be useful in the future.

Software (Limelight OS)

After the 2024 world championship, we launched Limelight OS versions 2024.6 (May 8th) through 2024.10.2 (October 27th). We recommend reading the linked change log, but the updates are summarized below:

Breaking Changes:

  • Remove “Camera Orientation” UI option, and replace it with “Stream Orientation”
    • This only affects the mjpeg stream. Teams will need to manually handle tx and ty changes while using rotated cameras.
    • We felt that “camera orientation” introduced too much confusion when combined with camera calibration and camera roll angle configuration.
  • Remove GRIP support
  • Remove “Driver” mode and add a “Viewfinder” pipeline type
    • The “Viewfinder” pipeline type does not perform any image processing. It allows teams to configure their ideal view-only mode.
  • .vpr pipeline files are now JSON based. Old pipelines will be converted automatically when they are uploaded to a camera, and you can pre-convert using the pipeline upgrade tool here: Limelight Tools
  • Results, status, and calibration JSON formats have been simplified.

Bugfixes

  • If a Google Coral is unplugged, neural pipelines no longer revert back to color pipelines. Instead, they display “check coral” on the image stream.
  • If a detector model is uploaded to a classifier pipeline, “check model” is displayed.
  • If a classifier model is uploaded to a detector pipeline, “check model” is displayed
  • Thanks to the FTC community, the Python SnapScript and Neural Network pipelines have been further ruggedized.

New Features:

  • USB Capabilities
    • Multiple USB Limelight support on RoboRIO and PCs with USB INDEX (settings tab).
      • The subnet changes to 172.29.(USBID).0/24 for Linux/Android/Mac and 172.28.(USBID).0/24 for Windows.
    • Improved connectivity on Windows/MacOS
    • USB implementation is now fully compatible with school-locked chromebooks.
    • USB Limelights are plug-and-play on the RoboRIO. They auto-populate networktables just like ethernet-connected cameras.
  • UI Warnings
    • Add warnings for common mistakes such as a static IP address that interferes with the FMS, incorrect AprilTag sizing, etc.
  • Python SnapScript pipelines now include the UMichigan AprilTag3 Library.

  • New Pipeline Types
    • “Focus” pipeline with stream quality and crop box sliders
    • “Barcodes” pipeline
      • Multi QR Code Detection (50-60 FPS)
      • Multi DataMatrix Detection (50-60 FPS)
      • (Beta) UPC, EAN, Code128, and PDF417 support (30 FPS)
  • CPU Neural Network Inference (Change runtime to “CPU” in neural pipeline)
    • Support for CPU classifiers (15-18 FPS)
    • Support for CPU detectors (8-10 FPS)
    • The online training pipeline generates CPU and Coral neural networks
  • Complete REST API rebuild
  • New Python library (limelightlib-python)
  • New Rust library (limelightlib-rust)
  • Added Modbus support for non-FIRST applications
  • Custom NT server support for usage with Glass and other NT4 clients outside of FRC.
  • Dynamic AprilTag 3D Offset support (NT fiducial_offset_set)
    • It is now possible to adjust the 3D Offset without changing pipelines. This is useful for situations in which your “aim point” needs to change based on distance or other properties.
  • Display warning if a static IP address falls outside of the .11-.19 range. Addresses outside of this range may interfere with the fms or other common devices.

AprilTag Pipeline Changes:

  • Display warning if configured AprilTag size does not match with field map tag sizes. This was the root of most of the accuracy issues we have debugged with teams.
  • MegaTag2 Improvements
    • Upgrade to NetworkTables 4.0 and add robot-side flush() for significant accuracy improvements while turning.
    • Add “Gyro latency adjustment” slider
    • Post STDDevs to networktables, the UI, and JSON. Don’t plug these numbers directly into your robot code - use them to characterize the min/max STDDevs while still to help you select STDDev values for your localizers.
    • MT2 now returns (0,0,0) when untenable geometric impossibilities are detected (previously, the camera was placed at (0,0,0) which would place the robot away from (0,0,0)
  • Several visualizer upgrades
    • Hide/Show individual target data, MT2 data, MT1 data, stddevs
    • Maps show red/blue alliance stations

Other Pipeline Changes:

  • Add “centroid” targeting region for more stable color object tracking. Centroid will target the center of mass of a contour rather than the center of a contour.

Calibration Changes:

  • Calibration settings are now cached. You no longer need to enter grid and mark sizes every time you want to calibrate.
  • Add “Download Calibration Mosaic” button.
    • Calibration mosaics show calibration coverage and per-image corner detection results.
  • Calibration accuracy has been improved. You can expect ~1 pixel or less reprojection error with as few as 15 images of paper targets.
  • Calibration is over 20x as fast as before - you can expect a calibration result within seconds even with a large number of images.

NetworkTables Changes

  • Add getpipetype
  • tcclass, tdclass strings for neural pipeline class output
  • Add t2d atomic 2d targeting array
    • [valid, targetcount, targetlatency, capturelatency, tx, ty, txnc, tync, ta, targetid, classifierID, detectorID, tlong, tshort, thor, tvert, ts(skew)]
  • Add crosshairs array [cx0, cy0, cx1, cy1] and remove individual cx0, cy0, cx1, cy1 entries
  • Add fiducal_offset_set
  • Remove tlong, tshort, thor, tvert, ts entries
  • Add rawbarcodes string array
  • Raw target arrays include up to 32 targets

JSON Changes

  • Add pType, stdev_mt1, stdev_mt2, PythonOut, tx, ty, txnc, tync, ta
  • Area value in target arrays returns calibrated/undistorted, normalized value

Thanks!

83 Likes
2

Love this standardized electrical testing.

Its non-trivial and some companies “secret sauce” - refined product validation and verification to ensure their components are “consumer-proof” and electrically ruggedized! What happens when all pins are Shorted to HIGH, GROUND, or EACHOTHER???

I love to see those scenarios being survivable (and hopefully self-recoverable, user recoverable at the most basic level)

7 Likes
3

This is nice to hear. Is the final design locked in at this point, or are you still considering iterations?

Also curious if you can share whether or not iterations of the control system components will be considered during the contracted use period, or if the system designs as-is will be locked in for the duration of the contract. I remember when the 4-slot CRIO was added as an option for teams - I want to say that was still in the middle of the CRIO’s contract period. Personally, I would not be opposed to further improvement and iteration on a control system, especially with modular components that can be swapped in and out.

3 Likes
4

I already think it’s a no but this wouldn’t include POE would it?

5

Do you have any information about the LL3’s canbus support? Be that software or hardware

6

@Jwal Recoverability is something that we control. For example, the current (subject to change) implementation of the RSL is that it immediately recovers once a short is removed. IO Ports (currently) recover when we tell them to recover via software.

@Andrew_L We would certainly want to minimize board-level iteration to ensure stability with respect to the user experience, global product certification, and documentation, but it’s always on the table. Everything has been designed to be as future-proof as it can be, with modularity only where it really makes sense.

@jedgar6 Passive POE also supports a maximum of 30V.

@cmdada It’s there and has been used for other projects, but we decided to drop the feature for FRC in the name of simplicity

5 Likes
7

Love to hear it!

3 Likes
8

Is there potential for an upgrade path to buy the board with the power supply and upgrade existing units. Also along the lines of upgrade paths if the limelight’s get upgraded to the cm5 it would be cool if we could upgrade it ourselves.

9

Does this mean there wont be a new “Limelight 4” before Reefscape?

2 Likes
10

What is the difference between the 3G and 3A in terms of performance capability? Why would I buy the more expensive 3G instead of a 3A? When is the 3A slated to release to FRC teams?

Sorry if I missed the answer to any of these questions, but I did skim through the thread.

1 Like
11

What NT key do we modify to set the camera offset pose?

12

I think the big difference is that the 3G is a “Global Shutter” camera, while the 3A is a rolling shutter. At FTC robot speeds, this probably doesn’t matter, but at FRC robot speeds, if you want to have any hope of seeing an April tag while moving, you need the global shutter.

3G is also a greyscale camera, so it can pretty much only be used to april tags. No gamepiece detection with google coral supported (I don’t know if the 3A can use the coral either).

3 Likes
13

A couple questions you may or may not be able to speak on:

  1. What’s the software setup look like? Will we have to use a specific image or will we be able to install other distributions + some library to handle the I/O stuff? I’m assuming the default image will probably be raspbian based but if we were able to install other distros like ubuntu that could make installing something like ROS 2 easier since that has tier 1 support for ubuntu vs tier 3 for raspbian.

  2. Is limelight handling the “driving” side of things like a new driver station and/or FMS? If so, will this be open source?

  3. Will it support 802.3af/at/bt POE?

Fully expect that you can’t talk about it much yet, but these were the questions the blog post left unanswered for me, so figured I’d give it a shot.

11 Likes
14

The OS is not Debian based. It is our own minimal Linux image with realtime patches that boots and advertises an SSID extremely quickly. It builds on years of tuning LimelightOS, including recent Chromebook USB-Ethernet support tweaks we made for Limelight 3A

Determinism is important, and you lose that when you use something like Debian. While it is technically possible with a lot of version pinning, it’s not what Debian is built for.

Other distros may eventually be supported for a wide variety of reasons, but I can’t comment on legality.

POE will not be supported.

Driverstation stuff isn’t finalized.

@nstrike camerapose_robotspace_set

16 Likes
15

We will do our best to support as many low-cost upgrade paths as possible. I’ve personally run a cm5 on a 3g board. You might need a thinner thermal pad for that particular upgrade.

I’ll get back to you about your board upgrade request.

7 Likes
16

Im not an FTC guy, far from it. But I appreciate informing the FRC audience on (part of) the why the static aspect was pursued aggressively. Obviously FRC can have static issues (2009, 2018)

2 Likes
17

We’d also be extremely interested in a power supply upgrade path - I’ve run all our limelights on individual VRMs historically to keep them off the VBatt rail and having the increased reliability in the box would be a game changer… Especially since I ran out of vrm back stock in my lab this year and wasn’t looking forward to specifying and testing a new external regulator.

1 Like
18

Should we expect the camerapose_robotspace_set to be reflected on the LL configuration page? I see it set in NT as expected, but not on the Advanced tab, even with a page refresh.

1 Like
19

Just a side question. Is it being translated by ai or something like google translate?

20

From a brief read as a Hebrew speaker it seems like a fair AI translation.

2 Likes