Modifying the output of Photonvision pipeline

Technical Programming
1

We’d like to modify the image displayed by Photonvision after it goes through the postprocessing pipeline (so generally, this would be the image hosted at photonvision.local:1182) to draw some custom crosshairs, figures, etc. based on information on the NetworkTable.

Is there any straightforward (or non-straightforward) way to accomplish such a task? I don’t see any way to add another step to PhotonVision’s pipeline. Hypothetically, I imagine we could have the RoboRio grab the vision pipeline and post-process it, then serve those images as its own pipeline, but the only post-processing example I can find is pulling the raw video from a USB cam connected directly to the RoboRio (Read and Process Video: CameraServer Class — FIRST Robotics Competition documentation), not post-processing an Mjpeg stream from some other source.

Any tips or tricks would be greatly appreciated!

2

(Possibly answering my own question)

So I haven’t gotten a chance to play with it yet, but it appears CScore includes an HttpCamera class that can proxy the data from another Mjpeg camera server.

Purely hypothetically, this may get us two goals:

  1. Additional post-processing of a camera stream (by drawing crosshairs on it, etc.)
  2. Multiplexing between camera streams—we could, I imagine, have six cameras on the robot, and use software selection to select between camera streams and only vend one of them from the RoboRio for the dashboard to view. I’ll need to look at how network packets are actually sent from the robot to confirm that the mere existence of those six other camera streams won’t crush our bandwidth (in theory, packets should only get sent if a stream has an active viewer attached to it, right?).
3

Yes, you can do this with cscore on the Rio, but you will find it adds substantial stream latency, as the Rio isn’t very fast and it will need to decompress and recompress the stream in addition to whatever drawing operations you’re doing. I would instead recommend looking at doing this on the dashboard side with either a shuffleboard plugin or custom web based dashboard.

For camera switching, some dashboards support selecting between different streams on the dashboard side based on a NT key.

4

I would be tempted to do this on the display side if you can swing it.

5

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.