Creates a proxy for imshow and waitKey see you can use the same code locally and on a remote vision computer. On windows it just uses imshow, otherwise it will stream the video with h264.
On my test setup (running windows) this command will view the stream:
This is really cool, and exactly what we want to do, but we haven’t had any success with this. We’ve been attempting to decode with gstreamer on Linux and macOS with no luck. VLC doesn’t see anything in the stream.
Can you get the RTP stream to show on a LabView Dashboard?
There’s no reason you couldn’t create a similar thing and send the output to robotpy-cscore. cscore can stream MJPEG via HTTP, which is the format supported by the various dashboards and can be used via a web page.
Edit: Doh! Robotpy-cscore bindings require Python 3. We’re on the TK1 with the nVidia optimized OpenCV and Python bindings, which means we are stuck on Python 2.7.
The TX1/2 is on our map for next year, so I’m sure we will be looking into cscore at that time.
What we were planning on doing (haven’t tested this yet though) was sending the byte array object of the image Mat() (we’re in Java opencv on the rpi 3) over NetworkTables and then decoding it and displaying it with our own java program running on the driverstation laptop.
During preseason I was trying to get libx264/lavf to stream h264 video from opencv image processing on the Android vision phones we use. I was hoping you’d actually solved this problem, but it turns out it’s just forking an encoder process and dumping frames.
But mjpeg is enormously inefficient. We were hoping to stream high resolution 30fps video from the robot.
Peter was talking about h264 last night, I would expect to see cscore support for streaming that from webcams that natively produce it this summer. Or sooner if someone else does it and makes a pull request. Hint. Hint.
But I want it streaming with markup from image processing. That is, encode arbitrary Mats to an h264 stream. When I tried, I believe I screwed something up because gstreamer stuttered playing the resulting (RTP) stream and VLC refused to play it at all. (By stutter I mean pause for about a second, then very quickly play the missed frames really fast, and repeat.) I can post my code somewhere if someone feels like fixing it
There’s a few challenges with using H.264 for that purpose. While it can use significantly less network bandwidth than MJPEG, there are a number of disadvantages.
It requires significantly more processing power to encode (or hardware support, which many platforms we’re interested in do not have)
It’s patent encumbered
MJPEG uses frame-by-frame compression, H.264 is temporal across multiple frames, so…
Latency is likely higher
Often lower quality images
It’s less friendly to network or frame drops (e.g. if the decoder can’t keep up, it takes more time to catch up once it starts dropping frames)
Bandwidth usage can vary more dramatically
So there’s definitely tradeoffs to consider. As Dustin mentioned, I will be looking at adding support this summer (in particular for direct camera feeds) but patent concerns will likely make it not possible to distribute binaries that support H.264 encoding/decoding unless we can use hardware encoding or one of the Cisco OpenH264 binaries (https://github.com/cisco/openh264/releases). So support will be platform-specific (e.g. some versions of Android could be, the roboRio would not be).
We have an entire GPU on the Android phones we use currently unused because we didn’t bother to install the opencl drivers to have opencv use the GPU. So processing power isn’t an issue.
It’s not like we’re charging ourselves to use our own stream
In zerolatency mode there are no B frames
Quality is fully configurable. For the same apparent visual quality, resolution, and framerate h264 tends to use less bandwidth than mjpeg. That’s basically why we want to use it.
You can configure the rate of I frames however you want. If a packet is dropped, you simply wait for the next I frame. (I should clarify that we intend to use h264 over rtp). This is in contrast with mjpeg over tcp, where a dropped packet means a retry, which (if bandwidth is currently close to the limit) could potentially lead to more retries backing up and even more latency.
Not if you set it to a fixed bandwidth. Which is what you’d do
We found a way to modify the code in the original post of this thread to make a MJPEG stream, and then send that through a Python http server. So now we have a MJPEG stream over http.
Our current stumbling block is figuring out how to display this through the dashboard. We have a custom LabVIEW dashboard, but haven’t figured out how to replace the default USB cameras plugged into the RoboRIO, with our new stream.
So, basically you’re doing the same thing as cscore.
If you publish the right stuff over networktables, it’ll find it and connect to it. It’s not really documented anywhere, but you can look at the source code of CameraServer in wpilib or in robotpy-cscore to figure it out. Or just connect one using the normal stuff, open OutlineViewer, and then emit the same data.
We’re in the middle of the Sacramento regional right now, but we’ll definitely hit this track on Monday. We were pondering making a customized client for the dashboard, but we really don’t like the headache caused by the deletion of any of the stuff we don’t have plans to use (e.g. dashboard camera controls). It would be great if we could just fake it through network tables entries.
We thought we had it figured out, but are still stuck. We found two Network Tables variables that we thought would give us success. We changed our port from 5800 to 1181, to comply with the expectation of the dashboard. We can select the camera, but no video feed shows.
We tweaked our stream for minimum latency (320x240@15 fps), and could read it from every web browser at our disposal. I’m sure there is something simple we’re missing.
We didn’t get to push our code to the server before getting pushed out the door by the custodians. I’ll update when our current changes are moved into our server repository.
The second was set to a single element array with only the value of “mjpg:http://axis-camera.local/axis.cgi/mjpg/video.cgi”. We could not change this from the robot, but could overwrite this information from our dashboard. We set it similar to the first with “mjpg” replacing “ip”.
Perhaps the dashboard is sending a http request that is confounding my simple server. It works well enough for web browsers that are just pointed to it, but they aren’t trying to include functionality like changing resolution, exposure, or frame rate. Right now, the stream does nothing when selected in the default or our custom LabVIEW dashboard.
You’ll likely have better luck with SmartDashboard. The LabVIEW dashboard makes several simplifying assumptions that are probably breaking your custom implementation, depending on how much your simple server is checking or parsing the request. You may want to print out the request you’re receiving.
If source starts with “ip:”, the LabVIEW dashboard ignores the sources value and treats the source like an Axis camera… e.g. always sends “GET /axis-cgi/mjpg/video.cgi?fps=%d&compression=%d&resolution=%s” as the request.
If source starts with “usb:”, the LabVIEW dashboard pays attention to the sources value for the location and parameters, but always appends additional parameters of “name=%s&fps=%d&compression=%d&resoltuion=%s”.
