Go to Post Words mean what words mean. If your robot is sitting on the ball, then by definition it is not hanging from the bar. NO 50 point bonus. - dlavery [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 02-22-2017, 01:34 PM
sparkytwd's Avatar
sparkytwd sparkytwd is offline
Registered User
FRC #3574
Team Role: Mentor
 
Join Date: Feb 2012
Rookie Year: 2012
Location: Seattle
Posts: 110
sparkytwd will become famous soon enoughsparkytwd will become famous soon enough
Stream opencv output remotely (like imshow, but for headless video processors)

What it looks like: http://imgur.com/a/KhwEd
The script: https://gist.github.com/anonymous/c0...6601dbc8d71c81
Example: https://gist.github.com/anonymous/67...ab185978faad1a

Creates a proxy for imshow and waitKey see you can use the same code locally and on a remote vision computer. On windows it just uses imshow, otherwise it will stream the video with h264.

On my test setup (running windows) this command will view the stream:

c:\gstreamer\1.0\x86_64\bin\gst-launch-1.0.exe udpsrc port=5800 caps="application/x-rtp" ! rtph264depay ! avdec_h264 ! autovideosink
Reply With Quote
  #2   Spotlight this post!  
Unread 03-18-2017, 05:09 PM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 201
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

This is really cool, and exactly what we want to do, but we haven't had any success with this. We've been attempting to decode with gstreamer on Linux and macOS with no luck. VLC doesn't see anything in the stream.

Can you get the RTP stream to show on a LabView Dashboard?
Reply With Quote
  #3   Spotlight this post!  
Unread 03-18-2017, 11:06 PM
virtuald's Avatar
virtuald virtuald is offline
RobotPy Guy
AKA: Dustin Spicuzza
FRC #1418 (), FRC #1973, FRC #4796, FRC #6367 ()
Team Role: Mentor
 
Join Date: Dec 2008
Rookie Year: 2003
Location: Boston, MA
Posts: 1,223
virtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant future
Re: Stream opencv output remotely (like imshow, but for headless video processors)

There's no reason you couldn't create a similar thing and send the output to robotpy-cscore. cscore can stream MJPEG via HTTP, which is the format supported by the various dashboards and can be used via a web page.
__________________
Maintainer of RobotPy - Python for FRC
Creator of pyfrc (Robot Simulator + utilities for Python), pynetworktables/pynetworktables2js (NetworkTables for Python & Javascript), and lots more...

2017 Season: Teams #1973, #4796, #6367
Team #1418 (remote mentor): Newton Quarterfinalists, 2016 Chesapeake District Champion, 2x Innovation in Control award, 2x district event winner
Team #1418: 2015 DC Regional Innovation In Control Award, #2 seed; 2014 VA Industrial Design Award; 2014 Finalists in DC & VA
Team #2423: 2012 & 2013 Boston Regional Innovation in Control Award


Resources: FIRSTWiki (relaunched!) | My Software Stuff
Reply With Quote
  #4   Spotlight this post!  
Unread 03-19-2017, 03:22 PM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 201
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Definitely checking that today.

Edit: Doh! Robotpy-cscore bindings require Python 3. We're on the TK1 with the nVidia optimized OpenCV and Python bindings, which means we are stuck on Python 2.7.

The TX1/2 is on our map for next year, so I'm sure we will be looking into cscore at that time.

Last edited by Levansic : 03-19-2017 at 03:27 PM.
Reply With Quote
  #5   Spotlight this post!  
Unread 03-19-2017, 07:44 PM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 201
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Got the receive to work on Linux. To receive and show the stream on Linux, replace the "autovideosink" with "xvimagesink".
Reply With Quote
  #6   Spotlight this post!  
Unread 03-19-2017, 07:52 PM
EmileH's Avatar
EmileH EmileH is offline
Electron Unclogger
AKA: Emile Hamwey
FRC #1058 (PVC Pirates)
Team Role: Student
 
Join Date: Dec 2014
Rookie Year: 2011
Location: New Hampshire
Posts: 680
EmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond reputeEmileH has a reputation beyond repute
Re: Stream opencv output remotely (like imshow, but for headless video processors)

What we were planning on doing (haven't tested this yet though) was sending the byte array object of the image Mat() (we're in Java opencv on the rpi 3) over NetworkTables and then decoding it and displaying it with our own java program running on the driverstation laptop.
Reply With Quote
  #7   Spotlight this post!  
Unread 03-20-2017, 07:49 AM
euhlmann's Avatar
euhlmann euhlmann is offline
CTO, Programmer
AKA: Erik Uhlmann
FRC #2877 (LigerBots)
Team Role: Leadership
 
Join Date: Dec 2015
Rookie Year: 2015
Location: United States
Posts: 453
euhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

During preseason I was trying to get libx264/lavf to stream h264 video from opencv image processing on the Android vision phones we use. I was hoping you'd actually solved this problem, but it turns out it's just forking an encoder process and dumping frames.

Quote:
Originally Posted by virtuald View Post
There's no reason you couldn't create a similar thing and send the output to robotpy-cscore. cscore can stream MJPEG via HTTP, which is the format supported by the various dashboards and can be used via a web page.
But mjpeg is enormously inefficient. We were hoping to stream high resolution 30fps video from the robot.
__________________
Creator of SmartDashboard.js, an extensible nodejs/webkit replacement for SmartDashboard


https://ligerbots.org
Reply With Quote
  #8   Spotlight this post!  
Unread 03-20-2017, 10:24 AM
virtuald's Avatar
virtuald virtuald is offline
RobotPy Guy
AKA: Dustin Spicuzza
FRC #1418 (), FRC #1973, FRC #4796, FRC #6367 ()
Team Role: Mentor
 
Join Date: Dec 2008
Rookie Year: 2003
Location: Boston, MA
Posts: 1,223
virtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant future
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by euhlmann View Post
But mjpeg is enormously inefficient. We were hoping to stream high resolution 30fps video from the robot.
Peter was talking about h264 last night, I would expect to see cscore support for streaming that from webcams that natively produce it this summer. Or sooner if someone else does it and makes a pull request. Hint. Hint.
__________________
Maintainer of RobotPy - Python for FRC
Creator of pyfrc (Robot Simulator + utilities for Python), pynetworktables/pynetworktables2js (NetworkTables for Python & Javascript), and lots more...

2017 Season: Teams #1973, #4796, #6367
Team #1418 (remote mentor): Newton Quarterfinalists, 2016 Chesapeake District Champion, 2x Innovation in Control award, 2x district event winner
Team #1418: 2015 DC Regional Innovation In Control Award, #2 seed; 2014 VA Industrial Design Award; 2014 Finalists in DC & VA
Team #2423: 2012 & 2013 Boston Regional Innovation in Control Award


Resources: FIRSTWiki (relaunched!) | My Software Stuff
Reply With Quote
  #9   Spotlight this post!  
Unread 03-20-2017, 11:46 AM
euhlmann's Avatar
euhlmann euhlmann is offline
CTO, Programmer
AKA: Erik Uhlmann
FRC #2877 (LigerBots)
Team Role: Leadership
 
Join Date: Dec 2015
Rookie Year: 2015
Location: United States
Posts: 453
euhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by virtuald View Post
Peter was talking about h264 last night, I would expect to see cscore support for streaming that from webcams that natively produce it this summer. Or sooner if someone else does it and makes a pull request. Hint. Hint.
But I want it streaming with markup from image processing. That is, encode arbitrary Mats to an h264 stream. When I tried, I believe I screwed something up because gstreamer stuttered playing the resulting (RTP) stream and VLC refused to play it at all. (By stutter I mean pause for about a second, then very quickly play the missed frames really fast, and repeat.) I can post my code somewhere if someone feels like fixing it
__________________
Creator of SmartDashboard.js, an extensible nodejs/webkit replacement for SmartDashboard


https://ligerbots.org
Reply With Quote
  #10   Spotlight this post!  
Unread 03-20-2017, 01:51 PM
Peter Johnson Peter Johnson is offline
WPILib Developer
FRC #0294 (Beach Cities Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Redondo Beach, CA
Posts: 317
Peter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant future
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by euhlmann View Post
But I want it streaming with markup from image processing. That is, encode arbitrary Mats to an h264 stream. When I tried, I believe I screwed something up because gstreamer stuttered playing the resulting (RTP) stream and VLC refused to play it at all. (By stutter I mean pause for about a second, then very quickly play the missed frames really fast, and repeat.) I can post my code somewhere if someone feels like fixing it
There's a few challenges with using H.264 for that purpose. While it can use significantly less network bandwidth than MJPEG, there are a number of disadvantages.
  • It requires significantly more processing power to encode (or hardware support, which many platforms we're interested in do not have)
  • It's patent encumbered
  • MJPEG uses frame-by-frame compression, H.264 is temporal across multiple frames, so...
  • Latency is likely higher
  • Often lower quality images
  • It's less friendly to network or frame drops (e.g. if the decoder can't keep up, it takes more time to catch up once it starts dropping frames)
  • Bandwidth usage can vary more dramatically

So there's definitely tradeoffs to consider. As Dustin mentioned, I will be looking at adding support this summer (in particular for direct camera feeds) but patent concerns will likely make it not possible to distribute binaries that support H.264 encoding/decoding unless we can use hardware encoding or one of the Cisco OpenH264 binaries (https://github.com/cisco/openh264/releases). So support will be platform-specific (e.g. some versions of Android could be, the roboRio would not be).
__________________
Author of cscore - WPILib CameraServer for 2017+
Author of ntcore - WPILib NetworkTables for 2016+
Creator of RobotPy - Python for FRC

2010 FRC World Champions (294, 67, 177)
2007 FTC World Champions (30, 74, 23)
2001 FRC National Champions (71, 294, 125, 365, 279)
Reply With Quote
  #11   Spotlight this post!  
Unread 03-20-2017, 06:17 PM
Peter Johnson Peter Johnson is offline
WPILib Developer
FRC #0294 (Beach Cities Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Redondo Beach, CA
Posts: 317
Peter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant futurePeter Johnson has a brilliant future
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by EmileH View Post
What we were planning on doing (haven't tested this yet though) was sending the byte array object of the image Mat() (we're in Java opencv on the rpi 3) over NetworkTables and then decoding it and displaying it with our own java program running on the driverstation laptop.
I would recommend you not do this, although there's at least one team who has. See my post in this thread for some reasons why. https://www.chiefdelphi.com/forums/s...d.php?t=155443
__________________
Author of cscore - WPILib CameraServer for 2017+
Author of ntcore - WPILib NetworkTables for 2016+
Creator of RobotPy - Python for FRC

2010 FRC World Champions (294, 67, 177)
2007 FTC World Champions (30, 74, 23)
2001 FRC National Champions (71, 294, 125, 365, 279)
Reply With Quote
  #12   Spotlight this post!  
Unread 03-21-2017, 01:01 PM
euhlmann's Avatar
euhlmann euhlmann is offline
CTO, Programmer
AKA: Erik Uhlmann
FRC #2877 (LigerBots)
Team Role: Leadership
 
Join Date: Dec 2015
Rookie Year: 2015
Location: United States
Posts: 453
euhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud ofeuhlmann has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by Peter Johnson View Post
There's a few challenges with using H.264 for that purpose. While it can use significantly less network bandwidth than MJPEG, there are a number of disadvantages.
  • It requires significantly more processing power to encode (or hardware support, which many platforms we're interested in do not have)
  • It's patent encumbered
  • MJPEG uses frame-by-frame compression, H.264 is temporal across multiple frames, so...
  • Latency is likely higher
  • Often lower quality images
  • It's less friendly to network or frame drops (e.g. if the decoder can't keep up, it takes more time to catch up once it starts dropping frames)
  • Bandwidth usage can vary more dramatically

So there's definitely tradeoffs to consider. As Dustin mentioned, I will be looking at adding support this summer (in particular for direct camera feeds) but patent concerns will likely make it not possible to distribute binaries that support H.264 encoding/decoding unless we can use hardware encoding or one of the Cisco OpenH264 binaries (https://github.com/cisco/openh264/releases). So support will be platform-specific (e.g. some versions of Android could be, the roboRio would not be).
- We have an entire GPU on the Android phones we use currently unused because we didn't bother to install the opencl drivers to have opencv use the GPU. So processing power isn't an issue.
- It's not like we're charging ourselves to use our own stream
- In zerolatency mode there are no B frames
- Quality is fully configurable. For the same apparent visual quality, resolution, and framerate h264 tends to use less bandwidth than mjpeg. That's basically why we want to use it.
- You can configure the rate of I frames however you want. If a packet is dropped, you simply wait for the next I frame. (I should clarify that we intend to use h264 over rtp). This is in contrast with mjpeg over tcp, where a dropped packet means a retry, which (if bandwidth is currently close to the limit) could potentially lead to more retries backing up and even more latency.
- Not if you set it to a fixed bandwidth. Which is what you'd do
__________________
Creator of SmartDashboard.js, an extensible nodejs/webkit replacement for SmartDashboard


https://ligerbots.org
Reply With Quote
  #13   Spotlight this post!  
Unread 03-22-2017, 12:52 AM
Levansic's Avatar
Levansic Levansic is offline
Registered User
AKA: Len Evansic
FRC #0585 (Cyber Penguins)
Team Role: Mentor
 
Join Date: Jan 2012
Rookie Year: 2008
Location: Tehachapi, CA
Posts: 201
Levansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud ofLevansic has much to be proud of
Re: Stream opencv output remotely (like imshow, but for headless video processors)

We found a way to modify the code in the original post of this thread to make a MJPEG stream, and then send that through a Python http server. So now we have a MJPEG stream over http.

Our current stumbling block is figuring out how to display this through the dashboard. We have a custom LabVIEW dashboard, but haven't figured out how to replace the default USB cameras plugged into the RoboRIO, with our new stream.

Here is the code we used to do this:
http://firstteam585.org:7990/project...se/remotecv.py - changed stream to MJPEG
http://firstteam585.org:7990/project...se/testcamp.py - we shut off the canny edges and opened a reference window
http://firstteam585.org:7990/project...jpeg-stream.py - dedicated http server for restreaming

Launch the web server first, before launching testcamp.py.
Reply With Quote
  #14   Spotlight this post!  
Unread 03-24-2017, 12:11 PM
Anon10W1z Anon10W1z is offline
Registered User
FRC #3256 (WarriorBorgs)
Team Role: Programmer
 
Join Date: Jan 2017
Rookie Year: 2017
Location: San Jose
Posts: 80
Anon10W1z is on a distinguished road
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by EmileH View Post
What we were planning on doing (haven't tested this yet though) was sending the byte array object of the image Mat() (we're in Java opencv on the rpi 3) over NetworkTables and then decoding it and displaying it with our own java program running on the driverstation laptop.
I've done this exact thing except with the additional step of converting to JPEG and then to byte[], and decoding back to JPEG on the DS.
Reply With Quote
  #15   Spotlight this post!  
Unread 03-24-2017, 12:25 PM
virtuald's Avatar
virtuald virtuald is offline
RobotPy Guy
AKA: Dustin Spicuzza
FRC #1418 (), FRC #1973, FRC #4796, FRC #6367 ()
Team Role: Mentor
 
Join Date: Dec 2008
Rookie Year: 2003
Location: Boston, MA
Posts: 1,223
virtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant futurevirtuald has a brilliant future
Re: Stream opencv output remotely (like imshow, but for headless video processors)

Quote:
Originally Posted by Levansic View Post
Our current stumbling block is figuring out how to display this through the dashboard. We have a custom LabVIEW dashboard, but haven't figured out how to replace the default USB cameras plugged into the RoboRIO, with our new stream.
So, basically you're doing the same thing as cscore.

If you publish the right stuff over networktables, it'll find it and connect to it. It's not really documented anywhere, but you can look at the source code of CameraServer in wpilib or in robotpy-cscore to figure it out. Or just connect one using the normal stuff, open OutlineViewer, and then emit the same data.
__________________
Maintainer of RobotPy - Python for FRC
Creator of pyfrc (Robot Simulator + utilities for Python), pynetworktables/pynetworktables2js (NetworkTables for Python & Javascript), and lots more...

2017 Season: Teams #1973, #4796, #6367
Team #1418 (remote mentor): Newton Quarterfinalists, 2016 Chesapeake District Champion, 2x Innovation in Control award, 2x district event winner
Team #1418: 2015 DC Regional Innovation In Control Award, #2 seed; 2014 VA Industrial Design Award; 2014 Finalists in DC & VA
Team #2423: 2012 & 2013 Boston Regional Innovation in Control Award


Resources: FIRSTWiki (relaunched!) | My Software Stuff
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 07:08 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi