Go to Post I was a rookie last year, I didn't expect much. Now I got a taste of what it is, I want more. - davidthefat [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 02-02-2017, 17:46
Acheron-X Acheron-X is offline
Registered User
AKA: Bryan Li
FRC #0449 (Blair Robot Project)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2016
Location: Maryland
Posts: 3
Acheron-X is an unknown quantity at this point
Delay in source switching and high latency for cameras

This is an follow-up to the thread I posted earlier here. The code I am working on is, as before, on Github at this link.

From the previous thread, I was able to get the MjpegServer working along with the camera switching. The code sends the image from the cameras connected to the RoboRIO to SmartDashboard. With the help of the reply in the previous thread to set the URL in SmartDashboard to the correct setting, the camera does display the images from the cameras. However, I am unable to achieve a relatively low latency nor a particularly good FPS.

I am using 160x120 for the camera sizes, with pixels very clear on the screen, and yet the FPS is unable to go above approximately 20. There is also a clear delay, with about .2 or more seconds of latency (e.g., when a hand is waved in front of the cameras, it takes around that time to register). Additionally, I have tried setting the FPS to 30 along with the camera size setting, and simply setting the FPS and not touching the size settings. None of the attempts have worked.

My code also uses
Code:
MjpegServer.setsource();
in order to switch between cameras on button press, and when this is attempted there can be up to several seconds of delay before the image from the other camera appears.

It is most likely not the problem with the cameras (they are HD streaming cameras and work when directly connected to computer), and it is also not the problem with the computer (~20% CPU used, ~50% memory, very low (<5-10%) for the rest) so I was wondering if there was a way to optimize the latency, FPS, and setSource camera switching delay.

The main things that our team wants optimized is the latency and FPS. Help would be appreciated.

Thanks in advance!

- Bryan Li
Team 449 Programmer
Reply With Quote
  #2   Spotlight this post!  
Unread 02-02-2017, 22:24
Peter Johnson Peter Johnson is offline
WPILib Developer
FRC #0294 (Beach Cities Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Redondo Beach, CA
Posts: 269
Peter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud of
Re: Delay in source switching and high latency for cameras

What camera models are you using and how much lighting do you have / what are your camera settings? With some testing with a HD-3000, FPS is 30 with a brightly lit scene, but can drop off to <10 fps given a dark scene as the camera goes into a low light mode. Setting the exposure controls to fixed rather than auto can fix these sorts of issues (open a web browser to http://roborio-449-frc.local:5800/ to get a simple settings gui to try things out).

The main reason you're seeing the delay in switching between cameras is because cscore automatically turns off the camera you're not using, so when you switch it needs to go through the entire connection process all over again. This is intentional to save CPU resources when there's no one connected to the camera stream, but there's a relatively easy workaround: create a CvSink, connect it to the camera, and enable it (it's not necessary to grab frames). This will cause the library to stay connected to the camera (because an enabled sink is always connected). E.g.:

Code:
public CvSink sink1;
public CvSink sink2;

...

sink1 = new CvSink("cam1cv");
sink1.setSource(cam1);
sink1.setEnabled(true);

sink2 = new CvSink("cam1cv");
sink2.setSource(cam2);
sink2.setEnabled(true);
Note you can use the CameraServer wrapper classes to get the other benefits of CameraServer (e.g. NetworkTables publishing), with code like the following rather than calling "new UsbCamera" explicitly.
Code:
cam1 = CameraServer.getInstance().startAutomaticCapture(0);
cam2 = CameraServer.getInstance().startAutomaticCapture(1);
server = CameraServer.getInstance().getServer();

// dummy CvSinks
sink1 = CameraServer.getInstance().getVideo(cam1);
sink1.setEnabled(true);
sink2 = CameraServer.getInstance().getVideo(cam2);
sink2.setEnabled(true);
__________________
Author of cscore - WPILib CameraServer for 2017+
Author of ntcore - WPILib NetworkTables for 2016+
Creator of RobotPy - Python for FRC

2010 FRC World Champions (294, 67, 177)
2007 FTC World Champions (30, 74, 23)
2001 FRC National Champions (71, 294, 125, 365, 279)
Reply With Quote
  #3   Spotlight this post!  
Unread 05-02-2017, 10:03
Acheron-X Acheron-X is offline
Registered User
AKA: Bryan Li
FRC #0449 (Blair Robot Project)
Team Role: Programmer
 
Join Date: Dec 2016
Rookie Year: 2016
Location: Maryland
Posts: 3
Acheron-X is an unknown quantity at this point
Re: Delay in source switching and high latency for cameras

I am testing with two different HD cameras; not sure if this could cause a problem. One is a Microsoft HD-3000 camera, the other is a Genius WideCam F100. The tests are performed in a brightly-lit room, so it seems like the main problem is the delay from the cameras having to go to the roboRIO first then going to the computer.

As for CvSink and CameraServer, I will try to use both your suggestions. Thanks!

EDIT: I'd like to ask if and how CameraServer/CvSink can switch between cameras. It seems like the code you had posted displays both cameras at once, although I haven't tested it yet. Since we will be sending and receiving over a radio/wifi connection, our team wants to minimize the bandwidth usage.

Last edited by Acheron-X : 05-02-2017 at 10:46.
Reply With Quote
  #4   Spotlight this post!  
Unread 05-02-2017, 19:51
Peter Johnson Peter Johnson is offline
WPILib Developer
FRC #0294 (Beach Cities Robotics)
Team Role: Mentor
 
Join Date: Jan 2010
Rookie Year: 2008
Location: Redondo Beach, CA
Posts: 269
Peter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud ofPeter Johnson has much to be proud of
Re: Delay in source switching and high latency for cameras

Quote:
Originally Posted by Acheron-X View Post
I am testing with two different HD cameras; not sure if this could cause a problem. One is a Microsoft HD-3000 camera, the other is a Genius WideCam F100. The tests are performed in a brightly-lit room, so it seems like the main problem is the delay from the cameras having to go to the roboRIO first then going to the computer.

As for CvSink and CameraServer, I will try to use both your suggestions. Thanks!

EDIT: I'd like to ask if and how CameraServer/CvSink can switch between cameras. It seems like the code you had posted displays both cameras at once, although I haven't tested it yet. Since we will be sending and receiving over a radio/wifi connection, our team wants to minimize the bandwidth usage.
Sorry if my message above wasn't clear.. just add the CvSink lines to your existing code (which uses SetSource to switch cameras). All the CvSink lines do is keep the camera connection open so that SetSource can instantly switch rather than being delayed due to taking a while to reconnect to the disconnected camera. It also makes both streams available simultaneously, but you can use SetSource to switch between them the way you were already doing.

I tested with both a HD3000 and a Logitech C210 with your code plus the CvSink lines and could switch instantly. I was getting 30 fps from the HD3000 in a brightly lit environment; the C210 wasn't quite as fast but I didn't spend the time to tweak settings.

In terms of delays/latency, if the client is not requesting a different resolution than the camera is set to (the default case for SmartDashboard or viewing the stream via a web browser), the roboRio is performing minimal processing on the image--basically it just gets the JPEG image from the camera (via the Linux video layer) and forwards it via the HTTP connection; it's about as fast as you can get for a USB camera. Again, it's most likely slower FPS is due to the camera trying to compensate for lighting conditions; I highly recommend you try fixed exposure etc settings on the camera.
__________________
Author of cscore - WPILib CameraServer for 2017+
Author of ntcore - WPILib NetworkTables for 2016+
Creator of RobotPy - Python for FRC

2010 FRC World Champions (294, 67, 177)
2007 FTC World Champions (30, 74, 23)
2001 FRC National Champions (71, 294, 125, 365, 279)
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 22:36.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi