Switch between two cameras using joystick button (java)


#1

We would like to have two cameras, one on the front and one on the back of our bot. A joystick button would switch from one camera to the other. Any ideas for how to write code for this?


Multiple USB cameras running on SmartDashboard?
MJPEGStream on Shuffleboard
Can the roborio control cameras on the Pi?
#2

We recently figured out how to do this with the stuff built into WPILib.

First you create two (or more if you want) USBCamera objects (not sure if it works on other types, but those are what we are using). After that, you create an MjpegServer. In the initialization you should set all the resolutions and FPS of them to the same to make things work correctly, and then you just have to use SetSource on the MjpegServer to switch which camera you are sending across the stream.

We haven’t figured out how to see the stream on the dashboard, but we can find it from the webpage.


#3

Use CameraServer.getServer() to get a stream that is published to the dashboard. Since you’re using setSource(), you should also change the connection strategy for the cameras to keep them open (otherwise cscore automatically disconnects the unused camera, which increases the delay when switching views). See http://wpilib.screenstepslive.com/s/currentCS/m/vision/l/708159-using-multiple-cameras


Flipping USB cam?
Using 3 cameras this year?
#4

That is really helpful, we were running into issues with the cameras taking a long time to switch and we will test that tomorrow!


#5

Just an update, we switched to keeping the connections open and the switching is almost instantaneous. Much improved over the second or so delay before.


#6

Is it possible to do this if your cameraserver is on the Pi and not the roborio?

Right now I can manually add or remove the two camera streams on the smart dashboard - is there a way to switch which camera is being displayed via the robot code?


#7

Yes, but you have to write a bit of code on both the Pi and the robot. The Pi can’t read the joysticks directly, so you will have to write robot code to read the joystick and publish to NetworkTables, then read the NT and do the camera switching on the Pi.


#8

Couldn’t I just set up the Pi camerserver as a new mjpeg server in the roborio code?

cameraserver.addserver(http://frcvision.local, port)?

then I could have the roborio start and stop automatic capture to turn on/off the smartdashboard view of the camera, right?

Seems like I could do it this way and not write any new code on the Pi.


#9

I have a few usb webcams running on a Pi using the FRC vision image - streaming MJPEG.

Can the roborio code control these cameras (i.e. which one is streaming to the smart dashboard)?

Can I put this in my robot.java:

cameraserver.addserver(http://frcvision.local/, port)

And then start/stop capture?


#10

You can have the RoboRIO act as the switch between two Pi camera feeds and the dashboard, but there are some downsides to this. There will always be slightly higher CPU usage on the the RoboRIO doing it this way (as it needs to handle incoming images from the cameras), but more seriously, setting the resolution/compression on the dashboard will introduce significant latency and CPU usage on the RoboRIO, as the RoboRIO will need to decompress the JPEG image received from the Pi and then recompress it to send to the dashboard. If the switching occurs on the Pi side, the Pi takes this processing burden (and it’s a much faster processor). Additionally, unlike the direct-to-dashboard case, the RoboRIO will need to find the Pi on the network. It’s possible to do this fairly robustly (more robustly than just using the frcvision mDNS address) by using the URLs published by the Pi to NT, but it’s more code (and more complex code) than having the Pi do the switching.

If indeed you want to pursue doing it this way, you will need to create 2+ HttpCamera instances on the RoboRIO as well as a MjpegServer. The HttpCameras will get the video from the Pi (similar to how UsbCamera gets the video from a USB camera). Once you have the HttpCameras, the rest of the switching implementation is similar to the example (e.g. use setSource on the MjpegServer).


#11

Are there any examples online on how to get the Pi to do the switching? I assume the roborio could publish something to network tables and the Pi can act on this. I’m using the standard WPI pi vision image.


#12

No, there aren’t any examples of doing that. But you have the exact idea. On both the robot and the pi, get an entry:

NetworkTableEntry cameraSelect = NetworkTableInstance().getEntry("/cameraSelect");

On the robot in teleopPeriodic() do something like this to pass the button press from the joystick to NetworkTables:

cameraSelect.setBoolean(joy.getButton(5));

On the Pi either add a listener, or put in a periodic loop and poll for changes ala what is shown on screensteps. The listener option would look something like this:

cameraSelect.addListener(event -> {
  if (event.value.isBoolean() && event.value.getBoolean())
    server.setSource(cam1);
  else
    server.setSource(cam2);
}, EntryListenerFlags.kImmediate | EntryListenerFlags.kNew | EntryListenerFlags.kUpdate);

#13

Could code like this be imported via the standard WPI PI image, or do I need to wipe that out to do custom code like this?


#14

That certainly would be nice if the WPI FRCVision image for the Raspberry Pi included this feature (i.e. the ability to remotely control the source of virtual camera) out of the box. Thanks for looking into adding it in the next release!


#15

Each of the example programs implements the same behavior as the “default” program. So you just need to choose a language, download that variant example, add the above, and upload it as the application.

Interesting idea. The tough part about that is picking what default behavior to provide, and it still depends on a team implementing the robot side to actually perform the selection.


#16

Peter
If it matters, the default behavior should be not to stream anything from the virtual camera. And in the network table you could provision a boolean entry to start or stop streaming from the virtual camera. Regardless a string or int entry should be used to specify the underlying physical camera (by name or by index).
It should be pretty easy for all the teams to write to the network table on the robot side to make the selection, and I think that this season pretty much all the teams that use FRCVision would be interested by this feature (as it’s cheap to plug four webcams on a Pi yet the bandwidth limit does not allow to watch four streams at once).
The reason I am pushing for you to natively add this feature in the FRCVision image is that manually updating its code is more complicated/painful because you need to hack the FRCVision image and redo so if the image gets ever updated.
Thanks!


#17

While I’ve opened an issue to add this to the image default program, I’m confused by this statement. What do you mean hack the image and redo? Generally speaking you should just be able to re-upload your program via the web interface after re-imaging. The only reason to update your code will be to pick up new web features (that’s why the upgrade from 2019.1.1 to 2019.2.1 recommended doing that). Of course from year to year will have more major changes, but within the season it shouldn’t.


#18

Thanks for opening an issue. A Plug and Play solution is always best.
I guess I was thinking that ideally the web interface would show information about the virtual camera status but it is not necessary.


#19

I will definitely add it to the web interface when the feature is added. Feel free to comment on the issue with other ideas.


#20

Do you think this could be implemented by bag day? The reason for my asking/pushing is that knowing we could probably rely on this feature being available would let us focus our energy on other important tasks in the short amount of time we have left… Thanks!