WPILibPi and Romi - how to get camera image from USB camera connected to Romi?

FIRST General Forum
,
1

I have a Pololu Romi with the latest WPILibPi installed on it. On a PC (localhost) connected by wifi to the Romi (10.0.0.2), I am able to build robot java code within the 2023 WPILib vscode and control the Romi using an XBox controller through the WPILib Robot Simulator on the PC. This part works great

If I connect a webcam to the host PC, I am able to see the input video feed on Shuffleboard (connected to localhost) if my robot code contains the following lines:

    UsbCamera camera = CameraServer.startAutomaticCapture();
    camera.setResolution(640, 480);
    CameraServer.getVideo();

I am also able to run some OpenCV processing code on the input images and use CvSource outputStream = CameraServer.putVideo(...) and outputStream.putFrame(...), which shows the processed result on a Shuffleboard camera widget.

However, what I want is for the webcam to be connected to the mobile Romi, not tethered to the bulky PC. In this case, there are two main options for image processing:

  • A) Analyze the Romi webcam image with code running on the Romi raspi, and then transfer the results to the Robot code on the PC (via network tables).

  • B) Somehow (but I don’t know how) transfer the image from the Romi webcam to the PC and have the Robot code on the PC perform the image analysis

Are there any major advantages / disadvantages of A or B?

I think option A (using the Romi raspi as an image co-processor as it was probably designed for) is preferable, but it seems difficult to debug. Specifically, I like the idea of creating different CvSource outputStream(s) and being able to see the intermediate stages of the analysis pipeline on Shuffleboard. That works when the webcam is connected to the PC, but not when it is connected to the Romi, i.e. so far, when I run (python code) output_stream.putFrame(output_img) on the raspi, I cannot see the image streams on Shuffleboard, which makes it very difficult to debug.

For option B (analysis code running on the PC robot code), I can easily see the outputStreams on Shuffleboard when the webcam is connected to the PC. But when the webcam is connected to the Romi (which is what I want), I would need to transfer the raw image data from the Romi raspi to the PC robot code.

To summarize, my questions with the webcam connected to the Romi are:

    1. if the analysis takes place on the raspi, how could I see intermediate stages in the image processing pipeline on Shuffleboard (with Shuffleboard connected to localhost so I can see other SmartDashboard values at the same time)?
    1. if the analysis instead takes place in the robot code running on the PC, how can I transfer the raw webcam image from the Romi to the PC ?

Sorry if I am missing something obvious, and if I am, could you please guide me?

2

You can use the CameraServer class in the Romi image processing code to publish additional streams. CameraServer.putVideo() returns a CvSource and also publishes information about the stream to NT so Shuffleboard will include them in the list of cameras.

Use cscore’s HttpCamera class with the Romi’s MJPEG stream location (e.g. http://<romi-ip>/stream.mjpg).

2 Likes
3

If you’re interested in a pre-packaged solution, PhotonVision is supported. You have to make the filesystem permanently writable in /etc/fstab, IIRC. This camera mount from thingiverse is a nice option for using a Pi-Cam. Else, it will support a USB cam. You can buy a top-plate and mount hardware from Pololu to create a mount point for your camera.

2 Likes
4

Thank you for the advice! When you say “CameraServer.putVideo() also publishes information about the stream to NT”, does that happen automatically within CameraServer.putVideo()? and will it appear in Shuffleboard even if Shuffleboard is connected to the host PC (as opposed to the raspi)?

The ideal case would be to have Shuffleboard simultaneously show SmartDashboard values published from on the PC robot code together with the raspi-connected webcam image processing streams.

5

Yes and yes. The raspi vision processing needs to be running as a NT client and also starting the NT DS client. The Romi service on the raspi creates a “fake” DS server that provides the PC’s IP address to the vision program when the PC is connected, so the vision program connects to the NT server running in simulation on the PC.

2 Likes
6

Thanks for this additional info! This is the step that seems to be problematic for me (skip to C for details, A and B work as expected):

A) When raspi Vision Settings | Network Tables is toggled to Server (i.e. Client toggled off) and when my uploaded Vision python code contains

    ntinst = ntcore.NetworkTableInstance.getDefault()
    ntinst.startServer()

… I can see incoming connections on the Vision Status console when I set Shuffleboard and OutlineViewer to the raspi IP address:

Waiting 5 seconds...
INFO:nt:Listening on NT3 port 1735, NT4 port 5810
INFO:nt:Got a NT4 connection from 10.0.0.156 port 7919
INFO:nt:CONNECTED NT4 client 'shuffleboard@1' (from 10.0.0.156:7919)
INFO:nt:Got a NT4 connection from 10.0.0.156 port 7983
INFO:nt:CONNECTED NT4 client 'outlineviewer@2' (from 10.0.0.156:7983)

This works as expected.

B) When I start a Robot Simulation (which runs an NT server) on the PC and I set Shuffleboard and OutlineViewer to connect to the PC IP address (10.0.0.156 or localhost both work), I can see the Shuffleboard and OutlineViewer connect to the NT Server running in the vscode console:

********** Robot program starting **********
NT: Listening on NT3 port 1735, NT4 port 5810
HALSimWS: WebSocket Connected
********** Robot program startup complete **********
Default simulationPeriodic() method... Override me!
NT: Got a NT4 connection from 10.0.0.156 port 9488
NT: CONNECTED NT4 client 'shuffleboard@1' (from 10.0.0.156:9488)
NT: Got a NT4 connection from 10.0.0.156 port 9490
NT: CONNECTED NT4 client 'outlineviewer@2' (from 10.0.0.156:9490)

This also works as expected.

C) However, when I change the raspi Vision Status | Network Tables to Client, and the python Vision code to

import logging
logging.basicConfig(level=logging.DEBUG)

ntinst = ntcore.NetworkTableInstance.getDefault()
ntinst.startClient4("wpilibpi")
ntinst.setServerTeam(team)
ntinst.startDSClient()

… I never see any connections from the NT client on the raspi to the NT server on the Robot Simulator. I am monitoring the raspi Vision Status console (client) and on the vscode console (server) for any indication of an connection.

What might be going wrong (or am I using calling the wrong functions or using them incorrectly)?

7

In the console is it showing it trying to make a connection to the PC’s IP? If so, your PC firewall is likely blocking it from making a connection.

1 Like
8

I was initially puzzled by your suggestion of the PC firewall, since PC programs (Shuffleboard and OutlineViewer) can successfully connect as clients to the Romi raspi (running as NT server), but the raspi as NT client cannot connect to NT server on PC Robot Simulation. It works in one direction, but not the other.

However, you are indeed correct. After specifically allowing C:\Users\Public\wpilib\2023\jdk\bin\java.exe through the firewall, I immediately see a connection!

It makes sense now and it works! Thank you very much!