Help: Using PhotonVision AprilTags With Romi - Beta 2023-6

Hello. I’m new to FRC, and as a mentor for Team SPORK - 3196. I am attempting a lesson in using AprilTags using Romi’s with Pi camera’s, with 3d PhotonVision calibrated AprilTag pipeline.

I am able to target an AprilTag in 3d on Photonlib (10.0.0.2:5800). However, I am unable to use the target data from the Photon Camera.

Ask: How does one use PhotonCamera data of a targeted AprilTag using Romi Timed Robot template?

Code sample below:
public class Robot extends TimedRobot {
private static final boolean runCameraTest = true;
private static final String kDefaultAuto = “Default”;
private static final String kCustomAuto = “My Auto”;
private String m_autoSelected;
private final SendableChooser m_chooser = new SendableChooser<>();

private final RomiDrivetrain m_drivetrain = new RomiDrivetrain();

// Constants such as camera and target height stored. Change per robot and goal!
final double CAMERA_HEIGHT_METERS = Units.inchesToMeters(24);
final double TARGET_HEIGHT_METERS = Units.feetToMeters(5);
// Angle between horizontal and the camera.
final double CAMERA_PITCH_RADIANS = Units.degreesToRadians(0);

// How far from the target we want to be
final double GOAL_RANGE_METERS = Units.feetToMeters(3);

// Change this to match the name of your camera
//PhotonCamera camera = new PhotonCamera("phtonvision");
PhotonCamera camera = new PhotonCamera("photonvision");

// PID constants should be tuned per robot
final double LINEAR_P = 0.1;
final double LINEAR_D = 0.0;
PIDController forwardController = new PIDController(LINEAR_P, 0, LINEAR_D);

final double ANGULAR_P = 0.1;
final double ANGULAR_D = 0.0;
PIDController turnController = new PIDController(ANGULAR_P, 0, ANGULAR_D);

//XboxController xboxController = new XboxController(0);
PS4Controller ps4Controller = new PS4Controller(1);

/**

  • This function is run when the robot is first started up and should be used for any
  • initialization code.
    */
    @Override
    public void robotInit() {
    m_chooser.setDefaultOption(“Default Auto”, kDefaultAuto);
    m_chooser.addOption(“My Auto”, kCustomAuto);
    PhotonCamera.setVersionCheckEnabled(false);
//Get the default instance of NetworkTables that was created automatically
  //when your program starts
  camera.setDriverMode(false);  

}

/**

  • This function is called every robot packet, no matter the mode. Use this for items like

  • diagnostics that you want ran during disabled, autonomous, teleoperated and test.

  • This runs after the mode specific periodic functions, but before LiveWindow and

  • SmartDashboard integrated updating.
    */
    @Override
    public void robotPeriodic() {
    double forwardSpeed;
    double rotationSpeed;

    forwardSpeed = -ps4Controller.getRightY();

    //if (xboxController.getAButton()) {
    if (ps4Controller.getCrossButton() && runCameraTest) {
    // Vision-alignment mode
    // Query the latest result from PhotonVision
    var result = camera.getLatestResult();

     if (result.hasTargets()) {
         // Calculate angular turn power
         // -1.0 required to ensure positive PID controller effort _increases_ yaw
         rotationSpeed = -turnController.calculate(result.getBestTarget().getYaw(), 0);
     } else {
         // If we have no targets, stay still.
         rotationSpeed = 0;
     }
    

    } else {
    // Manual Driver Mode
    //rotationSpeed = xboxController.getLeftX();
    rotationSpeed = ps4Controller.getLeftX();
    }

    // Use our forward/turn speeds to control the drivetrain
    m_drivetrain.arcadeDrive(forwardSpeed, rotationSpeed);

    //let’s follow the health of the romi
    SmartDashboard.putNumber("Drivetrain Left Enoder: ", m_drivetrain.getLeftDistanceInch());
    SmartDashboard.putNumber("Drivetrain Left Enoder: ", m_drivetrain.getRightDistanceInch());
    SmartDashboard.putString("PhotonVision Has Image: ", camera.toString());
    SmartDashboard.putNumber("PhotonVision Pipeline Index: ", (double)camera.getPipelineIndex());

}

/**

  • This autonomous (along with the chooser code above) shows how to select between different
  • autonomous modes using the dashboard. The sendable chooser code works with the Java
  • SmartDashboard. If you prefer the LabVIEW Dashboard, remove all of the chooser code and
  • uncomment the getString line to get the auto name from the text box below the Gyro
  • You can add additional auto modes by adding additional comparisons to the switch structure

  • below with additional strings. If using the SendableChooser make sure to add them to the
  • chooser code above as well.
    */
    @Override
    public void autonomousInit() {
    m_autoSelected = m_chooser.getSelected();
    // m_autoSelected = SmartDashboard.getString(“Auto Selector”, kDefaultAuto);
    System.out.println("Auto selected: " + m_autoSelected);
m_drivetrain.resetEncoders();

}

/** This function is called periodically during autonomous. */
@Override
public void autonomousPeriodic() {
switch (m_autoSelected) {
case kCustomAuto:
// Put custom auto code here
break;
case kDefaultAuto:
default:
// Put default auto code here
break;
}
}

/** This function is called once when teleop is enabled. */
@Override
public void teleopInit() {if(runCameraTest){this.cameraTest();}}

/** This function is called periodically during operator control. */
@Override
public void teleopPeriodic() {}

/** This function is called once when the robot is disabled. */
@Override
public void disabledInit() {if(runCameraTest){this.cameraTest();}}

/** This function is called periodically when disabled. */
@Override
public void disabledPeriodic() {}

/** This function is called once when test mode is enabled. */
@Override
public void testInit() {if(runCameraTest){this.cameraTest();}}

/** This function is called periodically during test mode. */

@Override
public void testPeriodic() {
if(runCameraTest){this.cameraTest();}
}

private void cameraTest(){
PhotonPipelineResult photonPipelineResult = camera.getLatestResult();
//let’s follow the health of the romi
SmartDashboard.putNumber("Drivetrain Left Enoder: ", m_drivetrain.getLeftDistanceInch());
SmartDashboard.putNumber("Drivetrain Left Enoder: ", m_drivetrain.getRightDistanceInch());
SmartDashboard.putString("PhotonVision Has Image: ", camera.toString());
SmartDashboard.putNumber("PhotonVision Pipeline Index: ", (double)camera.getPipelineIndex());

if (photonPipelineResult.hasTargets()){
  var bestTarget = photonPipelineResult.getBestTarget();

  SmartDashboard.putNumber("Area: ", bestTarget.getArea());
  SmartDashboard.putNumber("Fidicual ID: ", (double)bestTarget.getFiducialId());
  SmartDashboard.putNumber("Pitch: ", bestTarget.getPitch());
  SmartDashboard.putNumber("Pose: ", bestTarget.getPoseAmbiguity());
  SmartDashboard.putNumber("Skew: ", bestTarget.getSkew());
  SmartDashboard.putNumber("Yaw: ", bestTarget.getYaw());
  SmartDashboard.putString("Corners: ", bestTarget.getCorners().toString());
  SmartDashboard.putString("Separator", "---------------------------------------------");
}

}
}

1 Like

Hey! Could you expand more on what you mean by “unable to use target data”?

A few things to check:
your photonlib version (found in the vendordep.json file) matches the installed version (found in settings)
Your camera name in the PhotonCamera constructor matches what its called in the UI

Also, export your settings and send them over, that will help us debug.

Hi! This is what worked for me: You need to enable the debug network table server in the photonvision settings page, and then create a new network table instance in your code. Then create the photon camera using that networktable instance. Here is what that looks like using the wpilib 2023 beta: (apologies I’m on mobile currently)

var instance = NetworkTableInstance.create();
instance.setServer(“10.0.0.2”, 1735);
instance.startClient3(“client”);

var camera = new PhotonCamera(instance, “photonvision”);

1 Like

Hey mdurran1834. I double checked and the Photonlib and PhotonCamera class both use the default ‘photonvision’ name. Great question. Thank you.

This is great feedback. I am attempting your suggestions and not having success. However, I am certain your approach is correct. I believe there are some Romi configurations I am missing to get this all working. I’m sending all my environment information in a subesequent note.

Any additional thought are appreciated.

A quick note: I am importing the calibration settings each time I restart PhotonVision on the Romi.
Another note: I am unable to connect to port 1735. I think PIGPIO daemon not running correctly is an issue.
Finally: I am sure the NT Instance running the PhotonServer NT server is the correct way to go… I’m just not configured correctly on the Romi.

Version of PhotonVisino

Network Settings
{
“teamNumber” : 3196,
“connectionType” : “DHCP”,
“staticIp” : “”,
“hostname” : “photonvision”,
“runNTServer” : false,
“shouldManage” : true
}

Camera Settings
{
“baseName” : “mmal service 16.1”,
“uniqueName” : “mmal_service_16.1”,
“nickname” : “mmal_service_16.1”,
“FOV” : 70.0,
“path” : “/dev/video0”,
“cameraType” : “UsbCamera”,
“currentPipelineIndex” : 0,
“calibrations” : [ {
“resolution” : {
“width” : 320.0,
“height” : 240.0
},
“cameraIntrinsics” : {
“rows” : 3,
“cols” : 3,
“type” : 6,
“data” : [ 383.4148561521008, 0.0, 167.2674975712665, 0.0, 451.99230169974146, -22.878418622779453, 0.0, 0.0, 1.0 ]
},
“cameraExtrinsics” : {
“rows” : 1,
“cols” : 5,
“type” : 6,
“data” : [ -0.2519255501020508, 1.072541803779521, -0.13312894500981676, 0.0023472241505280832, -0.92467184361143 ]
},
“perViewErrors” : [ 1.1171133464317717, 0.9442032526567572, 1.312902610504925, 1.1973983034105131, 1.7201129343635815, 1.3844132725463103, 2.1242248075038317, 2.1369298204739544, 0.7384900271212791, 0.8964453082859388, 1.1481535305537578, 0.5731627254996762 ],
“standardDeviation” : 0.47922025360993187
} ],
“streamIndex” : 0
}

AprilTag Pipeline Settings
[ “AprilTagPipelineSettings”, {
“pipelineIndex” : 0,
“pipelineType” : “AprilTag”,
“inputImageFlipMode” : “NONE”,
“inputImageRotationMode” : “DEG_0”,
“pipelineNickname” : “AprilTag”,
“cameraAutoExposure” : true,
“cameraExposure” : -1.0,
“cameraBrightness” : 50,
“cameraGain” : -1,
“cameraRedGain” : -1,
“cameraBlueGain” : -1,
“cameraVideoModeIndex” : 0,
“streamingFrameDivisor” : “NONE”,
“ledMode” : false,
“inputShouldShow” : false,
“outputShouldShow” : true,
“hsvHue” : {
“first” : 50,
“second” : 180
},
“hsvSaturation” : {
“first” : 50,
“second” : 255
},
“hsvValue” : {
“first” : 50,
“second” : 255
},
“hueInverted” : false,
“outputShouldDraw” : true,
“outputShowMultipleTargets” : true,
“contourArea” : {
“first” : 0.0,
“second” : 100.0
},
“contourRatio” : {
“first” : 0.0,
“second” : 20.0
},
“contourFullness” : {
“first” : 0.0,
“second” : 100.0
},
“contourSpecklePercentage” : 5,
“contourSortMode” : “Largest”,
“contourTargetOffsetPointEdge” : “Center”,
“contourTargetOrientation” : “Landscape”,
“offsetRobotOffsetMode” : “None”,
“offsetSinglePoint” : {
“x” : 0.0,
“y” : 0.0
},
“offsetDualPointA” : {
“x” : 0.0,
“y” : 0.0
},
“offsetDualPointAArea” : 0.0,
“offsetDualPointB” : {
“x” : 0.0,
“y” : 0.0
},
“offsetDualPointBArea” : 0.0,
“contourGroupingMode” : “Single”,
“contourIntersection” : “Up”,
“solvePNPEnabled” : true,
“targetModel” : “k200mmAprilTag”,
“cornerDetectionStrategy” : “APPROX_POLY_DP_AND_EXTREME_CORNERS”,
“cornerDetectionUseConvexHulls” : true,
“cornerDetectionExactSideCount” : false,
“cornerDetectionSideCount” : 4,
“cornerDetectionAccuracyPercentage” : 10.0,
“tagFamily” : “kTag36h11”,
“decimate” : 1.0,
“blur” : 0.0,
“threads” : 1,
“debug” : false,
“refineEdges” : true,
“numIterations” : 200,
“hammingDist” : 1,
“decisionMargin” : 30
} ]

Starting PhotonVision Beta v6
[2022-03-14 02:01:11] [General - Main] [INFO] Logging initialized in debug mode.
[2022-03-14 02:01:11] [General - Main] [INFO] Starting PhotonVision version v2023.1.1-beta-6 on Linux Raspbian (Pi PI_4)
[2022-03-14 02:01:11] [Camera - PicamJNI] [ERROR] Couldn’t load libpicam shared object
[2022-03-14 02:01:11] [General - Main] [INFO] Native libraries loaded.
[2022-03-14 02:01:11] [General - ConfigManager] [INFO] Loading settings…
[2022-03-14 02:01:11] [General - ConfigManager] [INFO] Hardware config does not exist! Loading defaults
[2022-03-14 02:01:11] [General - ConfigManager] [INFO] Hardware settings does not exist! Loading defaults
[2022-03-14 02:01:11] [General - ConfigManager] [INFO] Network config file does not exist! Loading defaults
[2022-03-14 02:01:11] [General - ConfigManager] [DEBUG] Cameras config folder did not exist. Created!
[2022-03-14 02:01:11] [General - PigpioSocket] [ERROR] Failed to create or connect to Pigpio Daemon socket
[2022-03-14 02:01:11] [General - PigpioSocket] [ERROR] java.net.ConnectException: Connection refused (Connection refused)
at java.base/java.net.PlainSocketImpl.socketConnect(Native Method)
at java.base/java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:399)
at java.base/java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:242)
at java.base/java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:224)
at java.base/java.net.SocksSocketImpl.connect(SocksSocketImpl.java:403)
at java.base/java.net.Socket.connect(Socket.java:591)
at java.base/java.net.Socket.connect(Socket.java:540)
at java.base/java.net.Socket.(Socket.java:436)
at java.base/java.net.Socket.(Socket.java:213)
at org.photonvision.common.hardware.GPIO.pi.PigpioSocketLock.reconnect(PigpioSocketLock.java:47)
at org.photonvision.common.hardware.GPIO.pi.PigpioSocketLock.(PigpioSocketLock.java:43)
at org.photonvision.common.hardware.GPIO.pi.PigpioSocket.(PigpioSocket.java:52)
at org.photonvision.common.hardware.GPIO.pi.PigpioSocket.(PigpioSocket.java:40)
at org.photonvision.common.hardware.HardwareManager.(HardwareManager.java:72)
at org.photonvision.common.hardware.HardwareManager.getInstance(HardwareManager.java:60)
at org.photonvision.Main.main(Main.java:347)

[2022-03-14 02:01:11] [General - NetworkTablesManager] [INFO] Starting NT Client
[2022-03-14 02:01:11] [General - NetworkTablesManager] [ERROR] [NetworkTablesManager] Could not connect to the robot! Will retry in the background…
[2022-03-14 02:01:11] [General - NetworkManager] [INFO] Setting DHCP with team team 0
[2022-03-14 02:01:11] [General - ShellExec] [DEBUG] Executing “cat /etc/hostname | tr -d " \t\n\r”"
[2022-03-14 02:01:11] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:11] [General - ShellExec] [DEBUG] Executing “echo photonvision > /etc/hostname”
[2022-03-14 02:01:11] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:11] [General - ShellExec] [DEBUG] Executing “hostnamectl set-hostname photonvision”
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Executing “sed -i “s/127.0.1.1.*wpilibpi/127.0.1.1\tphotonvision/g” /etc/hosts”
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Executing “sudo service avahi-daemon restart”
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:12] [General - NetworkManager] [INFO] Set hostname to photonvision
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Executing “dhclient eth0”
[2022-03-14 02:01:12] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 02:01:12] [General - NetworkTablesManager] [INFO] NT Connected!
[2022-03-14 02:01:12] [General - ScriptManager] [INFO] Queued event: kNTConnected
[2022-03-14 02:01:12] [Camera - VisionSourceManager] [DEBUG] Matching 1 new cameras!
OR] java.net.ConnectException: Connection refused (Connection refused)

Make sure the network table server is enabled

edit: I’m not sure if these settings are up to date, since you confirmed that the camera name is “photonvision”, but these settings are showing:

Also I would make sure that the romi file system is in write mode if it isn’t already

Very interesting thread. Can you detail the setup for running Beta 6 on the Romi? Are you running the Beta 6 jar only or running the entire Beta 6 on the Romi?

We have created Romi programs from the WpiLib samples.

Thank you.

Re: NetworkTables on the Romi… the way the Romi works is your robot code is running on your desktop computer, so that’s where the NT server is. The Romi is locally running a server that mocks the Driver Station software bits that communicate the IP of the robot program to NT clients. So you can run a NT client on the Romi (including PhotonVision), as long as that client runs StartDSClient() (I don’t know if the PhotonVision implementation does this).

What happens is the NT client connects to the Romi mock DS server, which gives it the IP address of your computer running the robot program, and the client will then connect to your computer. Note for this to work you need to turn off the firewall on your desktop computer so the client can connect.

1 Like

Hey topgun.

We are installing PhotonVision using a manual copy of the wget link in the debain install example. We then run install.sh as sudo. We are able to connect to 10.0.0.2:5800 and setup an AprilTag pipeline. However, we are not able to get to the Network Table that PhotonVision is providing in test mode. We are stuck in the networking between the laptop and the Romi.

So, we have romi running photonvision, with the laptop running photonlib web.

I am planning to use the Romi PhotonVision to prep AprilTag solutions while our robot is being build in 2023.

I will be providing the PhotonVision log when we manually run startup: sudo java -jar /opt/photonvision/photonvision.jar

Here are all the configs/logsphotonvision-settings (1).zip (269.1 KB)

Ultimately, we are getting this error with this code:
Error: NT: ERROR: could not resolve myclient address (TCPConnector.cpp:99)

Code Snipet:
public void robotInit() {
m_chooser.setDefaultOption(“Default Auto”, kDefaultAuto);
m_chooser.addOption(“My Auto”, kCustomAuto);
PhotonCamera.setVersionCheckEnabled(false);

//Get the default instance of NetworkTables that was created automatically
  //when your program starts
  **inst = NetworkTableInstance.create();**

** inst.setServer(“10.0.0.2”, 1181);**
** inst.startClient(“myclient”);**
** camera = new PhotonCamera(inst, “rPi Camera 0”);**

Hey Peter.

I turned off my firewall on the laptop. Still not connecting. I am unsure how to find the proper IP address that PhotonVision is using for its NT server. Here is it’s process below. NOTE: I am getting a PI Gpio error prior to the output below:

[2022-03-14 04:36:32] [General - NetworkTablesManager] [INFO] Starting NT Client
[2022-03-14 04:36:32] [General - NetworkTablesManager] [ERROR] [NetworkTablesManager] Could not connect to the robot! Will retry in the background…
[2022-03-14 04:36:32] [General - NetworkManager] [INFO] Setting STATIC with team team 3196
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “cat /etc/hostname | tr -d " \t\n\r”"
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “echo photonvision > /etc/hostname”
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “hostnamectl set-hostname photonvision”
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “sed -i “s/127.0.1.1.*photonvision/127.0.1.1\tphotonvision/g” /etc/hosts”
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “sudo service avahi-daemon restart”
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 0
[2022-03-14 04:36:32] [General - NetworkManager] [INFO] Set hostname to photonvision
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Executing “ip addr add 10.31.96.2/8 dev eth0”
[2022-03-14 04:36:32] [General - ShellExec] [DEBUG] Got exit code 2
[2022-03-14 04:36:33] [General - NetworkTablesManager] [INFO] NT Connected!
[2022-03-14 04:36:33] [General - ScriptManager] [INFO] Queued event: kNTConnected
[2022-03-14 04:36:33] [General - ConfigManager] [DEBUG] Saving to disk…