Running two cameras with photon vision for pose estimation

Technical PhotonVision
1

How would I run two cameras with PhotonVison? We are running ov9281 with an orange pi, and it works well. To run another camera, can we plug it into our orange pi, and does photon vision support running two cameras with the same pipeline on the same coprocessor? Or would I need another orange pi? Also, I’ve got pose estimation working with my current setup, but how would I fuse the data together for pose estimation using two cameras?

2

Yes you can plug in a second camera, but it will be on it’s own pipeline and your O-Pi should be fast enough, but you will see a drop in performance. A second O-Pi is up to you.

Essentially, cycle over both cameras somehow and get the individual pose estimations and latencies which you add to your pose estimator’s vision measurements. How you do this depends on your code design, but if you got this far you should be able to do it.

1 Like
3

Could I see ur code to see how you are updating your pose estimator with the vision readings from the two cameras?

4

I can give you a general idea from our code, but be aware that we’ve structured our code to handle pose estimations from either LL and PV cameras in a general way, so it may be more than you need to deal with if just using PV alone. In fact, trying to explain how we are handling it is a bit more than I have time to get into, other than to say that you just need a way to iterate over your cameras, even if you just write the code to do it twice. Also, we have efficiency issues with scaling beyond 2-3 cameras, so going simpler may be better.

Here is the pose estimator update function we use. It loops over a map of cameras and gets poses from a common data structure. It also has a heuristic for when and when not to apply poses to try and filter out ‘bad’ ones.

This is the VisionValues class that it’s reading from.

Here’s our vision library. It’s a monster.

5

2 cameras on one device need to be different types, or you need to be able to rename the cameras. Arducam has a utility to do this for their cameras. I think it is done with the Serial Number Modification Tool.

6

We are getting better at identical cameras on the same device, provided they never change USB ports, as of 2024.2.7. I’m working on a docs page to explain more visually the actual camera matching process

7

I have two Arducam USB OV9281 and I am pretty sure they are randomly switching which one the photonvision on the PI is referencing in my pipelines. I tried the tool mentioned above but I get this error:
image

this is using the photonvision image for the Orange Pi. Anyone know if I’m going about this wrong?

8

It may not be an official Arducam or there’s a hardware issue. Can you see the picture from the camera using the Camera app (window) ? If so, it is likely a knock off.

We’re having success with three of these: https://www.amazon.com/gp/product/B096ZSSCGH

We renamed and changed their serial numbers and haven’t had an issue. We also have a fish eye ELP model for a driver cam also plugged in. Targeting is working with one (at the rear). One thing to note, they may or may not be focused from the factory. Initially we were not getting good results even at the highest resolution. Plugging them into a computer we worked on their focus and now they work really well. They’re kinda pricey but the global shutter and renameability are worth it.

Now we’re working on getting multi-camera pose working.

9

Actually that’s exactly the same camera we have. I got a reply from arducam support:

Hi,

Please try the tool in the link below:

So maybe the tool has been updated. I won’t have a chance to try this until tonight

1 Like
10

Interesting, it worked with the download from their site. Do you have the 120FPS or the 50FPS version?

11

I think it might be the 120 FPS one

12

Unfortunately the new file link didn’t work either. Feeling a bit stuck. Everything is working well except there’s a 50% chance my 2 cameras get swapped!

13

You could try the 50FPS one or a different model camera. I know for sure the 50fps one can be changed, we’ve changed 3 of 'em. I noticed innomaker was selling with the same model number sensors and we have some but ran into the same problem. We’ll keep them on hand as spares. Does the cameras say Arducam on the back?

14

Yes, I have an official arducam (checked the pcb and verified with the support person). I now have a new version of firmware from the supplier and I’m told this should work but I can’t try it until Sunday. I’ll report back then. Just in case anyone else is having my problem here’s the latest reply:

Please follow the video in the link below to burn the firmware into your camera. Kindly note that you need to replug the camera to take effect after the burning is done.

Then try the ArducamUVCSerialNumber application again.

1 Like
15

Ok, I’m happy to report that after updating to the latest version of photonvision (2024.2.8) and rebooted 4 times and cameras stay in the same place, fingers crossed!