Implementing a Raspberry Pi as a video coprocessor

I have searched for this, but could not find an answer.

We want to use a Raspberry Pi as a video coprocessor. Is there a step by step guide out there of how to hook it up and then access the live video on the driver station? I have downloaded and installed the 2019 FRC Raspberry Pi image.

Also, is there a guide on how to utilize the software in the 2019 FRC image to do vision processing? Our goal is to detect the white lines on the floor to center our robot.

Thanks.
Steve G., 6421 Software Mentor

1 Like

The release page links to this page on screensteps which gives step by step getting started. http://wpilib.screenstepslive.com/s/currentCS/m/85074/l/1027253-what-you-need-to-get-the-pi-image-running

1 Like

Thanks, but I’ve gotten that far. My Raspberry Pi is setup with the FRC image and I can access the control panel with a PC. I need to know the correct way to connect it on the robot (I assume via a network switch mounted on the robot), and how to access the live video on the drivers station. Do I need to implement code on the robot or is it just an HTTP connection on the drivers station?

Correct, you need to connect the Pi to the robot using a wired patch cable to the robot router. You do not need code on the robot to view the stream. You will need code on the robot to act on the feed if you wish to implement code for autonomous driving.

Make sure the client option is selected on the camera setup page and it matches your team number. The smartdash/shuffleboard will pick up the camera and allow for assigning a widget to view the stream from the Pi. The older dashboard, default, probably picks up the camera as well however our team has moved away from that dashboard option.

If we have this up and working at our lab, will it “just work” at the event venue? When they configure the radio will our connectivity to the rPI need to change to connect via the dashboard or via network tables? Do we have to connect to it via IP address? mDNS?

Well, if you connect it to the dashboard, there’s no reason it will function any differently at a match. Your radio will continue to route local traffic the same as in your lab. When you get to the field you’ll still be connecting to your robot wifi which will route the rPi traffic to your driver station dashboard. You can either specify your traffic all be static IPs (some recommend this) or allow your robot radio to hand out DHCP addresses automatically. The dashboards appear to be clever enough to pick up network tables and streams that are available on your local robot network and you can assign them nifty widgets in the shuffleboard.

1 Like

Ok, got RasPi with a camera connected to an enet switch on the robot. Did set up as client with our team number. When I add a camera server stream viewer to the smart dashboard, it does not show any cameras in the properties. Is there anything else I need to do?

The next step I’d take is to see if the camera stream is showing anything on the http://frcvision.local web interface. If that’s a “yes”, what options have you selected in the application tab? Is it just doing a straight streaming (default)? Is it using a python script? Next step, what’s showing in the console in the rPi web portal. If it’s throwing errors there you’re not going to have a good time ™.

I found out that the camera that was working yesterday is no longer working with the Raspberry Pi. It does work on the PC. I tried re-flashing the Raspberry Pi, now it no longer works. Typical.

The microSD card has failed? I had an issue with this as well (with etcher). I had to completely delete all the partitions on the mSD card using gparted in linux and then etcher happily flashed the rPi image again. However, this was using the internal SD card slot on my laptop and it’s always been a bit on the works when it feels like it side.

After a re-flash of the microSD card, I did get it working to a point where I was getting the video image on the drivers station, so thanks for all the help on that.

I still have a problem when adding additional cameras.

On the Raspberry Pi dashboard, video section, I press “Add Connected Camera”. A list shows up with all the cameras I have connected to the Pi (minus the ones that are already connected). Once the camera is added, I press “Open Stream”. It does not find the camera and I end up on a page that says “Cannot reach this page”

Sometimes this happens on the first camera connected to the Pi. Any ideas?

The stream is provided by the streaming application / vision processing code, and does not instantly update. You need to click “Save” in order for newly added cameras to be able to be streamed, as this writes the JSON file with the new configuration and restarts the application (which then re-reads the JSON file to know it should start streaming that camera). I should probably hide the “Open Stream” button for newly added cameras so this is less confusing.

Thanks for the info, I’ll try that.

Didn’t work. If I remove all cameras, then add them back in, it usually requires a restart just to get the first camera online. The other two cameras I have connected never work. If I try to “Open Stream” on them, I get:

Hmmm… cannot reach this page

Try this

Do you need anything from me to diagnose this problem? Thanks.

I just want to confirm again that you are clicking “Writable” and clicking “Save”, correct? If you reboot and the web dashboard doesn’t show the same camera list you had created, it’s not saving. If they are there on a restart, please go to the Vision Status tab in the web dashboard and enable the console, then click “terminate”. This will restart the vision program and you should see it trying to connect to the cameras. Can you post the console output here? The other file that would be useful if there are errors there is the /boot/frc.json file from the Pi.

I was able to get two cameras going. All three are being saved as they all show up when I restart the Pi. Here is the console output after I do a “Terminate”:

CS: rPi Camera 0: SetConfigJson: setting video mode to pixelFormat 1, width 160, height 120, fps 30

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: set format 1 res 160x90

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: set format 1 res 160x120

CS: rPi Camera 0: set FPS to 30

CS: WARNING: : SetConfigJson: could not read pixel format: [json.exception.type_error.302] type must be string, but is null (SourceImpl.cpp:201)

CS: : Connecting to USB camera on /dev/video2

CS: WARNING: : SetConfigJson: could not read pixel format: [json.exception.type_error.302] type must be string, but is null (SourceImpl.cpp:201)

CS: : Connecting to USB camera on /dev/video1

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.lan address (TCPConnector.cpp:99)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host

xxxxx

I was able to get two cameras going. All three are being saved as they all show up when I restart the Pi. Here is the console output after I do a “Terminate”:

CS: rPi Camera 0: SetConfigJson: setting video mode to pixelFormat 1, width 160, height 120, fps 30

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: set format 1 res 160x90

CS: rPi Camera 0: Connecting to USB camera on /dev/video0

CS: rPi Camera 0: set format 1 res 160x120

CS: rPi Camera 0: set FPS to 30

CS: WARNING: : SetConfigJson: could not read pixel format: [json.exception.type_error.302] type must be string, but is null (SourceImpl.cpp:201)

CS: : Connecting to USB camera on /dev/video2

CS: WARNING: : SetConfigJson: could not read pixel format: [json.exception.type_error.302] type must be string, but is null (SourceImpl.cpp:201)

CS: : Connecting to USB camera on /dev/video1

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.lan address (TCPConnector.cpp:99)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)
NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)
NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: connect() to 10.2.94.2 port 1735 timed out
NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: could not resolve roboRIO-294-FRC.frc-field.local address (TCPConnector.cpp:99)

NT: ERROR: could not resolve roboRIO-294-FRC.local address (TCPConnector.cpp:99)

NT: connect() to 172.22.11.2 port 1735 timed out
NT: connect() to 10.2.94.2 port 1735 timed out

NT: connect() to 172.22.11.2 port 1735 timed out

NT: connect() to 10.2.94.2 port 1735 timed out

NT: ERROR: select() to 10.2.94.2 port 1735 error 113 - No route to host (TCPConnector.cpp:173)

NT: ERROR: select() to 172.22.11.2 port 1735 error 113 - No route to host

It looks like you didn’t set the pixel format on at least one of the cameras. I should probably make it not default to blank—you should set it to MJPG or YUYV for each camera.

Also, you should change the team number to your team number so NetworkTables works so they show up in dashboard camera lists, but that’s unrelated to the stream link not working.

Does the raspberry pi FRCVision_image-2019.2.1.zip image include the opencv libs?
when I try to import cv2 in python I get an error
Do I need to install opencv ?

Thank you.
volunteer mentor for team 496
My apologies for asking these simple questions but i have a huge leaning curve and
no one in my team has experience with vision

It has opencv in the image, but only for python3. Are you running python or python3?

okay that work! thank you
I forgot that you need to type python3 to use python3

Thank you !