OCCRA
Go to Post I am sorry if something was missed at your regional, it will be checked at this event. - Al Skierkiewicz [more]
Home
Go Back   Chief Delphi > CD-Media > White Papers
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

photos

papers

everything



Using JeVois camera in FRC

billbo911

By: billbo911
New: 12-05-2017 10:48 AM
Updated: 12-08-2017 04:42 PM
Total downloads: 447 times


This "Quick Start Guide" will give you all the tools to get started running your Vision Tracking code on the JeVois camera in an FRC environment.

This document makes use of many of the tutorial steps provided by JeVois.org and adds to them a practical and straight forward approach to getting started with the JeVois camera in an FRC environment. The original upload is a preliminary version. As feedback is provided, updated versions will be uploaded. As it is now, this document is fairly complete and should be able to get you tracking with your code. This is written with the assumption you will be using OpenCV and Python. Translating this document for other languages should be fairly simple, but that will be left up to the reader.

Attached Files

  • pdf Using JeVois in FRC v0.31.pdf

    Using JeVois in FRC v0.31.pdf

    downloaddownload file

    uploaded: 12-05-2017 10:48 AM
    filetype: pdf
    filesize: 334.98kb
    downloads: 143


  • pdf Using JeVois in FRC v0.32.pdf

    Using JeVois in FRC v0.32.pdf

    downloaddownload file

    uploaded: 12-05-2017 07:42 PM
    filetype: pdf
    filesize: 336.95kb
    downloads: 97


  • pdf Using JeVois in FRC v1.0 .pdf

    Using JeVois in FRC v1.0 .pdf

    downloaddownload file

    uploaded: 12-08-2017 04:42 PM
    filetype: pdf
    filesize: 533.67kb
    downloads: 205



Discussion

view entire thread

Reply

12-05-2017 11:45 AM

Skyehawk


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by billbo911 View Post
Thread created automatically to discuss a document in CD-Media.

Using JeVois camera in FRC by billbo911, Team 2073.

This is a preliminary release. It is written for OpenCV and Python. Translating it for use with C, C++ ect. should be fairly simple.
Please provide feedback so future versions can be even more complete.
Ballin'. I'll definatally look at this soon. Thanks for the hard work.



12-05-2017 11:49 AM

firecrafty


Unread Re: paper: Using JeVois camera in FRC

Awesome! I've been looking forward to this paper!



12-05-2017 12:11 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by Skyehawk View Post
Ballin'. I'll definatally look at this soon. Thanks for the hard work.
Quote:
Originally Posted by firecrafty View Post
Awesome! I've been looking forward to this paper!
I am certainly glad to provide this. It is our hopes that it will make Vision Tracking something that even more teams can achieve.

Please provide your feedback on the document once you have taken the time to read and work through it. I fully understand that it may not be complete yet. Any and all input will be considered.



12-05-2017 12:28 PM

AllenGregoryIV


Unread Re: paper: Using JeVois camera in FRC

Thanks for posting this, seems like I have a new holiday project.



12-05-2017 01:17 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by AllenGregoryIV View Post
Thanks for posting this, seems like I have a new holiday project.
Wow, that means more to me than you could possibly know.
Why? It was a paper that was produced by Spectrum that inspired me several years ago to dive head first into vision tracking! In fact if you look at our code from as recently as two years ago, you can still see remnants of your code in there.
This paper, while it is not intended to show how to actually track an object, it is intended to show how to run a teams code on the JeVois.
Later this year, 2073 will release our actual code running on the JeVois. Even it will look a bit like what was originally presented by Spectrum. That is due to the fact that the approach is a commonly used approach for identifying targets based on color, shape and size.



12-05-2017 01:48 PM

Tom Bottiglieri


Unread Re: paper: Using JeVois camera in FRC

It looks like you based your trials on this page: http://jevois.org/moddoc/ObjectTracker/modinfo.html
Is that correct?

Does it track at 30FPS? How was the reliability of the platform (shock/power cycling/etc)? How long does it take to boot?



12-05-2017 01:57 PM

marshall


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by Tom Bottiglieri View Post
It looks like you based your trials on this page: http://jevois.org/moddoc/ObjectTracker/modinfo.html
Is that correct?

Does it track at 30FPS? How was the reliability of the platform (shock/power cycling/etc)? How long does it take to boot?
It boots in roughly 5-15s max. It has handled me doing some pretty stupid stuff to the serial lines without issue so far... I connected up ground/power and signal wrong more than once.

The power is USB Mini and not USB Micro... and from my experience, it seems to be a more reliable connector and I like it better for moving projects.

The only weirdness I've seen is that it seems temperamental about certain USB cables but then not all USB cables are made the same and I know I've got some of questionable quality in my USB cable bin.

I've beat it up while it has been on my desk - 10 or so drops from a height of 1-2ft and it keeps the image going.

I've been recommending this thing to a lot of folks and for the price it's worth the investment just to tinker with it.



12-05-2017 02:11 PM

Tom Bottiglieri


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by marshall View Post
...
Thanks for the info. Have you been able to interface with the roborio over USB?



12-05-2017 02:15 PM

marshall


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by Tom Bottiglieri View Post
Thanks for the info. Have you been able to interface with the roborio over USB?
I haven't tried BUT over USB it either shows up as storage or a webcam... I suspect there is a way to get it to talk serial or network over USB too but I haven't looked into it.

I've been playing with it hooked up to my laptop and a VM.



12-05-2017 02:30 PM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by marshall View Post
I haven't tried BUT over USB it either shows up as storage or a webcam... I suspect there is a way to get it to talk serial or network over USB too but I haven't looked into it.

I've been playing with it hooked up to my laptop and a VM.
Ours shows up on tty/ACM0 - at least sending "help" to that port returns back the expected help message. I haven't tried much more than that yet...

I do recall there's a config file in the jevois filesystem which tells the Jevois to use hardware serial or USB serial?



12-05-2017 02:32 PM

marshall


Unread Re: paper: Using JeVois camera in FRC

This might help some of you in your exploration:

https://www.thingiverse.com/thing:2692328



12-05-2017 02:32 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by Tom Bottiglieri View Post
It looks like you based your trials on this page: http://jevois.org/moddoc/ObjectTracker/modinfo.html
Is that correct?

Does it track at 30FPS? How was the reliability of the platform (shock/power cycling/etc)? How long does it take to boot?
Quote:
Originally Posted by marshall View Post
It boots in roughly 5-15s max. It has handled me doing some pretty stupid stuff to the serial lines without issue so far... I connected up ground/power and signal wrong more than once.

The power is USB Mini and not USB Micro... and from my experience, it seems to be a more reliable connector and I like it better for moving projects.

The only weirdness I've seen is that it seems temperamental about certain USB cables but then not all USB cables are made the same and I know I've got some of questionable quality in my USB cable bin.

I've beat it up while it has been on my desk - 10 or so drops from a height of 1-2ft and it keeps the image going.

I've been recommending this thing to a lot of folks and for the price it's worth the investment just to tinker with it.
Hi Tom,
I actually used many of the JeVois.org tutorials to build this guide. I combined information spread out across multiple tutorials and examples. The tracking code we running is a custom modified version of the code we have been using for the last three to four years on our robots.
The code we use to track runs at 60 FPS. It is acquiring frames from the camera in YUYV and sends the target location data to the RoboRio via USB, so at 480Mbs.

There is a bogus concept in the JeVois engine of reporting the FPS based on the time it takes to process the image and create the targeting data. This method claims ~150 FPS. In reality, you can only track as quickly as the camera can acquire images. So, 320 X 240 YUYV maxes out at 60 FPS.

The one thing I have noticed about the Mini USB connector is that it is really snug at first. When it is repeatedly inserted and removed hundreds of times during testing etc., it tends to loosen up. So, a method restrain it is advisable. The TTL connector has shown no signs of loosening up.

Boot time is on the order of < 10 seconds with reboots around 15.

As an additional note, there are four mounting holes in the bottom of the housing that provide a very convenient method of securing the camera.

As for durability, Marshal's description says it all.



12-05-2017 02:42 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by Tom Bottiglieri View Post
Thanks for the info. Have you been able to interface with the roborio over USB?
Quote:
Originally Posted by gerthworm View Post
Ours shows up on tty/ACM0 - at least sending "help" to that port returns back the expected help message. I haven't tried much more than that yet...

I do recall there's a config file in the jevois filesystem which tells the Jevois to use hardware serial or USB serial?
Bingo.
When seting up your scripts to run, you can use either a "script.cfg" file or use the JEVOIS/config/initscript.cfg file to set which serial interface to send data out. So, for your question:
Code:
setpar serlog None
setpar serout USB
This will prevent the error and status messages from being sent over USB and only allow sending tracking data via a command to be sent. ie

Code:
jevois.sendSerial(json_pixels)



12-05-2017 02:44 PM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Awesome awesome stuff, looks super useful already! Gonna toss this at the team tonight and we'll see what we can accomplish!



12-05-2017 03:19 PM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by billbo911 View Post
Bingo.
When seting up your scripts to run, you can use either a "script.cfg" file or use the JEVOIS/config/initscript.cfg file to set which serial interface to send data out. So, for your question:
Code:
setpar serlog None
setpar serout USB
This will prevent the error and status messages from being sent over USB and only allow sending tracking data via a command to be sent. ie

Code:
jevois.sendSerial(json_pixels)
...and to complete the integration between the roboRIO and JeVois, on the roboRIO side you just use the normal WPILib SerialPort API. We've also tested it as part of our 2018 beta testing and it worked fine....great stuff!

- Ron



12-05-2017 03:43 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
...and to complete the integration between the roboRIO and JeVois, on the roboRIO side you just use the normal WPILib SerialPort API. We've also tested it as part of our 2018 beta testing and it worked fine....great stuff!

- Ron
Excellent Ron, THANKS!

We had no intention of covering the roboRIO side in our original document, but maybe it would be a really good thing to add. The only question is, which language(s) to cover. We are a Java team, so that's a no-brainer.
OK, I'll add it to the list.



12-05-2017 03:49 PM

AllenGregoryIV


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by billbo911 View Post
Excellent Ron, THANKS!

We had no intention of covering the roboRIO side in our original document, but maybe it would be a really good thing to add. The only question is, which language(s) to cover. We are a Java team, so that's a no-brainer.
OK, I'll add it to the list.
It's always my philosophy that it is better to share and document what ever you do and don't worry about covering things you don't use. If there is demand someone will publish a labview or C++ example later if you publish the Java code in your document.



12-05-2017 07:45 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

I just uploaded v0.32.
This version includes links to the WPILib serialPort documentation.



12-05-2017 08:40 PM

Tom Bottiglieri


Unread Re: paper: Using JeVois camera in FRC

Thanks for the replies, this product looks really promising. I went ahead and ordered one to try out, maybe it will make it to our robot next year! The Android phone is much more expensive and annoying to deploy to.



12-06-2017 08:11 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
on the roboRIO side you just use the normal WPILib SerialPort API.
Did you use the hardware RS232 port on the RIO, or the USB serial device? We quickly attempted USB serial last night (kUSB), but were getting errors about it not being available. I ran out of time to properly debug, but was curious if anyone else had started down this path yet.



12-06-2017 10:34 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
Did you use the hardware RS232 port on the RIO, or the USB serial device? We quickly attempted USB serial last night (kUSB), but were getting errors about it not being available. I ran out of time to properly debug, but was curious if anyone else had started down this path yet.
Be very careful. The RS232 port on the roboRIO is not compatible with the TTL hardware port on the JeVois without a TTL to RS233 converter.
Use either the USB or the TTL pins in the MXP port.
Honestly, USB is the easiest method.



12-06-2017 11:13 AM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
Did you use the hardware RS232 port on the RIO, or the USB serial device? We quickly attempted USB serial last night (kUSB), but were getting errors about it not being available. I ran out of time to properly debug, but was curious if anyone else had started down this path yet.
We just used USB serial - if you're getting an "in use" exception I'm wondering if you are already using USB elsewhere in the code (like for a NavX for example). If you have NI MAX installed on the laptop, try using that as a first step to confirm the JeVois is connected and communicating (the JeVois should show up as an additional NI VISA device; select it and open the VISA Test Panel and you can try pinging the JeVois just like you would from a serial terminal). I can post screenshots tonight if helpful...

Also in case it helps here's some very simple Java code using IterativeRobot that should ping the JeVois and display the response ("ALIVE" and then "OK") to the console when you enable in teleop. This is a quick cut & paste from our github repo, so apologies in advance if any typos slipped through

Code:
public class Vision extends IterativeRobot {
	static final int BAUD_RATE = 115200;
	
	SerialPort visionPort = null;
	int loopCount = 0;
	
        @Override
	public void robotInit() {
		try {
			System.out.print("Creating JeVois SerialPort...");
			visionPort = new SerialPort(BAUD_RATE,SerialPort.Port.kUSB);
			System.out.println("SUCCESS!!");
		} catch (Exception e) {
			System.out.println("FAILED!!  Fix and then restart code...");
                        e.printStackTrace();
		}
	} 
	
        @Override
        public void teleopInit() {
              if (visionPort == null) return;
              System.out.println("pinging JeVois");
              String cmd = "pingn";
              int bytes = visionPort.writeString(cmd);
	      System.out.println("wrote " +  bytes + "/" + cmd.length() + " bytes, cmd: " + cmd);
        }

        @Override
        public void teleopPeriodic() {
                if (visionPort == null) return;
        	if (visionPort.getBytesReceived() > 0) {
		   System.out.println("Waited: " + loopCount + " loops, Rcv'd: " + visionPort.readString());
		   loopCount = 0;
		} else {
                   ++loopCount;
		}
	}
}



12-06-2017 12:18 PM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Thanks! Our code was largely doing the same things. I think we were gonna use an extra enableTermination() call to auto-do the n, but that's all I'm seeing different now.

I'm used NI-MAX for similar stuff in the past, I'll poke with that as well.

The only thought I had on it was that we have a thumb drive in the top port on the RIO, and the JeVois in the bottom port. By reading the WPIlib docs, I'm wondering if we actually need to be using kUSB2? Even though there's only one device which should present a USB serial port, there are technically two USB devices plugged in....



12-06-2017 08:54 PM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
We just used USB serial ...
So our progress from tonight: https://github.com/RobotCasserole173...team1736/robot

Your code was helpful! Not exactly sure what I did differently tonight, but ~30 test cycles later it seems to be connecting to serial without issue.

Next step is getting the MjpegServer to actually broadcast the image. Still not working at the moment. I am curious if the classes are having issue with the fact the "resolution" passed to the JeVois is not the actual image size that comes back....



12-08-2017 10:28 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

JeVois.org has just released a new image. It is supposed to address the issues with Windows 10!

I have confirmed it does work properly under Windows 10 for me.

I just uploaded the latest, and possibly final, version of the Whitepaper.
Updates included a link to the latest JeVois image, and details on how to receive the targeting data on the roboRIO.

Please feel free to leave feedback. If requests mandate addition information, or corrections, I will update the document.



12-11-2017 08:31 PM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
...
Next step is getting the MjpegServer to actually broadcast the image. Still not working at the moment. I am curious if the classes are having issue with the fact the "resolution" passed to the JeVois is not the actual image size that comes back....
If I'm understanding your comment above, are you passing serial commands from the roboRIO to the JeVois to initiate the video stream?

Have you tried just using the initscript.cfg to start the stream so that the roboRIO just has to pass along what it receives?



12-12-2017 08:31 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by billbo911 View Post
If I'm understanding your comment above, are you passing serial commands from the roboRIO to the JeVois to initiate the video stream?

Have you tried just using the initscript.cfg to start the stream so that the roboRIO just has to pass along what it receives?
We had tried that at one point - based on the response from the "help" command I believe we had selected the proper program. I was thinking initscript.cfg would be the easier way to go, but hadn't confirmed yet.

Where we were getting stuck was that the MJEPG server & USB camera classes had reported that they were connected. Doing a curl of the mjpg stream URL returns text that looks like an mjpg stream (with the boundaries and binary data inbetween). However, no web browser (Chrome, IE, Edge) seems to be able to display it.

We also, with the same software, plugged in a known-working USB camera, which produced a valid displayed image.

I guess what's weird here is I think the JeVois is transmitting video data, but the stream isn't appearing to be valid. We'll do some more debugging tonight, and I'll see if I can upload a snippet of the curl results.



12-12-2017 06:21 PM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
Where we were getting stuck was that the MJEPG server & USB camera classes had reported that they were connected. Doing a curl of the mjpg stream URL returns text that looks like an mjpg stream (with the boundaries and binary data inbetween). However, no web browser (Chrome, IE, Edge) seems to be able to display it.

We also, with the same software, plugged in a known-working USB camera, which produced a valid displayed image.
Looking at the test code that you linked a few posts up, the issue may be that you need to set the pixel format for the UsbCamera....it defaults to MJPG rather than YUV. So unless you've setup a mode on the JeVois to output MJPG, the streamer is probably indeed sending a botched stream.

The following works, at least with the current 2018 beta test release and I think it should with 2017 as well:

Code:
...
visionCam = new UsbCamera("VisionProcCam", 0);
visionCam.setPixelFormat(PixelFormat.kYUV);		
camServer = new MjpegServer("VisionCamServer", MJPG_STREAM_PORT);
camServer.setSource(visionCam);
...
Using the above you should see the JeVois DemoSaliency stream unless you also change the resolution (see below as to why that happens even if you've configured a different start-up mode in the JeVois)

Quote:
Originally Posted by billbo911 View Post
Have you tried just using the initscript.cfg to start the stream so that the roboRIO just has to pass along what it receives?
We've found that the CameraServer seems to always make an initial connection to the JeVois at 640x300, so even if you set initscript.cfg on the JeVois to start a different mode, it will kick over to whatever mode is at 640x300 (DemoSaliency with the default JeVois mapping) when CameraServer initially connects. So for now in our testing we're setting the pixel format, resolution, and fps via the .setVideoMode() method of UsbCamera to make it easier:

Code:
...
visionCam = new UsbCamera("VisionProcCam", 0);
visionCam.setVideoMode(PixelFormat.kYUV, 320, 252, 30);  // start ObjectDetect		
camServer = new MjpegServer("VisionCamServer", MJPG_STREAM_PORT);
camServer.setSource(visionCam);
...
Hope that helps!

- Ron



12-13-2017 08:34 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
Hope that helps!

- Ron

oooooooo! This might just be the secrete sauce we were missing! I don't know if it's worth posting or not, but I did get a curl of what the webcam was streaming. It was producing the usual --boundary markers with a bunch of binary data in between that no web browser (or VLC) could display. Something was definitely coming out, but was scratching my head as to what it was. Tonight or tomorrow, we'll be giving this a shot.

Also, yes, we noticed the initscript limitation. Most programs even seem to have a default which they force the camera into (much to the befuddlement of our programmers). We were able to debug with serial that we were at least starting our code properly, but as soon as anything connected it would swap to some different program. Amcap and VLC both did this, and it's good to know UsbCamera will do the same.



12-14-2017 09:14 AM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
We've found that the CameraServer seems to always make an initial connection to the JeVois at 640x300, so even if you set initscript.cfg on the JeVois to start a different mode, it will kick over to whatever mode is at 640x300 (DemoSaliency with the default JeVois mapping) when CameraServer initially connects. So for now in our testing we're setting the pixel format, resolution, and fps via the .setVideoMode() method of UsbCamera to make it easier
Turns out that our videomappings.cfg on the JeVois had the DemoSaliency module mapping at 640x300 marked as the default. According to the JeVois docs it will announce that default to the USB host (i.e. the roboRIO), so CameraServer probably isn't defaulting to 640x300 itself, but rather just connecting to whatever default that the JeVois advertises.

- Ron



12-14-2017 09:27 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
Turns out that our videomappings.cfg on the JeVois had the DemoSaliency module mapping at 640x300 marked as the default. According to the JeVois docs it will announce that default to the USB host (i.e. the roboRIO), so CameraServer probably isn't defaulting to 640x300 itself, but rather just connecting to whatever default that the JeVois advertises.

- Ron
There are a couple options here.
Edit the videomappings.cfg to point that resolution to your code, or, set the default to your image processing script. Do this by moving the "*" to where you want JeVois to announce it's default. Although, if you are reading the content of the videomapping.cfg file to figure out what was happening, I'm sure you already knew this.



12-14-2017 09:44 AM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Yep, exactly! We were curious why the initial CameraServer connection was always at 640x300 so we dug into it :-)

I'm hoping we'll have time to develop a general "execute GRIP pipeline" module and set of instructions as we get ready for build, which may also make sense to add into the whitepaper.....

- Ron



12-14-2017 10:30 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by rrossbach View Post
...
I'm hoping we'll have time to develop a general "execute GRIP pipeline" module and set of instructions as we get ready for build, which may also make sense to add into the whitepaper.....

- Ron
If you provide the details, the document will have room made in it for those additions.
Can you say "v1.10"?



12-15-2017 12:24 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Pushed code updates from tonight with some after-hours cleanup.

We can get MJPG streaming to run, and can read serial packets from custom code on the Jevois simultaneously.

RIO processor load seems to be super high. We also need to be sure we're getting high framerates from the vision data, but not flooding the usb streaming setup with that same framerate.



12-15-2017 09:08 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
Pushed code updates from tonight with some after-hours cleanup.

We can get MJPG streaming to run, and can read serial packets from custom code on the Jevois simultaneously.

RIO processor load seems to be super high. We also need to be sure we're getting high framerates from the vision data, but not flooding the usb streaming setup with that same framerate.
Are you running the video stream and the tracking data both over USB?
It would be an interesting exercise to run the video over USB, and the tracking data over the TTL serial link.
With that configuration, it would be easy to verify which of the 2 sets of data are pushing the Rio CPU the hardest. It would also be much easier to determine if a higher, or lower, frame rate would be beneficial. For example: Acquire your video at 60 frames fps but only send 15 fps over USB. That way you still get 60 frames per second of tracking data, but not saturate the USB link with the video stream.



12-15-2017 10:15 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by billbo911 View Post
Are you running the video stream and the tracking data both over USB?
It would be an interesting exercise to run the video over USB, and the tracking data over the TTL serial link.
With that configuration, it would be easy to verify which of the 2 sets of data are pushing the Rio CPU the hardest. It would also be much easier to determine if a higher, or lower, frame rate would be beneficial. For example: Acquire your video at 60 frames fps but only send 15 fps over USB. That way you still get 60 frames per second of tracking data, but not saturate the USB link with the video stream.
Yup! It's on the list. First pass was going to to be to establish if the YUV->MJPG conversion was the cause of most of the processor load (my strongest suspicion). After that, yah, playing with framerates, then possibly separating out the serial stream.

The mjpg stream is definitely a "nice-to-have" - I'd really really like to have a nice streamlined single-camera system where the JeVois does double duty as driver assist and vision process. Overall, I would hope the vision process algorithm runs at it's max possible, and we can tone down the video stream to something much slower. Then again, with the way the python architecture is set up, we'll still need some investigation to figure out how to do this. To the docs this weekend



12-15-2017 10:28 AM

AllenGregoryIV


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post

The mjpg stream is definitely a "nice-to-have" - I'd really really like to have a nice streamlined single-camera system where the JeVois does double duty as driver assist and vision process. Overall, I would hope the vision process algorithm runs at it's max possible, and we can tone down the video stream to something much slower. Then again, with the way the python architecture is set up, we'll still need some investigation to figure out how to do this. To the docs this weekend
How would it be the driver assist camera once you tune the contrast for the target? Just for aiming, or changing settings at different times?



12-15-2017 10:34 AM

gerthworm


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by AllenGregoryIV View Post
How would it be the driver assist camera once you tune the contrast for the target? Just for aiming, or changing settings at different times?
It's in another post - basically, send commands to the JeVois to swap between a nice-driver-autoexposure setting, and a vision process setting. When the driver hits the "auto align" button, the camera quickly flips to vision-process exposure and starts searching for targets.

This is as-of-yet untested. Not sure if we can execute the flip fast enough to be useful. But, I still feel it would be nifty if we could get it to work.



12-15-2017 11:16 AM

billbo911


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
Yup! It's on the list. First pass was going to to be to establish if the YUV->MJPG conversion was the cause of most of the processor load (my strongest suspicion). After that, yah, playing with framerates, then possibly separating out the serial stream.

The mjpg stream is definitely a "nice-to-have" - I'd really really like to have a nice streamlined single-camera system where the JeVois does double duty as driver assist and vision process. Overall, I would hope the vision process algorithm runs at it's max possible, and we can tone down the video stream to something much slower. Then again, with the way the python architecture is set up, we'll still need some investigation to figure out how to do this. To the docs this weekend
You can avoid much of the RIO's CPU loading by having JeVois do the YUV to MJPG conversion.

The command to launch a Script, or as you suggested to switch modes, contains all you need to acquire images in YUYV and then stream them over USB as MJPG.

For example:
Code:
MJPG 320 240 15.0 YUYV 320 240 60.0 JeVois EagleTracker
This command launches our tracking code using 60fps from the camera and streams it out USB in MJPG at 15 fps.
Now, as Allen mentioned, we will likely not use this mode. To optimize target identification, we tune the camera to the point that the image is basically useless for the driver.

Now, switching modes on the fly, that is where the command line reconfiguration of the JeVois will really make this camera shine!



12-15-2017 11:18 PM

rrossbach


Unread Re: paper: Using JeVois camera in FRC

Quote:
Originally Posted by gerthworm View Post
The mjpg stream is definitely a "nice-to-have" - I'd really really like to have a nice streamlined single-camera system where the JeVois does double duty as driver assist and vision process.
Quote:
Originally Posted by AllenGregoryIV View Post
How would it be the driver assist camera once you tune the contrast for the target? Just for aiming, or changing settings at different times?
Quote:
Originally Posted by billbo911 View Post
To optimize target identification, we tune the camera to the point that the image is basically useless for the driver.

Now, switching modes on the fly, that is where the command line reconfiguration of the JeVois will really make this camera shine!
Not having had the option in the past to switch on the fly, we actually ended up trying ways to isolate the targets in a normally exposed video source, rather than relying on a heavily tuned/darkened image.

We realized that the green LED ring lights have very little red in their light, and just about all the other ambient light at a typical event has more red it in than the ring lights. Once we realized that, we tried keeping the camera exposure/contrast/etc all normal, and instead of thresholding on HSV values to binarize the image, we "take the red out":

Code:
private Mat binarizeSubt(Mat origImg) {
   // extract the green and red channels;  subtract red from green
   // and threshold the result to get a binary image for contour
   // finding
   Mat greenChannel = new Mat();
   Mat redChannel = new Mat();
		
   Mat diff = new Mat();
   Mat binImg = new Mat(origImg.size(), CvType.CV_8UC1);

   Core.extractChannel(origImg, greenChannel, 1);
   Core.extractChannel(origImg, redChannel, 2);

   Core.subtract(greenChannel, redChannel, diff);
		
   Imgproc.threshold(diff, binImg, 0, 255, Imgproc.THRESH_OTSU);
   return binImg;
}
We then run findContours on the resulting binary image. Subtracting the red channel from the green channel yielded a grayscale image that excluded most artifacts. Doing an adaptive threshold on that gives a nice binary image with prominent targets. Sometimes there are still artifacts (like with the blue tower LED's in Stronghold) but we could easily filter those out based on size, aspect ratio, etc which you want to do anyway. The adaptive threshold takes longer than a standard threshold, but even with that the process is still fast enough to keep up with the normal video frame rate without lag.

A neat side effect of this was that we didn't need to calibrate for different event lighting. The subtraction approach worked consistently even under varying ambient light conditions at different events.



view entire thread

Reply

Tags

loading ...



All times are GMT -5. The time now is 10:39 AM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi