paper: Using JeVois camera in FRC

frc
jevois
team2073

#36

Are you running the video stream and the tracking data both over USB?
It would be an interesting exercise to run the video over USB, and the tracking data over the TTL serial link.
With that configuration, it would be easy to verify which of the 2 sets of data are pushing the Rio CPU the hardest. It would also be much easier to determine if a higher, or lower, frame rate would be beneficial. For example: Acquire your video at 60 frames fps but only send 15 fps over USB. That way you still get 60 frames per second of tracking data, but not saturate the USB link with the video stream.


#37

Yup! It’s on the list. First pass was going to to be to establish if the YUV->MJPG conversion was the cause of most of the processor load (my strongest suspicion). After that, yah, playing with framerates, then possibly separating out the serial stream.

The mjpg stream is definitely a “nice-to-have” - I’d really really like to have a nice streamlined single-camera system where the JeVois does double duty as driver assist and vision process. Overall, I would hope the vision process algorithm runs at it’s max possible, and we can tone down the video stream to something much slower. Then again, with the way the python architecture is set up, we’ll still need some investigation to figure out how to do this. To the docs this weekend :slight_smile:


#38

How would it be the driver assist camera once you tune the contrast for the target? Just for aiming, or changing settings at different times?


#39

It’s in another post - basically, send commands to the JeVois to swap between a nice-driver-autoexposure setting, and a vision process setting. When the driver hits the “auto align” button, the camera quickly flips to vision-process exposure and starts searching for targets.

This is as-of-yet untested. Not sure if we can execute the flip fast enough to be useful. But, I still feel it would be nifty if we could get it to work.


#40

You can avoid much of the RIO’s CPU loading by having JeVois do the YUV to MJPG conversion.

The command to launch a Script, or as you suggested to switch modes, contains all you need to acquire images in YUYV and then stream them over USB as MJPG.

For example:

MJPG 320 240 15.0 YUYV 320 240 60.0 JeVois EagleTracker

This command launches our tracking code using 60fps from the camera and streams it out USB in MJPG at 15 fps.
Now, as Allen mentioned, we will likely not use this mode. To optimize target identification, we tune the camera to the point that the image is basically useless for the driver.

Now, switching modes on the fly, that is where the command line reconfiguration of the JeVois will really make this camera shine!


#41

Not having had the option in the past to switch on the fly, we actually ended up trying ways to isolate the targets in a normally exposed video source, rather than relying on a heavily tuned/darkened image.

We realized that the green LED ring lights have very little red in their light, and just about all the other ambient light at a typical event has more red it in than the ring lights. Once we realized that, we tried keeping the camera exposure/contrast/etc all normal, and instead of thresholding on HSV values to binarize the image, we “take the red out”: :slight_smile:


private Mat binarizeSubt(Mat origImg) {
   // extract the green and red channels;  subtract red from green
   // and threshold the result to get a binary image for contour
   // finding
   Mat greenChannel = new Mat();
   Mat redChannel = new Mat();
		
   Mat diff = new Mat();
   Mat binImg = new Mat(origImg.size(), CvType.CV_8UC1);

   Core.extractChannel(origImg, greenChannel, 1);
   Core.extractChannel(origImg, redChannel, 2);

   Core.subtract(greenChannel, redChannel, diff);
		
   Imgproc.threshold(diff, binImg, 0, 255, Imgproc.THRESH_OTSU);
   return binImg;
}

We then run findContours on the resulting binary image. Subtracting the red channel from the green channel yielded a grayscale image that excluded most artifacts. Doing an adaptive threshold on that gives a nice binary image with prominent targets. Sometimes there are still artifacts (like with the blue tower LED’s in Stronghold) but we could easily filter those out based on size, aspect ratio, etc which you want to do anyway. The adaptive threshold takes longer than a standard threshold, but even with that the process is still fast enough to keep up with the normal video frame rate without lag.

A neat side effect of this was that we didn’t need to calibrate for different event lighting. The subtraction approach worked consistently even under varying ambient light conditions at different events.


#42

Asid61 has just posted a paper on using GRIP with JeVois in FRC.


#43

Could you use the MXP port on the roborio if you wired it like this:
https://i.imgur.com/CPlR7OR.png

JeVois Serial page:
http://jevois.org/doc/UserSerial.html


#44

The premise of your question is correct, you should be able to connect to the TTL serial pins in the MXP port.
I believe those pins are correct. I just can’t say with 100% certainty that those correct pins on the MXP, but it looks like you are correct.


#45

As I wait for may camera and LED ring assembly to print I been playing some. Seems the Camera feed locks up on me when I open it in GRIP. I get a few frames of video then frozen red X. Any hints to why this may be? I did manage to get the sample pipeline generated and loaded onto the Camera using this guide.


#46

The only time I experienced lock ups was when I had tracking data and video streaming over USB at the same time.
According to the JeVois Technical forums, this should work fine, but my experience is otherwise.
It was actually the AMCap that seemed to have issues with it.
It is highly possible I had some other configuration goofed up.


#47

Does the samples supplied in this paper push data out the usb or serial port or usb? I haven’t dug into the vision code line for line.


#48

Well, we haven’t released the code yet that we have developed for JeVois. That will be coming out in the next week or so.
That said, The beauty of JeVois is you can simply use a single command line to change which route the data takes out. Our current plan is to stream the data over USB, and not stream the video. We may end up using a second JeVois camera to stream video, but we haven’t made that final decision until we know what the game is and what our strategy will be.


#49

I have things sort of working maybe some of you can help. The pdf has the statement

streamon / streamoff ​These two commands are used to manually start the streaming of
images and/or data without the need of video capture software.

I have a Arduino Uno as a passthrough Serial device,
The setpar serout and setpar serlog set to “Hard”

Data does start to dump out the Hardware serial port if I start a AMCap If I try to send a streamon I get

ERR Unsupported command [streamon]

One thing I do get if I type help is a jumbled mess. I presume that is because I’m trying to use 115200 baud on a hardware serial port [UNO only has one hardware port]. The Streaming data from the vision is 99% clean. Every so often there is a stray character.


#50

That is the behavior I would expect.
If the module is started, and streaming is already occurring (started by AMCap), then the “streamoff” is invalid. Using AMCap to start a stream is not a “manual” approach, it launches the stream automatically. Confused yet?

“streamon/streamoff” are usually used when you have a set of commands in the initscript.cfg start your tracking code on power up, or when you launch it from a command line.

Bingo! The Uno shares it’s one serial port between the USB port and TTL pins. That is why the Pro Micro is recommended. They are fairly inexpensive at ~$7.00.
Sometime the stray characters are actually status messages from JeVois. You can use the following command to suppress them if desired.

setpar serlog NONE

#51

Yep I happen to have a UNO and a Nano currently available. (At home I have a MEGA)

I have tried to add

streamon
start

to the initscript.cfg but still no data out unless I open AMCap.
I do now get a list of errors when booting up.

Hardware Serial Monitor


ERR /jevois/config/initscript.cfg:2 Unsupported command ]
ERR /jevois/config/initscript.cfg:10 Unsupported command ]
ERR /jevois/config/initscript.cfg:14 Unsupported command [start]


I entered a usbsd to get the current initscript.cfg files and get
USB Serial Monitor

ERR Command error [usbsd]: Cannot export microSD over USB while streaming: Issue a 'streamoff' command first.

So Current

# JeVois initialization script
#
# This script is run upon statup of the JeVois main engine. You can here specify commands (like you would type them to
# the JeVois command-line interface) to execute upon startup, even before a module is loaded.

# Example: load the SaveVideo with no USB out module (check its mapping index, it was 0 at the time of writing this),
# start streaming, and start saving:
#setmapping 0
#setpar serlog Hard
#setpar serout Hard
#streamon
#start
setmapping 0 # sets the videomapping to the python script

setcam presetwb 0
setcam autoexp 1
setcam autogain 0
setcam autowb 0
setcam redbal 110
setcam bluebal 170
setcam sharpness 16

setpar serlog None	
setpar serout Hard
streamon
start

# Mess with stuff under here if required
setcam brightness 2
setcam contrast 6
setcam gain 16
setcam absexp 50


#52

I deleted the white space and lines 2 and 10 errors went away. now just the start


#53

Here is what our initscript.cfg looks like. I will add comments to it here.

# JeVois initialization script
#
# This script is run upon statup of the JeVois main engine. You can here specify commands (like you would type them to
# the JeVois command-line interface) to execute upon startup, even before a module is loaded.

# Example: load the SaveVideo with no USB out module (check its mapping index, it was 0 at the time of writing this),
# start streaming, and start saving:
#setmapping 0
#setpar serlog Hard Sets where errors and status messages are sent out
setmapping2 YUYV 320 240 30.0 JeVois EagleTrkNoStream This is the actual module to load.
setpar serout USB Set where data is sent out
streamon Start the stream

script.cfg is used to set parameters like exposure etc.


#54

I think some of the issues are in trying to merge the two guides. Yours and https://www.chiefdelphi.com/forums/showpost.php?p=1714200&postcount=42 my initscript is from this other guide.


#55

Thanks for writing this guide. I added the beginner kit to my Amazon wish list and my wife’s parents got it for me for Christmas. As a word of caution to those who plan on getting it, the included USB MicroSD reader was a VERY tight fit in my laptop’s USB ports and it wouldn’t fit in two of them. I’m not sure if mine was on the thick end of the tolerance zone or if they’re all made this way.