paper: Using JeVois camera in FRC

frc
jevois
team2073
#41

Not having had the option in the past to switch on the fly, we actually ended up trying ways to isolate the targets in a normally exposed video source, rather than relying on a heavily tuned/darkened image.

We realized that the green LED ring lights have very little red in their light, and just about all the other ambient light at a typical event has more red it in than the ring lights. Once we realized that, we tried keeping the camera exposure/contrast/etc all normal, and instead of thresholding on HSV values to binarize the image, we “take the red out”: :slight_smile:


private Mat binarizeSubt(Mat origImg) {
   // extract the green and red channels;  subtract red from green
   // and threshold the result to get a binary image for contour
   // finding
   Mat greenChannel = new Mat();
   Mat redChannel = new Mat();
		
   Mat diff = new Mat();
   Mat binImg = new Mat(origImg.size(), CvType.CV_8UC1);

   Core.extractChannel(origImg, greenChannel, 1);
   Core.extractChannel(origImg, redChannel, 2);

   Core.subtract(greenChannel, redChannel, diff);
		
   Imgproc.threshold(diff, binImg, 0, 255, Imgproc.THRESH_OTSU);
   return binImg;
}

We then run findContours on the resulting binary image. Subtracting the red channel from the green channel yielded a grayscale image that excluded most artifacts. Doing an adaptive threshold on that gives a nice binary image with prominent targets. Sometimes there are still artifacts (like with the blue tower LED’s in Stronghold) but we could easily filter those out based on size, aspect ratio, etc which you want to do anyway. The adaptive threshold takes longer than a standard threshold, but even with that the process is still fast enough to keep up with the normal video frame rate without lag.

A neat side effect of this was that we didn’t need to calibrate for different event lighting. The subtraction approach worked consistently even under varying ambient light conditions at different events.

0 Likes

#42

Asid61 has just posted a paper on using GRIP with JeVois in FRC.

0 Likes

#43

Could you use the MXP port on the roborio if you wired it like this:
https://i.imgur.com/CPlR7OR.png

JeVois Serial page:
http://jevois.org/doc/UserSerial.html

0 Likes

#44

The premise of your question is correct, you should be able to connect to the TTL serial pins in the MXP port.
I believe those pins are correct. I just can’t say with 100% certainty that those correct pins on the MXP, but it looks like you are correct.

0 Likes

#45

As I wait for may camera and LED ring assembly to print I been playing some. Seems the Camera feed locks up on me when I open it in GRIP. I get a few frames of video then frozen red X. Any hints to why this may be? I did manage to get the sample pipeline generated and loaded onto the Camera using this guide.

0 Likes

#46

The only time I experienced lock ups was when I had tracking data and video streaming over USB at the same time.
According to the JeVois Technical forums, this should work fine, but my experience is otherwise.
It was actually the AMCap that seemed to have issues with it.
It is highly possible I had some other configuration goofed up.

0 Likes

#47

Does the samples supplied in this paper push data out the usb or serial port or usb? I haven’t dug into the vision code line for line.

0 Likes

#48

Well, we haven’t released the code yet that we have developed for JeVois. That will be coming out in the next week or so.
That said, The beauty of JeVois is you can simply use a single command line to change which route the data takes out. Our current plan is to stream the data over USB, and not stream the video. We may end up using a second JeVois camera to stream video, but we haven’t made that final decision until we know what the game is and what our strategy will be.

0 Likes

#49

I have things sort of working maybe some of you can help. The pdf has the statement

streamon / streamoff ​These two commands are used to manually start the streaming of
images and/or data without the need of video capture software.

I have a Arduino Uno as a passthrough Serial device,
The setpar serout and setpar serlog set to “Hard”

Data does start to dump out the Hardware serial port if I start a AMCap If I try to send a streamon I get

ERR Unsupported command [streamon]

One thing I do get if I type help is a jumbled mess. I presume that is because I’m trying to use 115200 baud on a hardware serial port [UNO only has one hardware port]. The Streaming data from the vision is 99% clean. Every so often there is a stray character.

0 Likes

#50

That is the behavior I would expect.
If the module is started, and streaming is already occurring (started by AMCap), then the “streamoff” is invalid. Using AMCap to start a stream is not a “manual” approach, it launches the stream automatically. Confused yet?

“streamon/streamoff” are usually used when you have a set of commands in the initscript.cfg start your tracking code on power up, or when you launch it from a command line.

Bingo! The Uno shares it’s one serial port between the USB port and TTL pins. That is why the Pro Micro is recommended. They are fairly inexpensive at ~$7.00.
Sometime the stray characters are actually status messages from JeVois. You can use the following command to suppress them if desired.

setpar serlog NONE
0 Likes

#51

Yep I happen to have a UNO and a Nano currently available. (At home I have a MEGA)

I have tried to add

streamon
start

to the initscript.cfg but still no data out unless I open AMCap.
I do now get a list of errors when booting up.

Hardware Serial Monitor


ERR /jevois/config/initscript.cfg:2 Unsupported command ]
ERR /jevois/config/initscript.cfg:10 Unsupported command ]
ERR /jevois/config/initscript.cfg:14 Unsupported command [start]


I entered a usbsd to get the current initscript.cfg files and get
USB Serial Monitor

ERR Command error [usbsd]: Cannot export microSD over USB while streaming: Issue a 'streamoff' command first.

So Current

# JeVois initialization script
#
# This script is run upon statup of the JeVois main engine. You can here specify commands (like you would type them to
# the JeVois command-line interface) to execute upon startup, even before a module is loaded.

# Example: load the SaveVideo with no USB out module (check its mapping index, it was 0 at the time of writing this),
# start streaming, and start saving:
#setmapping 0
#setpar serlog Hard
#setpar serout Hard
#streamon
#start
setmapping 0 # sets the videomapping to the python script

setcam presetwb 0
setcam autoexp 1
setcam autogain 0
setcam autowb 0
setcam redbal 110
setcam bluebal 170
setcam sharpness 16

setpar serlog None	
setpar serout Hard
streamon
start

# Mess with stuff under here if required
setcam brightness 2
setcam contrast 6
setcam gain 16
setcam absexp 50

0 Likes

#52

I deleted the white space and lines 2 and 10 errors went away. now just the start

0 Likes

#53

Here is what our initscript.cfg looks like. I will add comments to it here.

# JeVois initialization script
#
# This script is run upon statup of the JeVois main engine. You can here specify commands (like you would type them to
# the JeVois command-line interface) to execute upon startup, even before a module is loaded.

# Example: load the SaveVideo with no USB out module (check its mapping index, it was 0 at the time of writing this),
# start streaming, and start saving:
#setmapping 0
#setpar serlog Hard Sets where errors and status messages are sent out
setmapping2 YUYV 320 240 30.0 JeVois EagleTrkNoStream This is the actual module to load.
setpar serout USB Set where data is sent out
streamon Start the stream

script.cfg is used to set parameters like exposure etc.

0 Likes

#54

I think some of the issues are in trying to merge the two guides. Yours and https://www.chiefdelphi.com/forums/showpost.php?p=1714200&postcount=42 my initscript is from this other guide.

0 Likes

#55

Thanks for writing this guide. I added the beginner kit to my Amazon wish list and my wife’s parents got it for me for Christmas. As a word of caution to those who plan on getting it, the included USB MicroSD reader was a VERY tight fit in my laptop’s USB ports and it wouldn’t fit in two of them. I’m not sure if mine was on the thick end of the tolerance zone or if they’re all made this way.

0 Likes

#56

Followup - No RIO yet, but testing with my desktop PC - our latest code will stream 352x288 at ~15 FPS (as measured by AMcap) while doing the image acquiring at that same resolution but 60FPS. Our vision pipeline is currently running in about ~6.5ms with multiple targets in view, and streaming serial data at 60 packets/sec.

Powershell scripts written to speed up deploy & debug process while on windows PC’s.

I was able to use your code examples in a number of spots for our processing pipeline and configuration. Thanks for posting again!

One potential pitfall - I believe MJPG Streamer on the RIO will actively open/close the USB cam connection as clients connect and disconnect. I would prefer if either a) the JeVois was able to dynamically swap between programs depending on whether a camera was hooked up or not, or b) MJPG Streamer had an option to keep the camera connected at all times. Till then, we’ll just use two distinct modes on the robot (stream for debug, or no-stream for “production”).

Next steps:

  1. Test RIO code to ensure we can get the same performance on that host platform.
  2. Start tuning some images.
  3. RoboRIO code to extract the proper mapping or resolution settings based on a fixed module name, and the results of the “listmappings” command - should be easier than trying to keep the JeVois and RIO code in sync.
0 Likes

#57

I haven’t fully bought into the Jevois I’m still trying. Here is the mount I created. It uses the hood another Thingivers user created here

https://www.thingiverse.com/thing:2734559

0 Likes

#58

Much like! I think we’ll be building one similar…

0 Likes

#59

I like to support the cables. A couple of 2-52 bolts and a zip tie around the USB does the trick.

0 Likes

#60

One super late follow-up: We finally broke our microUSB port on our initial one after what I would estimate was about 3000 cycles over a year of usage. Would definitely recommend a mount where the connector can be secured relative to the camera, so as not to stress the microUSB port itself.

0 Likes