Re: Vision: what's state of the art in FRC?
 

This goes without saying, but while this thread has a lot of great information ... there is still more out there (within CD) worth finding as some teams may be out of steam to respond now.

One of my goals is to benchmark roboRIO in regards to vision processing, by using the iMac calls within NI Vision, and I'll post results in the FRC Beta Test Central forum. It will be at least two weeks away from now though.

I figure if I tell you this now... it will help motivate me to get it done. ;)

Re: Vision: what's state of the art in FRC?
 

One of these days I should post a pic of the spent PS3 Eye casings in our supply cabinet. They are fantastic cameras, simply because they were designed with computer vision in mind. You also get a lot of flexibility in terms of lenses and filters.

I've been looking out for good USB3 cameras to offer better resolution at a high framerate, but so far nothing can beat the price of the PS3 eye.

Re: Vision: what's state of the art in FRC?
 

I understand there are a number of USB3 board cams on the horizon. What are the details and comparison points of the PS3 eye?

Greg McKaskle

Re: Vision: what's state of the art in FRC?
 

Vision libraries for Windows and the PS3 Eye:
http://codelaboratories.com/products/
http://codelaboratories.com/research...typical-webcam
http://codelaboratories.com/research...ye-disassembly
They also produce the DUO with 2 cameras.

Camera specifications:
http://en.wikipedia.org/wiki/PlayStation_Eye
http://www.psdevwiki.com/ps3/PlayStation_Eye

Removing the IR filter:
http://www.peauproductions.com/cameras.html

Some unusual fun:
http://peauproductions.com/store/ind...ndex&cPath=136
http://peauproductions.com/store/ind...x&cPath=136_16

Re: Vision: what's state of the art in FRC?
 

I need to check out that eye driver. Do you know if it will run on Linux? I also do not know where to find a free version!

Re: Vision: what's state of the art in FRC?
 

I would have to ask your Linux kernel version and distro but the gspca driver for the OmniVision CMOS cameras often finds itself in the newer kernels for free.
So there is a chance if you plug in a PS3 Eye you can just use it with Cheese or VLC.
Standard USB device location finding for Linux may apply.

I suggest you read up on udev and video4linux.
It is actually pretty easy once you nail down the setup to retrieve raw webcam video at the Linux prompt without a GUI.

On Mac OSX check out the current Macam project:
http://webcam-osx.sourceforge.net/

Re: Vision: what's state of the art in FRC?
 

Really?! OpenCV already uses V4L for video input :D. Things should be much easier then!

Re: Vision: what's state of the art in FRC?
 

I am aware of the OpenCV using V4L. I mentioned it elsewhere in the forums.

The issue you need to be aware of is that in several Linux distros when udev finds the webcam it will not map a given webcam the same way consistently.

Without understanding the layers it will be difficult to understand how to craft a udev rule or where to troubleshoot to fix issues you may encounter. So I can either suggest how to build the foundation or I can merely pretend that looking at OpenCV is enough. The value OpenCV really adds is that you get lots of operations you do not have to write to handle raw video and set camera parameters once you have a clear path to communicate to the camera.

I've had students in the past simply use the raw video quite successfully.

Re: Vision: what's state of the art in FRC?
 

Typically a PS3 Eye can do: 640 x 480 @ 60 fps, or 320 x 240 @ 120 fps based on the mfg spec.
The Axis camera we all have has a max 30fps at all resolutions it supports.

Note this is the rate at which the camera can capture frames. Your application will need to download the image, and then process it. So your actual frame rate is dependent upon how fast you can download the image from the camera, and then process it.

For 2014 we limited the fps of our axis camera to 20fps to ease the burden on our processing. Our calculation loop was 10ms (running on a beagle bone), and it took about 60-90ms to download a 640x480 image over our local network and store it into a 2D array which was a compressed jpeg and our images were never more than 20kb. This was over Ethernet, and direct USB should be much faster, but the time it takes to capture the image, and store it in a 2D array even over USB 2.0 is not negligible. Anyone have any good numbers for how long it currently takes them to capture a 640x420 frame and store it over USB?

I am not sure what compression is available on the PS3 eye, but if there isn't any, your 640x480 images will be of much larger size, and thus take longer to download, which will reduce your overall per-frame processing time.

If you already have the camera on hand, then I understand the reason for using them, and I would say just go ahead and try it out now in an off-season test. However, if you are in the market for a new camera, and one that will last for the next couple of years. I would suggest to make sure that the camera you get has built in compression like h.264, and can support at least 30fps at that resolution.

For us, I would rather have 15fps at a higher resolution, then to have higher frame rate at lower resolution. Think about that.

The axis IP cams we have can do this, the Logitech c920 also has built in h.264 encoding, and is a very popular Linux webcam. However, other features you might want to look into is the ability to adjust the camera settings. I do not know the process for adjusting the exposure and shutter settings on the ps3 eye, or Logitech c920, but one of the reasons I like the axis cam is not only can you adjust its settings from its built in web server, regardless of what system you use to process its images, but it includes an API access through the camera URL which allows you to change camera settings on the fly.

This means you can turn your exposure all the way down for auto to do object detection, and then turn it back up in teleop so it can provide a full colormatch recording during a match. More info on the API can be found here. Most people don't know about this functionality built into most AXIS camera products.
http://www.ce.rit.edu/research/proje...TP_API_3_00.pd and http://www.axis.com/techsup/cam_serv...urce_i0_sensor see section 2.2.16

Camera settings are very important, and if not adjusted correctly, you can be wasting unnecessary time processing your camera because the images returned contain a lot more than just the target objects. If your goal is to process objects that are not retro-reflective (like the balls in 2014) then higher resolution will help you, not higher frame rate.

Other very important specs for your camera are shutter time, and light sensitivity when shopping for a camera.

FIRST is targeting official support for the Microsoft LifeCam HD3000, which is a 720P cam, and states it can do 30fps at that resolution. FIRST mentioned they would be sending these cams to certain beta teams, so If we get one, I will be trying it out.

Regards,
Kevin

P.S. Post 200, I should retire from Chief now!

Re: Vision: what's state of the art in FRC?
 

Hadn't looked at the PS3 Eye before. Hard to resist at $9.25 with Prime shipping on Amazon.

Re: Vision: what's state of the art in FRC?
 

This is the rate that the PS3 Eye can capture frames and the rate at which it can send them to the computer. That is important to note. Per one of my links above some webcams play a game where they send you more frames than the sensor can actually produce effectively creating a high frame rate that is bogus.

The PS3 Eye was specifically designed to be able to capture and send those high frame rates. However you are entirely correct if your USB port or the underlying processing is too slow it does not matter if you send 20fps or 120fps most of the frames will be useless and will have to be ignored. In fact the PS3 Eye can bury a USB 1.1 port because the data flow it produces is just too great.

The value is that the PS3 Eye can achieve those frame rates with minimal compression artifacts and right up to the port. What the user chooses to plug into it is up to them.

In the past we had a student pulling raw frames from V4L talking to a PS3 eye at over 50fps (throwing away the extras) in OpenJDK on a dual core netbook legal for a FIRST robot and doing some very simple retroreflective tape tracking. That's pretty fast considering that it wasn't fully compiled code and had the JVM overhead. Obviously with a netbook and potentially the Jetson TK1 (never tried this on the Jetson but the Jetson supports this camera) you can compile code and potentially leverage CUDA so you can raise the frame rate you can fully process very high.

From this link I also provided above:
http://codelaboratories.com/research...typical-webcam

Read down to see how the PS3 Eye got around needing more bandwidth than USB2 offers. They packed and reduced the color data in a specific way.

Compression like that in H.264/MPEG-4 AVC produces artifacts as it removes image data. Hence why the more you compress the video from say an Axis camera the worse it appears.
http://www.cs.ucla.edu/classes/fall0...4_Tutorial.pdf

It also requires the device that is receiving the compression to decode that compression which increases time required to get the raw frames.

Both of these facts are something to be considered if your webcam forces you to use compression to your image processing system. You want to get as close to the actual images as you can even if you are setting up the camera to be more sensitive under a certain circumstance.

The PS3 Eye achieves those frame rates now - it doesn't need changes to make it work - whether your PC or embedded device can handle that pile of inbound data is really up to your choices.

A much better way to retain the data and reduce the flow is setup the camera to reduce the amount of color data per pixel that needs to be sent. For example the PS3 Eye uses YUYV/YUY2 also called YUV 4:2:2

http://linuxtv.org/downloads/v4l-dvb...-FMT-YUYV.html

This means it sends less than 24bit color but because of lighting conditions that is not such a big deal.

There is JPEG compression support in the PS3 Eye as well. Though I suspect it is probably less advanced than the latest H.264 standard. It is a slightly older camera but it was specifically designed with motion and object tracking in mind.

The higher resolution cameras need compression to make it possible to send the color data for the increasingly higher number of pixels down a pipe that is too small to support it. The price you pay for that high resolution is therefore the compression. Though a real upside of the compression can be that if you really tinker with the camera color balance you can let the compression toss away the blacked out portions of the image in the camera which of course means you have already lost image data because of that camera setup.

I literally have a small crate of webcams. Including 5 older Microsoft webcams we bought when we were doing the evaluation the PS3 Eye. The Microsoft webcams were more expensive. The drivers for Linux were not nearly as well developed (we were in contact with the package maintainer) and the quality of the white balance was inconsistent (easily saturated but slow to recover and dramatically sensitive to light conditions). Obviously with each version there is an opportunity for Microsoft to work out how they process their data but the core of that camera is still an OmniVision CMOS sensor and what configuration Microsoft will document and expose is variable. If you would like I will send you for free one of my older Microsoft cameras as they are collecting dust and I will even pay the shipping.

The core point is this:

Sony made the PS3 Eye to track quick movements of game players for their typically stationary console.
Microsoft made their webcam to make videos from a desk.
Neither was designed for mounting on a mobile robot.

If the goal is to track the whole end of the field from half-field you probably need resolution and minimal frames. So in that case the PS3 Eye might not be the best choice.

If you the goal is track the movements of an actuator or say a ball thrown over the bridge then you probably do not need high resolution you need high frame rate. That was a perfect use of the PS3 Eye and it would only have been better if the balls had LED lights in them.

Re: Vision: what's state of the art in FRC?
 

Ordered one this morning. Believe it or not, it will be delivered Sunday. Amazon Prime is now able to hold to 2 day delivery even on Sunday!!

Re: Vision: what's state of the art in FRC?
 

We are probably only going to use only one camera. Even if that is not the case, I have a plan, where I map the cameras using some extremely basic marker detection!

Cameras:
Hope my ASCII art perspective was great.

Basically, using the color of the banner, placed in front, at camera boot (or maybe a small red/green dot drawn on the camera), at program boot, the cameras can be marked as left or right!

Re: Vision: what's state of the art in FRC?
 

+1

Simplify, simplify, simplify. We've had success using vision processing on the robot (2006, 2009, 2012-2014) largely through simplification of the processing.

In 2006 we did in-camera processing in the CMUcam. In 2009, we did on-board processing in the cRIO. In 2012-2014, we did vision processing in the driver station laptop and sent processing results (target information) back to the robot. All of these can be made to work effectively.

In all of these cases, we tried to simplify the problem as much as possible. Use camera exposure, orientation, mounting, etc., to constrain the problem as much as you can. Reduce image sizes as much as you can. (You'll be surprised how few pixels you need; we've never had to go above 320x240, even with a full-court shooter in 2013.) Look for the easiest-possible target.

Think about fault-tolerance in your processing. If the approach you are trying is susceptible to false alarms, or color shift, or ..., change/simplify your approach to avoid the issue; don't be tempted to try to solve the issue by additional processing; try a different approach that works around the problem in a simpler way.

Test your vision processing in as many different environments as possible. Natural light, artificial light, etc. Tweak your approach until is is highly tolerant to image variation.

In short, for the vision approach to be effective, it has to be very reliable or it won't get used. A great way to increase reliability is to keep it as simple as possible.

Lastly, I'll share one of our "secrets" that nobody ever believes because it isn't the "normal vision processing approach": instead of determining range-to-target by size of a target in an image (which typically degrades poorly with lighting changes, color changes, viewing angle, etc.), hard-mount the camera to the robot at a different height than the target and use "elevation of target in field of view" to get range. We came up with this approach in 2006 (when size of target wasn't really an option) and have found it to be just as accurate as size-of-target processing, but much more tolerant of image degradation. At 15' to 20' we can resolve robot "range to target" within 2 inches. In 2013, at a range of 50' (full-court shooter) we had accurate "range to target" within 4 inches.

Re: Vision: what's state of the art in FRC?
 

Was a volunteer at RampRiot 2014 this weekend...

As predicted a few people sent their Axis camera video to their driver's station in most cases just so they could display it. At least 5 teams skyrocketed their missing packet count because the camera consumed the available bandwidth. Only one team had their software setup so that despite missing the packets, not quite up to the point of disable, the robot would generally keep moving the last direction it was given. Everyone else of those 5 teams noticed performance issues as the result of missing inbound direction from the field to the robot. In some cases the lost packets were so often that the robot ended up disabled because the robot software you do not control will disable a robot that does not receive an FMS packet before the timeout.

My personal advice...backed by several years of experience as a CSA and as an experienced network engineer...the available bandwidth you have on the field is at the mercy of the field operations at the competition and the field operations at FIRST. Be prepared if you send video back to the driver's station to potentially have to reduce the quality of that video or even have to turn it off (especially if you do not know how it is configured or can not reconfigure it). I dislike having to point this issue out to teams every year over and over and I do not think it is fair to the teams that stumble into it because they saw other teams do it. For example, if you watched the team that lost 3,500 packets on the field over the course of the match and kept moving, you would swear if you did not understand the details that there was no issue even though 3,500 missed packets were 3,500 directions their robot did not get. Obviously if the team that can loose 3,500 packets is okay with that limitation that is a design choice I have to honor. However even in that case all it would take would be a subtle change in the field and they would loose 2x, 4x or more packets and then they would likely end up disabled because of the timeout as well.

Just be clearly aware - if you send video back to the driver's station you can find yourself at the field on a competition day and see your missing packet count skyrocketing on the field (there's a display next to the field we use to do diagnostics it has 6 x 4 round red/green circles on it) and the FTA can dig in the field software (for a short time after a match) and see graphs that will show your robot is smacking the bandwidth limits. There are other things that can cause this but I am starting to think I should just write up a sign with this information so I can save the talking ;)

Another way teams manage to pull off sending video to the driver's station (other than designing their software to better deal with missed packets) is they will setup the camera so that the picture is no longer really the sort of video people expect (obviously these 5 teams at RampRiot wanted the sort of video people expect from a TV so that is what they sent). In these cases (and Ken just suggested this again) they changed the camera setting so that large portions of the video are drown out. This not only makes identifying a piece of retro-reflective tape easier but it causes the camera compression to crush down the drown out portion of the video because that information is all the same. This is also a compromise. It makes the video stream back to the driver's station even smaller, but again, it is possible you could still eventually find yourself dropping packets because of that. All it would take is for there to be a lot of illuminated retro-reflective surfaces to send, or a change in the field settings.

If it wasn't for the retro-reflective tape the camera setting change would be much more difficult to accomplish. With a proper light source the retro-reflective tape shines brightly back at the camera allowing the camera to be setup in such a way that the rest of the image is very dark or even black and the tape is still visible. If the goal was to track something not lit by a light source these camera setup tricks would not be nearly as effective because you would likely end up sending full color TV style video again.

If you really must send image data back to the field please consider sending a series of images. Each time you can break the socket if you use TCP and that can restrain the bandwidth that will be used. You can also use UDP if you don't mind writing some protocol. Otherwise if you really want to process color video, like the sort of video you would expect on a TV, then consider doing it on the robot and not sending that video back to the driver's station.

Every year someone argues this, teams end up having this issue, and too often if they just pull the plug on the camera the problem related to that just disappears. In some cases they recode a portion of their software so they can turn off the camera without consequences. So I also recommend that you write your software to tolerate a disconnected camera. That is a fast way to troubleshoot this issue. Video to field disappears - problems with missed packets go away - it is surely just a coincidence ;) nothing to see there (especially since you turned off the camera :)).

The deeper problem that shows up is when this is not merely the only problem. So you end up fixing the camera issue just to have that mask another problem. By the time a team is 1 match in and sees the camera problem, then fixes the camera problem in the 2nd match, or maybe the 3rd, then finds the next problem they have lost significant opportunity. Plus they might be having wear and tear between matches such that the camera problem has an even greater impact. So I think this is really something be aware of. Doing this wrong could cost a team a whole competition.