Optical Mouse Navigation

[phrontist]

A little after build season was over for the 2004 season I started thinking about new ideas for this year's season, particularly autonomous mode. I was poking around CD when I saw a post by Craig Putnam in which he mentioned he was working on an optical mouse based navigation system. What a great idea, I thought! I thought about it periodically during the off season and eventually I had come up with how I wanted to do this, and had almost everything figured out except how to interface the mouse, or mice, to the RC. So I emailed Craig Putnam and he explained his progress on making such a system and much to my delight I found that much of what he had done was what I was planning to do, so I couldn't be too far off track. He also mentioned something mind-boggling...

But before I talk about that I'd like to mention something else. When I first got this piece of information from Mr. Putnam a few days ago, I didn't tell anyone and hoped he wouldn't either. This was pretty stupid for a variety of reasons, which don't need to be enumerated. Anyway, in the past few days I've seen several posts that here that are dancing around the problem, namely the following:

• TTL Serial Port and PS/2 Mouse Interface
• External Lens for Optical Mouse
• New Serial Port Driver
• Looking for Mouse Part

I realized that a lot of people are now working on Optical Mouse Navigation, and it wouldn't be fair to not share everything. So then...

There is a chip that converts PS/2 to RS232, made specifically for interfacing PICs to mice! It's from Al-Williams and it's called the PAK-XI. It slices, it dices, it makes a mean Crème Brûlée! Mr. Putnam, a mentor from team 42, P.A.R.T.S., tipped me off to it after I emailed him about it. But wait, theres more! It's also cheap! Hey! So what's the catch? Well, it may or may not be legal.

Here are a few snippets from my correspondence with Mr. Putnam:

Originally Posted by Phrontist

Panic!

I'm really hoping I'm wrong, but isn't the PAK-XI illegal for use in
FIRST accoording to last year's rule:

<R71>Additional electronic components for use on the robot must be
currently available from or equivalent to those available from Newark
InONe (http://www.newarkinone.com), Future Active
(http://future-active.com), Radio SHack (http://www.radioshack.com) or
Digi-key Corporation (http://www.digikey.com). Additional electronic
components include any object that conducts electricity other than IFI
relays and voltage controllers, wires, connectors and solder. The
total catalogue value of additional electronic components must not
exceed $300.00 USD. This cost is counted as part of the $3,500 limit.
No single electronic component shall have a catalogue value of over
$100.00 USD

Is the whole system in dire peril, dependent on a rule change? Can it
be argued that their chip is essentially equivalent to a PIC with
special software on it? For that matter couldn't it be argued that a
massive collection of bipolar semiconducters could theoretically do
the same thing ;-)

Originally Posted by Mr. Putnam

Oh good grief! Somehow, when I started this project way back when, I had
convinced myself that the PAK chip was OK to use. Given the current rules
though, it would seem problematic.

I suspect that the PAK chip is indeed nothing more than a PIC (or
equivalent) microprocessor with the software already developed and loaded.
Whether we could get a ruling from FIRST on this at this point (before
kickoff) is probably also problematic but worth a try. I'll send them a
message tonight and see what happens.

I will continue on with this project anyway - just because it is a challenge
and useful in teaching our courses. But it will be interesting to see
whether or not it can be used on a FIRST robot. I'd sure hate to see it not
be usable due to the lack of a $5 interface chip!

Originally Posted by Phrontist

*fingers crossed*

Originally Posted by Mr. Putnam

Well, I sent my question off to FIRST last night and got a response today.
Or, more accurately, a non-response (which didn't really surprise me very
much).

As I expected, they won't comment on any possible changes in rules for the
upcoming season. That's not a surprise as it could give one team a
competetive advantage over another.

Regarding whether use of a PAK chip would have been legal last year or not -
they couldn't say.

While poking around on the Microchip web site I found some application notes
and technical briefs that may be helpful. There must be a way to communicate
with the mouse without using the PAK chip. After all, the PAK chip itself
does that and I suspect it is probably some flavor of PIC microprocessor
itself (or something equivalent). You can see some examples of how Microchip
has done that if you look at TB055 and AN956. In the latter case, they are
using USB, but switching from a PS/2-based mouse to a USB-based mouse should
be a trivial exercise.

Good luck!

(I've already ordered some of those chips, and I'm not quite sure if I'm going to go ahead using them and risk their illegality, ideally I can find another solution)

Originally Posted by Phrontist

Well, now everything has come full circle!

I initially emailed you after finding nothing on the web about this
task except for those documents on the Microchip site. Oh well.

Were you able to find C18 versions of this kind of thing, because
there is little chance that I'll be learning assembler in time to make
this thing work for shipment. Bah.

Mr. Putnam has also put together a great presentation on the topic which can be found here.

I'm currently building a prototype mouse assembly, and the problem of the moment is sufficent illumination. Here are some more snippets from my correspondance with Mr. Putnam.

Originally Posted by Phrontist

We've been scratching our heads as far as illumination goes, in your
presentation there was a picture of mighty LED cluster. Now, from
expermentation I've ascertained that the LED's in mice are modulated,
and that the CMOS will only work with images illuminated with
modulated light. How did you add more LEDS? We've thought about just
making a cluster and soldering it to the old LED terminals but fear
this will cause too much draw and upset the circuit. Your help is
greatly appreciated.

Oh, and what about using IR leds? I've seen sites on which
case-modding folks do this just to be different, but rumor also has it
that the sensor is more sensitive to infared light.

Originally Posted by Mr.Putnam

Modulated huh? Hmmm... It never occurred to me to even consider that!

I simply found and whacked in a "mighty LED cluster" (as you so succinctly
put it!) and it worked just fine. The LED clusters are replacement tail
lights that you can get for cars and trucks. I got mine from
http://www.superbrightleds.com/. They come in various "strengths" (number of
LEDs in the array) and dispersion patterns. Getting them from this source is
also a whole lot less inexpensive than going to an auto parts store
(something like 1/2 to 1/3 the price for the exact same unit!).

I don't power the cluster from the mouse for the same reason you cited. I
just run it directly from the 12V supply. Even the biggest array I have
(which is the one in the photo you saw) draws very little.

Now you've got me wondering whether I got lucky with the mouse I used. I
don't recall off-hand the manufacturer of the mouse but I'm sure I can
figure it out once I get back to my office (this is my home email address).
I *do* recall that it uses an Agilent chip, but again I can't tell you right
now which one.

Perhaps the LED is not modulated on my mouse. Or perhaps the little clear
plastic piece that fits over the *other* side of the chip (from where the
optics & floor are located) supplies enough light from the board-mounted LED
(which I left in place) to satisfy the chip. I have had a feeling that the
plastic piece was some kind of a light pipe as it seems to direct some of
the LED's light onto the back side of the chip. I think you can see that
piece in one of the shots of the circuit board in the presentation.

The web site for SuperBrightLEDs has specs on their LEDs as does the Agilent
web site for their chip. I have the details at the college, but the
wavelength of the peak output of the LED array I am using is almost exactly
what is expected by the chip. As I recall it is well inside the visible
(red) spectrum.

Originally Posted by Phrontist

The plot thickens!

Well, I simply guessed it may be modulated after waving my finger back
and forth and finding that if I did it at just the right speed I could
see several fingers. Ultra-scientific, I know. Maybe dragging out ye
olde O-Scope is in order.

We've modified a mouse chip by resolding it's LED to the same
terminals, but on the bottem of the chip. Our thinking is that when
the mouse detects a change it "brightens" the LED until a period of
non-change occurs. Thus we can see if the CMOS is picking up anything
at all by simply holding it at various heights without any optics on
the CMOS. By carefully adjusting the angle of the LED we've
ascertained that 4 inches (measured from the bottem of the chip to the
ground) is the highest we can go before the LED brightness "times out"
indicating the CMOS is detecting zilch. Now we're not putting a whole
lot of stock in this, for obvious reasons.

Here is another idea, if it is modulated, connecting the LED cluster
directly isn't the only way to modulate them acoordingly. Couldn't
some sort of solution be reached with a sort of "amplifyer"? Maybe I'm
just being daft but couldn't a semiconducter or two correctly used
modulate a larger current based on a smaller one.

Hmm, looking at your diagram in the presentation, I'm beginning to
think that you got lucky. I think a single LED is sufficient
illumination to get good data at the distance shown in your diagram.

So this is where things stand at the moment. The purpose of this thread is to have a central place for everyone working on this to talk about it. Please post anything you find that's relevant.

(Innumerable thanks to Mr. Putnam, who doesn't seem to post on CD to often. I hope he doesn't mind me quoting him! Note: These are out of chronological order, for the sake of clarity)

[NoRemorse]

I'll admit that I did read almost any of that post, but... I had this idea last year. We ruled out that idea becuse of one main reason: The maximum rate of speed that the mouse can mesure was well under the speed of our robot. That might be somthing to look into if your thinking about a mouse for navigation.

[phrontist]

Originally Posted by NoRemorse

I'll admit that I did read almost any of that post, but... I had this idea last year. We ruled out that idea becuse of one main reason: The maximum rate of speed that the mouse can mesure was well under the speed of our robot. That might be somthing to look into if your thinking about a mouse for navigation.

If you had bothered to read Mr. Putnam's excellent presentation, you would have noticed that the problem is solved by raising the mouse off the ground and the addition of some optics. Furthermore, if the PAK-XI chip is allowed, it can supposedly adjust the resolution of the mouse it's connected to. The chip isn't anything special either, and any coprocessor based solution could do this. (Some?) Mice are programmable.

[jonathan lall]

Well now that it's out in the open, we've toyed with this concept for some time as a terrain-mapping navigational system, but we never got past the conceptual stages because of the specs on the sensor devices. I was pushing for some research into this the past two years on my team actually, and we did (we believe) figure out how to get two mouse inputs into the controller (we would place one on either side of the robot and judge distance and pitch from those readings), but the main problem with this was purely technical on the sensor side. Simply put, your robot would have to be going very slowly for the sensor to be able to position you. A robot that blasts off at say 10 feet per second in autonomous mode is by a large margin (I don't have the numbers offhand) too fast for such a device. The framerate is not high enough such that the sensor (which is really a camera) can take two frames and compare them. Thus, no movement would be detected. Perhaps we've missed something, but our preliminary research into this lead us to that conclusion.

[phrontist]

Originally Posted by jonathan lall

Well now that it's out in the open, we've toyed with this concept for some time as a terrain-mapping navigational system, but we never got past the conceptual stages because of the specs on the sensor devices. I was pushing for some research into this the past two years on my team actually, and we did (we believe) figure out how to get two mouse inputs into the controller (we would place one on either side of the robot and judge distance and pitch from those readings), but the main problem with this was purely technical on the sensor side. Simply put, your robot would have to be going very slowly for the sensor to be able to position you. A robot that blasts off at say 10 feet per second in autonomous mode is by a large margin (I don't have the numbers offhand) too fast for such a device. The framerate is not high enough such that the sensor (which is really a camera) can take two frames and compare them. Thus, no movement would be detected. Perhaps we've missed something, but our preliminary research into this lead us to that conclusion.

I'm not sure what sensor you are refering to specifically, but I'm fairly confident that a mouse can resolve at that speed accurately with the correct optics. If it's "zoomed out" by a factor of 5, wouldn't it logically follow that it's seeing 1/5 the movement, and hence it's normally 12 inch per second max become a 5 foot per second max? I'm not saying it's a solved problem, but it's certainly not a desperate situation by any means.

Mr. Putnam is using an accellerometer to gauge odometry, and the mouse to measure rotation. I was planning on using a mouse for both, each one going through a PAK-XI and then to a serial port. This is possible through Kevin Watson's new serial port driver. This would make debugging hard, as Max Lobovsky pointed out, because you couldn't use printf's to send data back to your laptop anymore, as all the ports would be used. His idea was to multiplex the two streams...

Questions Questions!

[phrontist]

Okay, I think my plan is to go ahead and use the PAK-XI's on the offchance they'll be allowed, while also working on code (In C, INSTEAD OF THIS ASM NONSENSE ) to replicate the function of the PAK-XI should it be banned. Does anyone have any idea which PIC would be best suited for simultaneous PS/2 and RS232 communication?

[Dave Flowerday]

Wildstang experimented with optical mice last year as well. I'm having a hard time telling from that long post just how far others have gotten. Has anyone actually communicated with their mouse?

Last year we were able to get an optical mouse connected to our robot controller and read X/Y positioning from the mouse chip. We did not use PS/2, however. Instead we removed the PS/2 driver chip from inside the mouse and connected the RC directly to the optical sensor chip from Agilent (BTW, Agilent is the only company making optical mice components, so if you open one up that is what you'll find). The Agilent chip can speak in a simple synchronous protocol which is how we communicated with it. We implemented this using the normal input/output pins on the RC and bit-banging the protocol. This allows us to read out the X/Y deltas as well as obtain the actual image captured by the mouse, the surface quality measurement, and a bunch of other goodies. The chip is pretty nice because it will remember how far it's moved since the last time you queried it, so you can query it at your leisure (as long as it's fast enough that the counter inside the sensor doesn't overflow).

We also affixed a different lens to the mouse to change its field of vision to accommodate larger speeds. Illumination is a problem, as was already mentioned. We fixed a ring of superbright LEDs around the lens to combat this. However, that is where the bad news starts. With all that out of the way, we mounted the modified mouse to a cart and started doing tests to see if it accurately tracked motion. We found that it did not. When the cart was moved at different speeds over the same distance, the mouse would report different amounts of measured distance. This was disappointing, because before we fitted the new lens we tested the mouse by affixing it to an X-Y table in our model shop with a very accurate readout and found that without the new lens it was good to something like a few thousandths of an inch. At this point it was getting pretty late in the season so with the mouse concept not looking too promising we had to abandon it and concentrate on reusing our old positioning system that we used in 2003.

Agilent's site is pretty useful, with datasheets for the various mouse sensors. If you dig around there you will find that they give you lots of info on the optics used as well as what wavelengths the sensors are most sensitive to, etc.

Also something to keep in mind is that even if you get to the point where the mouse can track movement accurately, it does not handle rotation. You'll still need a compass or something to know your heading which needs to be combined with the vector obtained from the mouse. I think you can substitute two mice on opposite sides of the robot instead of the gyro, but I haven't yet worked through the math to prove it to myself. There's some odd cases if you do a tank-style spin and such that I have to think about a little bit to see if it can still work.

[phrontist]

Originally Posted by Dave Flowerday

Wildstang experimented with optical mice last year as well. I'm having a hard time telling from that long post just how far others have gotten. Has anyone actually communicated with their mouse?

Last year we were able to get an optical mouse connected to our robot controller and read X/Y positioning from the mouse chip. We did not use PS/2, however. Instead we removed the PS/2 driver chip from inside the mouse and connected the RC directly to the optical sensor chip from Agilent (BTW, Agilent is the only company making optical mice components, so if you open one up that is what you'll find). The Agilent chip can speak in a simple synchronous protocol which is how we communicated with it. We implemented this using the normal input/output pins on the RC and bit-banging the protocol. This allows us to read out the X/Y deltas as well as obtain the actual image captured by the mouse, the surface quality measurement, and a bunch of other goodies. The chip is pretty nice because it will remember how far it's moved since the last time you queried it, so you can query it at your leisure (as long as it's fast enough that the counter inside the sensor doesn't overflow).

We also affixed a different lens to the mouse to change its field of vision to accommodate larger speeds. Illumination is a problem, as was already mentioned. We fixed a ring of superbright LEDs around the lens to combat this. However, that is where the bad news starts. With all that out of the way, we mounted the modified mouse to a cart and started doing tests to see if it accurately tracked motion. We found that it did not. When the cart was moved at different speeds over the same distance, the mouse would report different amounts of measured distance. This was disappointing, because before we fitted the new lens we tested the mouse by affixing it to an X-Y table in our model shop with a very accurate readout and found that without the new lens it was good to something like a few thousandths of an inch. At this point it was getting pretty late in the season so with the mouse concept not looking too promising we had to abandon it and concentrate on reusing our old positioning system that we used in 2003.

Agilent's site is pretty useful, with datasheets for the various mouse sensors. If you dig around there you will find that they give you lots of info on the optics used as well as what wavelengths the sensors are most sensitive to, etc.

Also something to keep in mind is that even if you get to the point where the mouse can track movement accurately, it does not handle rotation. You'll still need a compass or something to know your heading which needs to be combined with the vector obtained from the mouse. I think you can substitute two mice on opposite sides of the robot instead of the gyro, but I haven't yet worked through the math to prove it to myself. There's some odd cases if you do a tank-style spin and such that I have to think about a little bit to see if it can still work.

Craig Putnam has indeed communicated with his mouse, but using an component that may or may not be legal. The maths are the only part I am sure of, and if I can interface two mice, I'll be golden. That little step is proving harder than origionally thought. I'll have to look into this aligent chip business...

Thanks for the info!

UPDATE: Looking at some PDFs on their site, it would seem the particular chip I'm looking at outputs PS/2 directly! Was this the case with yours?

[Craig Putnam]

I guess its time for me to get back into the conversation...

I'll begin by correcting one error in the thread so far. We are using the mouse to tell us how far we have moved, not the rotation (as it is pretty insensitive to rotations about its optical axis). We are using one of the old gyro chips to measure the rotation rate and then merging the inputs from the two sensors to give us a measure of the robot's motion since the last time we looked. All of that is going into a PID feedback loop (the details of which we are still being worked out). The intent is to enable the robot to accurately travel along any mathematically defined path: straight line, circular arc, spline curve, etc.

Using two mice (one on each side of the robot) is indeed another way around the rotation problem as is using one mouse at the center of the robot to measure distance traveled and a second mouse mounted at the edge of the frame to measure turning motion.

Re: the mouse not being able to track the robot's speed. By lifting the mouse board up and inserting the appropriate optics, we have effectively changed the size of the "Mickey". So instead of (for example) getting 200 Mickeys per inch, we are getting a much smaller number. While our resolution has gone down, the speed that we can track has gone up. We expect to eventually have a resolution of about 1/4 inch and be able to easily track the top speed of a FIRST robot.

We do indeed speak to the mouse quite well using the PAK-VIa chip. [ If you go to the Al Williams site you will see that there is now a chip specifically designed for communicating with mice - the PAK-XI. We starting with the PAK-VIa chip however and have found that works well enough for our needs at the moment. ] As has been pointed out however, the use of any PAK chip may well be illegal. So I am very interested in hearing from anyone who has successfully communicated directly with either a PS/2 or USB based mouse (or directly communicated with the Agilent chip as was noted above).

[Kevin Watson]

The optical mice are going to have a problem because one blurry patch of grey FIRST carpet is going to look like the next. I agree that this is a cool idea, but think it'll be hard to get it to work (but I'm willing to help nonetheless <grin>).

-Kevin

[sanddrag]

Thinking about this, yes optical mice are cool but what came before them? Ball mice! It may not be practical for all games (like climbing a step) but for a game like 2002's Zone Zeal, maybe you could just stick a ball and a couple wheels under the robot. Think Technokats 2003 Ball robot, but maybe 1/4-1/3 the size and instead of motors powering the little wheels there would be shaft encoders hooked to the little wheels. Yeah, that would work, at least on a flat field. Make a cradle in the center of your robot and put a ball in there with two little rollers at 90 degrees from each other contacting the ball at all times.

[Dave Flowerday]

Originally Posted by phrontist

UPDATE: Looking at some PDFs on their site, it would seem the particular chip I'm looking at outputs PS/2 directly! Was this the case with yours?

We've seen both types. The ones we mainly used contained a ADNS2610 chip which only does the sync serial. The other one we saw had 2 interfacecs, either PS/2 or good old quadrature output. If you have trouble with the PS/2 and don't want to try the sync serial to the Agilent chip, you could always use the quadrature output. That's really easy to decode.

[phrontist]

Originally Posted by sanddrag

Thinking about this, yes optical mice are cool but what came before them? Ball mice! It may not be practical for all games (like climbing a step) but for a game like 2002's Zone Zeal, maybe you could just stick a ball and a couple wheels under the robot. Think Technokats 2003 Ball robot, but maybe 1/4-1/3 the size and instead of motors powering the little wheels there would be shaft encoders hooked to the little wheels. Yeah, that would work, at least on a flat field. Make a cradle in the center of your robot and put a ball in there with two little rollers at 90 degrees from each other contacting the ball at all times.

Max Lobovsky said the exact same thing. But there is one problem with this! Try using a ball mouse with a carpet patch as your mouse pad for an hour and I garuntee your enthusiasm for this idea will be curbed.

[Craig Putnam]

Originally Posted by Kevin Watson

The optical mice are going to have a problem because one blurry patch of grey FIRST carpet is going to look like the next. I agree that this is a cool idea, but think it'll be hard to get it to work (but I'm willing to help nonetheless <grin>).

-Kevin

By zooming out (demagnifying) with the optics we are going to see more of the surface underneath the robot. This can cut both ways - more surface may give us the chance to see more variation. On the other hand, zooming out means we will see less detail in whatever it is that we do see.

My experience with testing various surfaces under my prototype optical system seemed to support that it will work OK. But I didn't have any of the FIRST carpet to test with so I can't say for sure that the modified mouse will work well with that. Time will tell...

The other thing that can really help is the angle of the light. Think about how an optical mouse works. The light is shining not directly down but at a very low angle to the surface. The idea is to cast as many shadows from surface variations as possible in order to give the chip something "interesting" to look at.

[phrontist]

Originally Posted by Dave Flowerday

We've seen both types. The ones we mainly used contained a ADNS2610 chip which only does the sync serial. The other one we saw had 2 interfacecs, either PS/2 or good old quadrature output. If you have trouble with the PS/2 and don't want to try the sync serial to the Agilent chip, you could always use the quadrature output. That's really easy to decode.

Quadrature! Sweeeeeeeeeeeeeeeeeeeeeeet!

That's the geekiest thing I've ever said.