Wildstang experimented with optical mice last year as well. I'm having a hard time telling from that long post just how far others have gotten. Has anyone actually communicated with their mouse?
Last year we were able to get an optical mouse connected to our robot controller and read X/Y positioning from the mouse chip. We did not use PS/2, however. Instead we removed the PS/2 driver chip from inside the mouse and connected the RC directly to the optical sensor chip from Agilent (BTW, Agilent is the only company making optical mice components, so if you open one up that is what you'll find). The Agilent chip can speak in a simple synchronous protocol which is how we communicated with it. We implemented this using the normal input/output pins on the RC and bit-banging the protocol. This allows us to read out the X/Y deltas as well as obtain the actual image captured by the mouse, the surface quality measurement, and a bunch of other goodies. The chip is pretty nice because it will remember how far it's moved since the last time you queried it, so you can query it at your leisure (as long as it's fast enough that the counter inside the sensor doesn't overflow).
We also affixed a different lens to the mouse to change its field of vision to accommodate larger speeds. Illumination is a problem, as was already mentioned. We fixed a ring of superbright LEDs around the lens to combat this. However, that is where the bad news starts. With all that out of the way, we mounted the modified mouse to a cart and started doing tests to see if it accurately tracked motion. We found that it did not. When the cart was moved at different speeds over the same distance, the mouse would report different amounts of measured distance. This was disappointing, because before we fitted the new lens we tested the mouse by affixing it to an X-Y table in our model shop with a very accurate readout and found that without the new lens it was good to something like a few thousandths of an inch. At this point it was getting pretty late in the season so with the mouse concept not looking too promising we had to abandon it and concentrate on reusing our old positioning system that we used in 2003.
Agilent's site is pretty useful, with datasheets for the various mouse sensors. If you dig around there you will find that they give you lots of info on the optics used as well as what wavelengths the sensors are most sensitive to, etc.
Also something to keep in mind is that even if you get to the point where the mouse can track movement accurately, it does not handle rotation. You'll still need a compass or something to know your heading which needs to be combined with the vector obtained from the mouse. I think you can substitute two mice on opposite sides of the robot instead of the gyro, but I haven't yet worked through the math to prove it to myself. There's some odd cases if you do a tank-style spin and such that I have to think about a little bit to see if it can still work.