fine control of robot with mecanum wheels

[StevenB]

This may be something you've already tried, but you might want to consider a nonlinear joystick mapping. That is, when you push the joystick halfway, you only move at 1/4 speed instead of 1/2. This gives you finer control of slower movements, while allowing you to still use the full speed range. Very rarely do you care if you move at 3/4 speed vs full speed, but you definitely care about 0 vs 1/4 - thus, you can stretch the axes accordingly.
All you have to do is square the input, making sure to preserve the sign.

A second note - if this is a first or second year team, my suggestion is, "get used to it." Fast robots are almost always hard to control, and the best fix is simply driver practice. I remember during my rookie year feeling that our robot was uncontrollable, but we eventually learned to drive it, and it wasn't as bad as we thought.