[Alan Anderson]

We fixed our highly inconsistent encoder readings in software, by interrupting on both rising and falling edges of the Phase A signal. The apparent root cause of the problem was mechanical jitter, tricking the software into counting multiple times as the encoder stayed roughly in the same place. An electronic solution (search for "7474") didn't fix our problem, but it might for you.