Thanks for the suggestions everyone. Unfortunately, nothing we tried solved the problem. The encoder still inconsistently oscillates between 0-n*0.25 where n is the number of ticks/rev set in the code.
Here's what we tried:
- Resoldering the connections to eliminate any possibility of faulty wiring
- Setting the ticks/revolution in the code among other parameters specified in WPI Screensteps
- Switching the wiring to see if the oscillation jumped pins (it did)
- Reading the raw output of the encoder
Sadly, nothing we tried got us usable values from the encoder. I guess this means our encoder itself is busted. My advice for anyone reading this thread in the future is to be extra careful when assembling the optical encoders and avoid touching the sensor as much as possible. Thanks again for the help!