I tightened the encoder discs on the axle as shown in the video, and I made sure there wasn't any slipping. When I rotated the encoders by hand, the Distance count was kind of consistent: I'd say about +-10 counts.
I tested an IIR filter of the encoder speed and compared it to a moving average. You can see an example of my results in
this video. The problem with the PID loop adjusting the speed is still present. I'm still not sure if there is a good filter or not.