MK4 Swerve module offsets not consistent

We’re trying to get viable offsets for our swerve but the values are changing the steer angle in unexpected ways (i.e. <5 degree changes are basically flipping the whole wheel). We’re using SDS’s SwerveLib (which we have run before on the same swerve modules without any problems). Why could this issue be happening?

Is your magnet for the encoder glued in?

Yes it is. I’m 99% sure everything is okay mechanically with the swerve modules.