[BL0X3R]

I have a few questions about your system so that we might be able to help you better, if you don't mind me asking.

What language are you using?

Are you controlling speed or position with your PID system? Getting the encoder speed has been known to not work entirely properly all the time, having large jumps in value, because of how the roboRIO does timing loops. Taking a manual derivative [(oldSpeed - currentSpeed) / periodicInterval (20 ms, unless you are making your own loops)] works much better and smoother.

Is there any other system controlling the motor? I know that in labVIEW, the same motor set to different values in different places will cause the motor to rapidly twitch between them, due to the parallel nature of the language. Our team hasn't run into this problem in any of the text based options, but we also haven't had as much experience with them, and have very little experience in command-based programming (we've programmed all our new robots iteratively). If you are doing any threading or parallel processes, then definitely search for race conditions.

Now, to answer your questions:

"Is using an encoder not a good idea for PID?"
Encoders work great - they are the best (only?) method of getting the position and velocity of a robot. We've used them ourselves in 2014 on our drive train and had great success with them. If you know what you're doing, you can do some really cool things with them (note: this isn't completely PID, but it is motion profiling with encoders as feedback sensors)

"Is there a way to set the max output for PID?"
the PIDSubsystem and PIDController classes both seem to have "setOutputRange" methods. I haven't used them myself, but the documentation says "Sets the maximum and minimum values to write", so I would start there.

"Maybe its resolution is too coarse?"
Most teams use encoders with resolutions of 360 or 256 and have reasonable success with both, so it's not likely that the resolution is where the problem lies.

"Maybe we should try setting the units per pulse to .1 and set point to 3600?"
If you are controlling the position of the encoders, then that would work well for finding any offending behaviors in the PID controller (obviously it's not a permanent solution).

Best of luck, it's good to see mentors helping students learn and explore more advanced control options in FRC