Disclaimer: I am not an FRC programmer, but I have dealt with these kinds of issues before…
As for the “jerkiness” of the robot, first, are the speed controllers in brake mode? That may be your problem right there. Beyond that, it’s probably, at least partially, because with voltage compensation, rather than scaling your control between 0% and 100% of available voltage, you’re scaling between 0% and 100% of maximum possible voltage, regardless of what the actual available voltage is, meaning if your available voltage is less than max, your controls become truncated beyond that point. This means that whereas before you would always have 0%-100% control range on your joysticks regardless of voltage, now if your battery is at 80% capacity, you only have 0-80% control range on your joystick, and the other 20% is just dead-band.
Technically speaking, with some practice, this would allow your drivers to have more consistent control at a given voltage, but it can be tricky to control, especially depending on the type of controllers you’re using. A proper PC Joystick, for instance, is going to have more mechanical resolution than an Xbox controller (simply because of the difference in size), so reducing that resolution on an Xbox controller even further by truncating the top-end of your control range can cause issues. To that end, running Windows joystick calibration might not be a bad idea.
In addition to what others have talked about, another option that might help would be to create a “minimum” voltage limit that your controller outputs (before skipping to “0v” at idle), since most drive systems don’t operate below a certain voltage, having that voltage as part of your control range is effectively a waste of your controllers resolution, since you’re not actually moving. So for example instead of 0-1 input giving 0v-12v of range, it would give you 0v, 3.5v-12v of range (the 0v being the idle position).
Just a thought, the actual programmers here can tell you if I’m totally off-base. It has been a while since I’ve dealt with this, so it might be totally different now.