Thanks for all the replies. In the end I tracked it down to the Victor SP calibration. I noticed when moving forward/reverse that the red/green colors came on at different times. The VictorSP manual says that they are calibrated for the RoboRio by default, so could be there is still something I have to do in the RoboRio, but for now running through the calibration steps in the Victor SP manual fixes the issue.
For anyone curious, or running into this later here is the current manual:
http://content.vexrobotics.com/vexpr...e-20151209.pdf
Of course I unplugged the motors while running the calibration step.