[ozrien]

You're sensor is not single-direction (not a tachometer). So you want to select Quad-encoder. That's part of the problem.

So you're initial closedloopErr is +270 ( target - currentPos), such that currentPos is about '0'. 270 times a pgain of '1' will yield '270'. That's about 26% positive throttle (1023 is full, see Duty-Cycle Section 17.5 in software reference manual). So I think that's ok.

Also I see that you reversed the sensor. I'm guessing you followed the procedure in section 7.4 (or similar) to ensure that positive throttle makes "Sensor Position" move in the positive direction.

Just curious how did you hookup the encoder to the motor mechanically. I mean what's the gear ratio between the two?