[Brian C]

Electric motors are designed to operate with more effective power in one direction than another. This is done through the timing of where the armature and commutator in relation to the field magnets.

It sounds as if your initial button push is spinning the motor in "reverse" and your second movement is the motors "forward" rotation. If you're trying to stop an arm at a specific point then limit switches or a potentiometer would really be the best option.