The motor controller is behaving erratically because the PWM frequency
and the duty cycle are wrong. The analogWrite() function is generating this PWM signal:

(from Arduino.cc)
...but your motor controller is expecting this:
The
frequency of the signal in Hz is the number of pulses per second. The reciprocal of the frequency is the
period, which is the time between the rising edges of the PWM signal. For your motor controller, this should be 50 Hz, or a 20 ms (millisecond) period. According to the Arduino documentation, the analogWrite() function produces a 490 Hz PWM signal, or a 2 ms period.
The
duty cycle of the signal is the ratio between the pulse width and the period. For your motor controller, the pulse width is "mapped" as follows:
Full Forward: 2 ms pulse / 20 ms period = 10% duty cycle.
Neutral: 1.5 ms pulse / 20 ms period = 7.5% duty cycle.
Full Reverse: 1 ms pulse width / 20 ms period = 5% duty cycle.
As others have suggested, the Arduino servo library should be able to produce the correct signal. 180 degrees on a servo is the equivalent of "full forward" on a motor attached to the motor controller (0 degrees is full reverse, 90 is neutral, etc.)
The motor controller does just this - it generates a 0-100% duty cycle PWM signal to the motor, using a circuit called an "H-bridge" to reverse the voltage when changing directions.
Hope this helps...