You could post your code so others could help you troubleshoot, but first I suggest:
- Use the console or dashboard to display the value of the stick, and "let go" a few times. Take note of the range of values it can be and make that your deadzone (the range in which the input shouldn't cause the motors to move).
- Make sure that when you are in the deadzone, that you are sending a command of 0.0 ("stop"), not just "do nothing". Sending no command would cause the motor to move at the last known value (this one sounds like your issue).
- Your joystick may just be loose?