[EricH]

Your problem can be summed up in one word.

Casters.

Use omni wheels if you possibly can. If you must use casters, use ball casters.

While weight and the fact you aren't on carpet may have something to do with your problem (and if I were you, I'd test on carpet), casters are your problem right now.

Casters tend to do exactly what you're finding out that they do: mess up maneuverability.