Use omni wheels if you possibly can. If you must use casters, use ball casters.
While weight and the fact you aren’t on carpet may have something to do with your problem (and if I were you, I’d test on carpet), casters are your problem right now.
Casters tend to do exactly what you’re finding out that they do: mess up maneuverability.
We’ll probably use an FTC stunt, by using a 3 unit channel attached to the bottom of our C-Channel to displace the 8 inch wheel, therefore leveling it with the pneumatic
Easy solution to omnis, you will need the following:
A kit wheel of your choice.
Pool tube.
Zipties.
Take the pool tube cut to go around the outside of the wheel. Cut down the middle and put over the outside of the tube using zipties every so often to hold the tube in place. Cut the ends off of the zipties and tada! You have a poor man’s omni wheel.
My old team used these in 2005, 2006, and 2008 with much success. Good luck!