Cold-finished steel probably would have a better size tolerance. Your "wobble" is compounded not only by bearings, but the fit in the wheel hub. Vex aluminum shafting has a really good fit, and we used them on a cantilevered drive shaft with no bending all season, and are still running that robot for public demos.
We also ran the shafts through our frame and used a bearing there to relieve some of the load on the outboard tranny bearing, reducing the cantilevered shaft distance. Hope that helps.