The side loads are all transferred through the spokes, into the hub/axle. This can often cause cracks either in the hub itself or in the spokes. For example,
http://www.chiefdelphi.com/media/photos/31155 can happen. In 2007, with the first iteration of the AndyMark KoP wheels, we found that it was necessary to replace the wheels in our swerve modules at the rate of about 2/competition, due to either 1) cracking of the hub, or 2) cracking of the spokes. I recall that in one of the elimination matches at WMR that year, we took a solid hit from another robot, and a chunk of the wheel literally just broke off, so the wheel resembled a pac-man.
However, this problem (I believe) should become less significant as the wheels get smaller in diameter.