I would say go for it in the design. I personally think that your turning gear placement should be fine. From the picture it seems like it could be weak but this is a module, and not an entire frame so I can't say that when implemented in a frame that it wouldn't work. I really like the worm gear idea very interesting. I try to avoid using a banebot on a drivetrain because I really like their functionality for other areas of the bot. But that is all preference really.
I know what you are saying, but when the thing is built you could have a lot of people frustrated with the design, say they want it changed, and you are forced to go back in and fix something. I would look into timing belts (a lot of people don't like them) but in a swerve drive they can be very effective.
http://sdp-si.com/ has a really cool belt length calculator to make designing easier. Also are you using CREO? You can also get into equations and make the design change according to what reduction you prefer on the belts.
Just food for thought looks very sharp.