Worm gears work in both directions. For example, the Window motor transmission uses a worm gear. However, they tend not to backdrive very easily, or at all, depending on their design and efficiency. The biggest problem with a worm gear drivetrain would be the relatively low efficiency. Due to the way the worm gear interacts with the worm wheel, the design is inherently less efficient than a standard spur gearset. For a drivetrain this is particularly important.
A more efficient solution (but still less efficient than regular spur gears) would be to use bevel gears. These come with their own problems with alignment and thrust loading, but they aren't unsurmountable.
Here's an example by team 192.
Generally I think efficiency is much more important in a drivetrain gearbox than packaging, so I'd use a regular spur gearbox in a drivetrain most of the time. Depending on your drive layout you may be able to arrange the transmission with the CIMs over the wheels to save space.