Amount of coupler purchase required on 1/2'' hex?

[Chris is me]

Assuming you can keep the gearbox and shaft concentric and that the wheel shaft is supported on both ends, I think you shouldn't have any weird forces on the coupler to worry about.

The biggest challenge would be making sure everything is lined up and concentric.

I'd have to do some math to see how the hex engagement would work out, but basically you want to engage as much of the wheel shaft as practical (at least 1/2"). A 1" coupler could fit in a 2x1 if you either omit a bearing (ehhh) or if you use a large ID bearing that goes around the coupler (better).

This is a bit of a tangent, but looking at your specific design, maybe you could build a gearbox that straddles your rectangular tube? Instead of driving a live shaft, have a gear off the gearbox drive another gear bolted to the wheel. I feel like this might achieve what you're trying to do (shifting gearbox with CIMs over drive wheels) a bit better than the coupler solution.