RC: If you'd like a different version of this design that takes a bit less machine time, hit me up and I'll have something for you...

To the OP: I'm guessing that the hex you're using is from the McMaster standoffs? To make it easier to machine, you might want to drill the perpendicular hole onto the flat of a hex. Getting a drill to not walk, and to bite right while on a point is a little bit difficult.

Keep working with this design, I bet you'll land at the same place that Madison and I ended up after a few iterations!