[Travis Covington]

You need to accommodate manufacturing tolerances in all of the parts involved (both extruded tubing, as well as any parts you make yourself). As such, lengthening the shaft some to accommodate tubing width variations, as well as bearing housing manufacturing tolerances is necessary and not really a big deal. This is pretty common. If you want to get really picky, you can add precision shims later to make up for the difference, but again, this is rarely ever worth the effort. Making things "line to line" never works well in the real world as you have already figured out.