[Cory]

My only concern with this would be that it might be harder to keep the broach aligned with the bore since you no longer are relying on the fit between the .500 dia shank on the broach and the hole in the part to ensure initial alignment.

Otherwise this is a perfectly good way to broach something. You'll see that any parts that have been rotary broached will end up with a rounded center and only have flats on the corners, like Andy drew above, since the broach requires some additional clearance over nominal.

At some point the best solution is just to drop $500 on a new arbor press that is actually large enough so that you can do this easily and more repeatably, if you're going to be broaching a lot of parts.