[DonRotolo]

Clamp that part of the workpiece better.

I think what you are saying is that the waste part of the hole you're making starts jumping around, right? If so, make the cut in more than one pass - almost all the way through to start, then using only a very light cut for the finish. Or exclude the middle of the sides at first, then take them last. Or clamp the middle of the waste rectangle (drill a hole and bolt thru to the work table)

And yes, slowing down the cutting speed (not the tool speed!) may help.