We had a similar issue during OKC last year, where drivers would lose control of the robot. For us, the issue had nothing to do with vision. Instead, occasionally our controller inputs would “freeze” to the last inputs and hold them there. The only way for us to fix this was to un-plug and re-plug in our controller.
This is similar to your case in that this issue would persist independent of code, but removing vision wouldn’t fix the issue.