Generally, we use a USB hub, and the joysticks don't get unplugged from it, so our controllers are consistently wired to the same port all the time, unless we swap them around or use a different cRIO (even if it's the same code)
Anyway, it's pretty easy to make sure your joysticks are read in the port that you want them to. Just go to the setup tab of your Driver Station...
Then just click and drag to reorder your joysticks around.
If you need to find our which joystick to put in a port, when you press a button the joystick text will turn blue (as shown above).
So if you're constantly removing your joysticks, I would suggest making sure they're setup right before enabling each time, just to be safe.