Yes, if you mount the camera to your turret rigidly, you'll get a lot of motion on the turret when you're searching for a target, which may not be desirable (although there certainly are designs that can make this work).
It sounds from your description that you're mounting the camera on the turret, then tracking with the camera while turning the turret until the camera servo is zero'd. As i think you've found out, controlling rotating pieces attached to other rotating pieces can be a bit tricky
Instead, try moving your camera so it's not on the turret. Now it can still rotate 180 degrees about your robot, and you'll know the exact angle to another robot. from there, throw a rotary encoder on your turret and use feedback from that to turn the turret angle to match.