First question. Are you using
omni wheels or
mecanum wheels?
The answer will change depending. If you are using omni wheels, they should be aligned at four corners like
so. If not, you will need to use mecanums or move them to that orientation. Mecanums need to be on the correct side, as there are left mecanums and right mecanums.
After that, is this an FRC robot or something else? If you are, there's a nice method in WPILib called
mecanumDrive_Cartesian that will resolve the motcrs from joystick axes for you. If you are not using WPILib, then you can still look at that code to find the math.