You can use it to talk directly to the RC without the TTL converter. See page 19 of the same manual. Just make sure you get the pinout right on the port.
As I understand the specifications on the camera, when using the TTL interface on the camera board itself, it’s speed is limited to 9600bps. When using the serial adapter your speed is 115k bps. As far as I can tell, that is the only difference.