Hi Steve Fang. This is a great first post!
The RS775 motors are not very friendly to use in custom applications because they have a very small, short, and round shaft. If you would like to use these motors, they do come with prefabricated planetary gearboxes like
this. Another good and similar option for a winch would be the
Vex Pro Versa-planetary gearboxes. These can be configured to fit the various small motors, but their main advantage is that you can attach the powerful CIM motor. They are available in reductions from 3:1 to 100:1. The reason I would choose the Vex Pro option is because they have a hex output shaft that allows for easy mounting of a spool. As a young team this would be a much easier, faster, and more reliable way to obtain a winch. Custom gearboxes are great for making compact and fine tuned drivetrains, but offer little advantage when designing a winch. Oh and one note about attaching a spool to one of these gearboxes: you must support the end of the shaft in a bearing to avoid forces that could cause the gearbox to lock up or become damaged.
Hope this helps!