You need to have the wires with the contacts in the connector for them to snap together. Without the wires with the contacts, they will slip in and out.
Check out the 4th picture down on
this link showing the cutaway view of a connector. The metal bump shown at the end of the connector snaps over the other metal bump on the mating connector. That's what holds them together. Without those, they will slip in and out freely.