Normally I'd say hot-melt glue, but 50 of them?
If an ice-cube tray size is not too large, then you can use any of several substances, first choice in my mind would be liquid
epoxy potting compound, just pour and wait. The downside is that it's difficult to remove if repairs are ever needed.
To have any substance release easily from an ice-cube mold, use
silicone spray liberally. This is how it is done in industry. An alternative would be to get a
flexible silicone ice-cube tray so it can be peeled off instead (but still use the release spray!)
Silicone "caulk" is not an ideal substance. As part of the curing process it releases acetic acid (vinegar), which attacks electrical connections. In electronics, it is used only as a mechanical connector to hold heavy components in place on a PC Board, and should never come in contact with metal connections. You CAN buy
silicone potting compound which uses a different chemistry and is safe for electronics, but it isn't cheap.
If you do use epoxy, consider a custom mold that creates mounting ears for your switch, or some other mechanical features that will improve the device. Molds can be made from hard plastic (polyethelyene), metal, or even machinable wax (Google it, I use
Freeman Supply for mine).
Good luck!