Ok, so I finally got a weird solution to the problem
Turns out that for every digital input we initialized (the trackers), we had to create a victor in the corresponding port. Meaning
Code:
DigitalInput tracker = new DigitalInput(1);
Victor motor= new Victor(1);
For some reason, initializing this victor makes the photoswitches works correctly