I couldn't see in your picture, but I assume you switched over to speed mode.
What this sounds like to me, if everything is hooked up correctly, is that you did not set your P, I, and D constants. With them at the default 0, your motor won't turn no matter what speed you tell it to go.
Start by setting a P value of a few hundred (P is the proportion correction to the error between the desired speed and actual speed) and see if that gets your motor spinning.
Be careful, while tuning your PID constants the motor can sometimes move erratically. Leave I and D at zero for now.
Make sure you set the number of encoder lines (ticks per revolution) in the configuration tab.
The encoder/potentiometer radio buttons are grayed out in speed mode because speed mode only supports encoders at this time.
Whether this works or not, also be sure you're running the latest firmware on the Jaguars and the latest BDC-COMM (all version 87). They can be found here:
www.luminarymicro.com/jaguar
If you are still having trouble, post some more info about your setup, and a full screen shot of the BDC-COMM screen.
-David