Kevin Waston provides some modular code for setting up encoders on the IFI hardware. The EDU hardware is closest to Vex, so I'd download that workspace. Any of the code can be easily ported to work with Vex, so I guess it doesnt matter.
That code will work as long as rc_dig_int1 is defined as the interrupt flag you want to look at and it is placed in the interrupt handler found in user_routines_fast.c. I'm guessing you are looking directly at the input pin, which is causing the code inside the If to execute every time.