[Mr. Lim]

I know this will only give you 1/4 the resolution you want, but does at least this work?

	const unsigned char aMask = 0x10;
        const unsigned char bMask = 0x20;

	unsigned char a = (PORTB & aMask) == aMask;
        unsigned char b = (PORTB & bMask) == bMask;


	if((PORTB ^ oldPORTB) & aMask == aMask)
	{
		if(a)						// a transitioned high
		{
			if(b)					// b is high
				++encoderCount;
			else					// b is low
				--encoderCount;
		}
	}

	oldPORTB = PORTB;

	INTCONbits.RBIF = 0;     /*     and clear the interrupt flag         */

If this does work, then chances are it's an ISR timing problem. If this doesn't work, then it's likely a logic issue that's not obvious to us. Either way, it'd help us narrow down the nature of the beast.

-Shawn T. Lim...