OP: Let me try to explain a little bit what we are trying to get at and also a little about how the encoder works.
As far as I can tell, you have the S4 encoder attached to an idler sprocket in a gearbox, which is spinning at a constant 1500rpm (please correct me if any of this inaccurate). Then the two data wires are connected to DIOs 2 and 3 on your digital sidecar. Each of these wires carries a square wave, 90 degrees out of phase with each other (this part is what makes it a quadrature encoder), which the FPGA reads to keep track of distance traveled. In 4x mode, it watches for rising and falling edges on both channels A and B to determine which direction the encoder is spinning and then increment or decrement it's local counter (making it an incremental encoder). It also computes the period, the length of time between the last two of these "ticks." The average period is computed over a certain number of samples. If this average period is above a maximum, the FPGA treats the encoder as "stopped." If not, it takes this average period, divides your Distance/Count by it, and comes up with the Rate number. All of this computation is done in hardware pretty lightning fast inside the FPGA.
When you call the Encoder Get VI, you are asking the FPGA to tell you how many ticks it's counted and the current rate it has computed. When you initialize the encoder, you initialize all these parameters I described above, the 1x/2x/4x mode, the Max Period, the samples to average, etc.
Noise on the rate can come from a few places
1. The rate is actually, physically, that noisy. You can make sure this isn't the case by removing all load from your gearbox to ensure that the motors can get it up to a constant speed.
2. The encoder is spinning so fast that the FPGA is missing some of the ticks go by, so sometimes it's getting much longer periods than usual, so the calculated rate is inconsistent. The FPGA runs pretty dang fast: the last time I looked into this I think the encoder had to be running something upwards of 5 or 10,000 rpm to outrun the FPGA. But this could be your problem, and you can help the FGPA keep up by telling to only count 1 edge of 1 channel (1x mode). Knowing the CPR of the encoder helps you know just how fast it can run.
3. It's also possible that you are not average any samples at all, you are just always looking at the very last period calculated by the FPGA.
This really shouldn't be that inaccurate, but it could be. Edit: see Ether's post about this. You could make sure that you are averaging more like 10 samples to be sure of this.