I'm not familiar with the particular options in robot builder. But if you are wiring up channels A and B to the digital side car, then you're using the encoder as a quadrature encoder. IF there's an option that says quadrature, pick that one.
For info on what quadrature is, check out these links:
http://www.dynapar.com/Technology/En...ture_Encoders/
http://www.eehomepage.com/report.php?report=20080225
Otherwise if you could provide a list of named options, or a screenshot, I could help direct you better.