This was our original thought.
After some testing we noticed that there was a weird spin on the ball and concluded that it was because of the larger wheels spinning faster than the smaller wheels. To solve this we settled on using the 2.875" wheels all the way across and it works much better.