Well make sure the rollers on the wheels form an "X" in the center(Front right and Back left rollers should make an invisible line with each other and FL/BR should also line up. Ex:
http://robosavvy.com/store/images/ne.../10009_03.jpg). We had some really weird things happen, that sounds similar to what you have, when one wheel was put on backwards.
Hope this helps!