Ghost in the machine

[phurley67]

At the Week 1 Standish-Sterling event we (862) had a very disturbing experience. Our robot while disabled and on the field decided to move. It did this 3 times after power cycles before we were disqualified for the match. We have now seen this behavior with 3 different motors including both PWM and CAN motor controllers -- so we are making the assumption that the issue lies with the RIO.

The problem seemed limited to immediately after a power cycle or when attaching either a USB or ethernet cable to to the RIO.

Given time constraints and an already rough start to our first outing we decided to try reformatting the RIO, which appears to have resolved the problem (we have not seen it since); however, we are a little nervous because this does not seem like the type problem that just reformatting should affect.

We have not done any low level coding or modification of the FPGA, WPIlib, etc. The code that was on the robot is labview code, that was put back on the robot after the reformat.

Wondering if anyone else had ever seen anything similar.

p.s. going into lunch after a very disappointing qualification outing, we were resigned to doing the swap, until a very gusty scouting job by our wonderful alliance partners 5488 and 6098 selected us (2nd from last in quals) and we managed to pull everything together and finish as finalists. Congratulations to 1918, 4377, 5166 look forward to playing with all of you later in the season.

[Greg McKaskle]

I don't think the reformat had any effect.

The times I've seen a robot move while disabled, it has been due to a PWM cable that would periodically short, resulting in a very short PWM glitch. I've almost always seen it with Y cables. If you carefully mash on the Y of the cable, you will see the occasional burst from the controller. Rolling it in between the thumb and finger was the most reliable way to make it happen.

I have also seen it in a robot that was shorting power and causing the motor controllers to reboot. On power up, the controllers would glitch and reboot again, rinse, repeat.

I have never seen this with a CAN controller. I'd like to hear details.

Reformatting the RIO is equivalent to formatting the hard drive on a linux computer. It is necessary when you have a corrupted file, which shouldn't really happen, and it is necessary at the beginning of the season to install a new set of files.

Greg McKaskle

[engunneer]

One option with the (talon srx) can controllers is that they will actually respond to pm signals on the can wires. So if it was some kind of power glitch that the controller thought was a pwm pulse, it could maybe twitch the motors.

[phurley67]

I agree the reformat was unlikely to have any real affect; however as we saw it multiple motors (connected to different motor controllers, including some controlled via PWM and others by CAN, it seems very unlikely we have that many defective motor controllers or that much bad wiring. And despite our team name, the robot was never struck by lightning.

Given all of that, the only option I can really buy is something defective with the RIO, but with the problem no longer happening (we ran 9 matches without seeing it again after the reformat), we are reluctant to spend all of our next unbag time rewiring our robot.

[Ryan_Todd]

Originally Posted by engunneer

One option with the (talon srx) can controllers is that they will actually respond to pm signals on the can wires. So if it was some kind of power glitch that the controller thought was a pwm pulse, it could maybe twitch the motors.

I raised the possibility of a power glitch as a potential explanation for the PWM controllers' misbehavior, but I couldn't think of any way that the CAN controllers could be made to behave in the same way, so we haven't yet pursued that possibility very far.

Come to think of it, however, I believe that we are indeed running Talon SRX's for the CAN-controlled motors... After checking out the user guide for these controllers (link), I'm thinking that we should indeed reopen that branch of the investigation!

[adciv]

Check your motor controllers as well. I've come across instances where the controller fails on (Victor 88X). Mind you, it sounds like a different issue here.

[Ryan Caldwell]

On a more supernatural level burning sage is suppose to clear evil spirits...

[GreyingJay]

Just wanted to chime in and say that the team I was on last year saw ghost glitches and they too were eventually attributed to loose PWM wiring. Once we tightened all that up, everything was fine.

[hardcopi]

Last night we had a similar issue. Our robot during unbag time was powered on and the light started clicking and blinking as though it were in teleop. Turned out a metal shaving had fallen into one of the controllers.