E-Stop Lag... now part of our arm is broken

[RyanN]

So we had a crazy problem last night. We were trying to get PID running with our arm using presets. Everything was going fine until we changed one little thing... motor direction.

Now our logic is this... and sorry for no LabVIEW code.

Preset-->PID-->Negate Motor-->Arm Limits-->Arm Motor

Very straight and simple logic. The Arm Limits is based on motor direction, and our absolute analog encoder value. The limits were working fine. Reversing the direction that the PID responded with shouldn't have affected anything and we still don't think it did.

We programmed it with the negate on the motor output value of the PID to correct the response. It was going completely reversed, but the arm limits were working perfectly.

Well, we deployed the new code and it immediately ran the motor full forward until the arm hit the physical stop and kept going. We saw it about to happen (in like a second in advance), so the programmer hit the E-stop button, but it didn't work. The arm snapped the physical stop, and kept driving down. We had to stop it by hitting the reset breaker on the back of the robot that was to remain clear because of the said arm.

So a few issues here... the robot ignored the coded and tested limits on the arm. The robot ignored the E-stop button. And one of the craziest things is that the arm wasn't even in preset mode. It was in joystick mode. (We have 3 modes: joystick, throttle, and preset... when the robot is first run, it defaults to joystick mode). We verified it was in joystick mode because that was what was set on the front panel. Another issue is that we know it knew it's position. It wasn't supposed to go below 90 degrees, but when it lost the signal, it was at 59.6 degrees.

So, programmable limit was ignored, e-stop was ignored, joystick mode was ignored...

I can't even imagine what happened. All the programmers sat down for a while and tried to put the pieces of the puzzle together, but we still have no idea what was wrong.

Even weirder is that we re-deployed the same code, except with the arm breaker removed and a probe on the motor output, and it worked flawlessly. Flawlessly enough to plug the breaker back in and try it. Worked fine. No code change at all.

Some extra background. We were definitely in Teleop mode. The arm is only called in teleop, and autonomous.

Seeing that the problem is corrected for now... I don't know if anyone can tell us how to fix it. Is there a known issue for the robot just doing random stuff?
What was up with the E-stop not working at all?

All we can think is that we lost communication for a split second, which caused the motors to retain their values and ignore all other code until the watchdog would timeout... which I guess it never had a chance to do.