E-Stop Lag... now part of our arm is broken

So we had a crazy problem last night. We were trying to get PID running with our arm using presets. Everything was going fine until we changed one little thing… motor direction.

Now our logic is this… and sorry for no LabVIEW code.

Preset–>PID–>Negate Motor–>Arm Limits–>Arm Motor

Very straight and simple logic. The Arm Limits is based on motor direction, and our absolute analog encoder value. The limits were working fine. Reversing the direction that the PID responded with shouldn’t have affected anything and we still don’t think it did.

We programmed it with the negate on the motor output value of the PID to correct the response. It was going completely reversed, but the arm limits were working perfectly.

Well, we deployed the new code and it immediately ran the motor full forward until the arm hit the physical stop and kept going. We saw it about to happen (in like a second in advance), so the programmer hit the E-stop button, but it didn’t work. The arm snapped the physical stop, and kept driving down. We had to stop it by hitting the reset breaker on the back of the robot that was to remain clear because of the said arm.

So a few issues here… the robot ignored the coded and tested limits on the arm. The robot ignored the E-stop button. And one of the craziest things is that the arm wasn’t even in preset mode. It was in joystick mode. (We have 3 modes: joystick, throttle, and preset… when the robot is first run, it defaults to joystick mode). We verified it was in joystick mode because that was what was set on the front panel. Another issue is that we know it knew it’s position. It wasn’t supposed to go below 90 degrees, but when it lost the signal, it was at 59.6 degrees.

So, programmable limit was ignored, e-stop was ignored, joystick mode was ignored…

I can’t even imagine what happened. All the programmers sat down for a while and tried to put the pieces of the puzzle together, but we still have no idea what was wrong.

Even weirder is that we re-deployed the same code, except with the arm breaker removed and a probe on the motor output, and it worked flawlessly. Flawlessly enough to plug the breaker back in and try it. Worked fine. No code change at all.

Some extra background. We were definitely in Teleop mode. The arm is only called in teleop, and autonomous.

Seeing that the problem is corrected for now… I don’t know if anyone can tell us how to fix it. Is there a known issue for the robot just doing random stuff?
What was up with the E-stop not working at all?

All we can think is that we lost communication for a split second, which caused the motors to retain their values and ignore all other code until the watchdog would timeout… which I guess it never had a chance to do.
:confused:

Can you post your code? There might be something that you guys just overlooked (though the E-Stop issue stumps me).
In lieu of that, I would check the following, though some of this may be irrelevant:
-How are you passing data from Periodic Tasks to Teleop? Is there a chance that one of these Globals is being written in 2 spots at the same time?
-With the code that you deployed, did you make the controls you changed “set as default”?
-How did you change your motor input based on the Arm limits? We implemented it by reading limit switches (though it sounds like you are using the actual sensor itself) and changing the maximum bounds of the PID.vi We never tested this, and I believe that it should work but if you are doing it the same way I would look at that too.
-I referenced this before (http://www.chiefdelphi.com/forums/showpost.php?p=1023378&postcount=6) but is it possible that your encoder wasn’t working correctly and was not reading values consistently?
-With the E-Stop issue, did it actually say on the DS “Emergency Stop” (or whatever the text is? Did you guys ever disable the E-Stop button?

We’re using C++, but if there is a problem at the FPGA level, it would affect any language.

We’ve had a few instances where the robot stops responding and latches whatever the last used speed controller outputs were. In all cases, we’ve used the space bar to disable the robot, so I don’t know what the behavior of the e-stop would have been.

A couple times we’ve gotten crashes when we read the encoders and the WPILib encoder class tries to access the counter. When that happens and we’re connected to the robot, we can see the WindRiver error box pop up and point us to that location.

Last night we added a few prints to debug an issue and got some serious lag in our driver control. This was seen by runaway movement of the motor and lack of sensor updates coming back to the dashboard. We don’t have any hard evidence of what the problem was, but when we removed the prints things worked fine. We didn’t have time to dig in and narrow things down.

So, this obviously doesn’t solve your problem, but I just wanted to put this out there so that people can keep an eye out for any similar situations.

Ryan,

You absolutely cannot , cannot just reverse the output to the motor to reverse the motor output if your pid is working correctly.

Reversing the motor output will destabalize your control loop by creating positive feedback. see this which can cause catastrophic failures in critical systems. Nuclear explosion is an example of positive feedback.

To reverse the direction, you have to either
1.negate the output to the motor and negate the input from the encoder/potentiometer.
or 2. negate the preset line.

Also I have never heard of an analog encoder. can you explain what you mean?

We never disabled the E-Stop button on the dashboard. There was a 2-3 second delay between the button being pressed and the robot cutting out.

And here is the specs for the Analog Encoder
http://usdigital.com/products/ma3/

I know that the arm damage has probably put you behind schedule, but please try to check things out to see what does and doesn’t work and to verify you can trust the safety systems.

  1. E-Stop the robot. The buttons aren’t the most robust things, and if yours is broken, toss it. As mentioned, hitting spacebar disables the robot and is good to teach in addition or instead.

  2. Break communications with the robot by pulling the enet cable on the cRIO. The outputs should stop due to the system watchdog.

  3. Review code looking for logic issues that could lead to a wild value being computed.

  4. Examine the digital breakout. In the very rare cases I’ve heard of where a robot moved even when disabled, it was due to a fused digital breakout board that didn’t disable digital lines properly. I don’t remember the exact issue, but it was a damaged board. That doesn’t make sense with a PWM signal, but verify the board looks good anyway.

Please let us know if you see anything else that looks flaky, anything else that needs a close inspection or additional testing.

Greg McKaskle

We had an incident this season where the fuse to the DSC was missing and the system was DISABLED, but one victor was enabled. It looked like the DSC was being back powered to some extent. The 5V LED was on, the BAT and 6V LEDs were off. After the fuse was installed, everything worked correctly.

I found a few issues in the old code. Here’s the new code if anyone wants to try having a look at it. I don’t know if the question marks for the Joysticks, Motors, and PID will go away if someone with FRC LabVIEW opens it or not. I have plain ole’ LabVIEW on my desktop computer at work, so none of those show up.

Here it is: Arm Control.vi (30.1 KB)

Arm Control.vi (30.1 KB)


Arm Control.vi (30.1 KB)

We had something similar happen to us at a fall off-season event. The spike relays (nothing else) were firing while disabled and the RSL was never turning on. Our theory at the time was that the signal was getting crossed inside the digital side car. Replacing the side car got us running again but it was very odd and unsafe that our spikes started turning on randomly.

As far as E-Stops not working how they should, we had a scary occurrence today at the WI regional. During one of the qualifying matches, i tried to move the telescoping part of our arm using position mode. The encoder wasn’t plugged in correctly, so it kept on going outward. About halfway out, i slammed on the E-stop button, but the arm kept going. Even after hitting it numerous times, the arm never stopped until it ran itself off of the ball screw

What we try to do when first testing the functionality of limit switches is to run in manual mode in the middle of the operating range at low speed and then manually trigger the appropriate limit switch to see if it stops the motion and allows operation in the reverse direction. This simple test allows for checking the sense of the logic, the wiring of the motors, the wiring of the limit switches and the logic of the control code.

Even after hitting it numerous times, …

Was this on the field or in the pits? The small USB button is not active if the DS is plugged into the FMS. This is also true if the DS is plugged into the dlink and the robot is using the field radio.

In other words, if you are using the DS to lower an arm, but the FMS connected message shows up, you have no estop button. It is expecting you to use the red push-twist button at the station.

Can you give more info on what happened?

Greg McKaskle

Sorry if I was not clear. When i said E-Stop, I meant the red twist button (this was during a real match). However, this turned out to be a minor problem compared to the rest of the competition. In about half of our matches we played in, we had connectivity problems or the watchdog erroring out. One factor we discovered was that switching to a different laptop (instead of using the classmate) helped alot, but did not completely solve our problems. The other issues probably have something to do with our CAN configuration and how the programming is set up.