Motor(s) randomly firing in Disabled

[scca229]

We have a strange issue that has been happening since first powering up the bot this year and hasn't gone away through firmware/software updates. I don't know if this is electrical, programming, or hardware....hence choosing this forum.

At random times, couple times a minute or even 10+ minutes apart, we have at least one motor that will activate for anywhere from a split-second blip to up to 2 seconds when the bot is in Disabled mode. Sometimes it happens right when turning the bot on, ie., the RoboRio hasn't even loaded yet. This is the pair of controllers running one side of the drivetrain.

Even stranger is that it doesn't seem to happen at all when the bot is in Enabled mode. I have to attempt to confirm that, but I don't believe it happened after 20+ minutes in Enabled last night after going off many times in the preceding half hour timeframe in Disabled. Since it is so random, trying to see what the LEDs are doing on the suspected pair of controllers is problematic (thinking of setting my phone recording video to try to catch it) but it "looks" like only one of the controllers seems to be doing it.

We are using Victor 888's attached to the RoboRio with Y-PWM cables controlling CIMs. Visual inspection of the connections show no stray copper crossing. All controllers have been calibrated multiple times. Programming done in LabView which I am very green in but the condition remains with the basic zero change tutorial loads as well. RoboRio has also been reformatted a few times with no change.

One more piece of info that may be pertinent, when the bot is turned on (or moved from Enabled to Disabled), the flashing LEDs on the controllers are not in sync. 4 of the 6 flash at the same time (and don't look to be the culprits), the other two flash differently and not in sync with each other either. If the PWM cable is unplugged from the RoboRio for only the suspect pair and then the bot is powered on, they flash in sync with the other 4 (and after 30 minutes, no motors fired off).


Is this a known condition with a known cause or am I chasing gremlins at this point? I am going to try moving the PWM cable to another port on the RoboRio (a new Y and extension as well) and then also swap in the spare controller that we have onboard. I am just curious if anyone saw this condition before. It is very unnerving to have the bot suddenly come alive (although on blocks) when working with it.


Nate

[gpetilli]

Since it is happening in Disabled, it is unlikely software related.

If it is always the same motor, I would start by replacing the speed controller.

An electrical short to the frame could create some very weird behavior. With the battery disconnected, try measuring with a Digital Volt Meter (DVM) set on ohms the resistance from each terminal of the PDB connector to the frame. The inspection checklist says that this should be >10kohm (probably >1Mohm).

Last, but least likely, sometimes if the PWM cables are close to noisy power cables, the coupled noise can look like a signal to the speed controller.

Are your PWMs connected directly to the roboRio or are you using a MXP board?

[Jon Stratis]

It sounds like there might be an issue with that particular Y cable. You said there weren't any visual strands poking out, but if the cable got crimped at some point, or if there was a manufacturing defect in the insulation when it was assembled, there could be someplace in the cable that is shorting out occasionally. It may be small enough that, when enabled, the PWM signal from the roboRIO is strong enough to mask it out, regardless of what value you're instructing the motor to go.

Try just replacing the Y cable, and if that works cut the other one in half* and toss it!


* Make it unusable so someone else on the team doesn't see it sitting on top of the trash and think "why is this being thrown out? I'll save us a buck and stick it in our spare parts drawer so we don't have to buy another one down the line."

[rich2202]

In disabled mode, the likely options are the motor controller, the PWM cables, or the PWM port itself.

Try swapping controller, cables, and PWM port.

[MrRoboSteve]

I used to blame the digital sidecar for this sort of thing, but can't any more.

Another thing to check is whether you have swarf in your speed controller.

[scca229]

Update:

Still letting it run to make sure but it looks like it was the PWM port on the RoboRio. I moved it immediately after posting the original query, changed it in the code and redeployed. No random motor firings at this time and all 6 controllers are flashing in sync. It has been about 1 hour since I was fully up and testing with the code change and physical PWM move.

For completeness: Was in PWM 1, now in PWM 3.


Nate

[Brandon Ha]

I didn't read carefully, but it did not seem like you guys checked if there was a short. This would cause sporatic activities for some motors if they weren't grounded properly. You may want to check this out.

EDIT:
Also, the blinking lights do mean something. Look up on the victor website or somewhere else to find out the meaning of each light code.

[scca229]

Originally Posted by Brandon Ha

I didn't read carefully, but it did not seem like you guys checked if there was a short. This would cause sporatic activities for some motors if they weren't grounded properly. You may want to check this out.

EDIT:
Also, the blinking lights do mean something. Look up on the victor website or somewhere else to find out the meaning of each light code.

The checking for shorts was what I meant by no stray copper crossing. Checking all sorts of places for continuity and such doesn't show anything weird nor does a visual look. Moving the PWM connection does seem to have cured the symptom as it hasn't done it since. We will still be wary of it just in case but I feel more comfortable at this point, which is good since I need to actually take the thing off blocks to get final adjustments completed.

The LED on a Victor 888 has 4 modes, the first 3 being in relation to calibrated throttle. Green (full forward), Red (full reverse), Orange (Neutral), Flashing Orange (no PWM sensed). Normally when you power on the bot, all of the Victors will be flashing in sync since the RoboRio is booting up and going into Disabled mode. This pair, connected by a Y-Cable, was not synced which means it was getting "something" on PWM immediately at power-on and then at random times afterward.

I did catch the LED during one of the last "events" and it was solid Green, which is the Full Forward color for that side of the drivetrain, and then went back to flashing orange...all while the bot was in Disabled.

[Alan Anderson]

Noise on a PWM control line can -- and often will -- be interpreted by a speed controller as a throttle command. Normally the impedance of the PWM output of the controller is low enough that noise is not a problem. If something is not quite right with the driver and/or pullup resistor on the problematic PWM channel, the cable might be picking up random impulses that the speed controller reacts to by turning on.

I know it's getting kind of late to do any sort of experimentation, but if you can verify that using PWM1 can reproduce the problem, you might want to ask if NI would like to have it back for analysis.

[Ozuru]

I had an issue similar to this with PWM ports (but not moving in disabled) and it wound up being the talons. Try calibrating your talons if you haven't already.

[RoboNerd01]

We are having the same problem! I will update this when we fix it.

[GeeTwo]

Originally Posted by scca229

Update:

Still letting it run to make sure but it looks like it was the PWM port on the RoboRio. I moved it immediately after posting the original query, changed it in the code and redeployed. No random motor firings at this time and all 6 controllers are flashing in sync. It has been about 1 hour since I was fully up and testing with the code change and physical PWM move.

For completeness: Was in PWM 1, now in PWM 3.


Nate

Check PWM port 1 for swarf or similar shorting. If you can't find a cause for the problem, lock that port out - tape over it or something similar to reduce the likelihood that someone will use it again.

[theawesome1730]

If you can isolate the issue to PWM channel 1 (or 0?) I would get in contact with NI. It's likely they would want to take it apart to find faults or this particular failure mode.

[GreyingJay]

We noticed an issue like this on our practice field, and eventually narrowed it down to PWM cables that had started to work their way loose out of the roboRio over time. Tightening everything down fixed it, but not before a lot of head scratching!

[daliberator]

We had a similar problem happen to us on Friday, except it would only be triggered when the frame was bumped. We tested with a rubber mallet to see if it was a short and eventually just changed the Y-PWM cable and it worked like a charm.