NI Pneumatics Module "sputtering"

[nick1023]

We've been experiencing an intermittent problem with our NI pneumatics module and or our cRio. Our module will randomly start "sputtering"; turning the engerized solenoids off and on quickly (250ms or so).

We first suspected a short somewhere in the system so we started unplugging things until we were left with only the cRio loaded with only the pneumatics module tethered to the driver station via a network cable.

I'm comfortable with Labview and the use of the solenoid vis. To be sure I was not doing something weird in programming I used the example program for testing. Using Labview and the solenoid example I can light up one light on the solenoid module when I set the front panel control to true. Since no changes have been made to the example (except the cRio IP) I'm confident the problem is not programming logic.

When the program runs, the solenoid light on the module that I have energized will light up and "flicker" intermittantly. Sometimes it goes minutes with no flicker. Sometimes it will "flicker" repeatedly over a short period. This "flicker" is whats causing our sputtering on the solenoids when we have them hooked up.

I've re-imaged the cRio. From the steps I've taken I believe the solenoid module to be flakey.

I'll be starting a support call with NI on monday morning. In the mean time...

Anyone seen similar behavior with their module or have further troubleshooting ideas?

[MrForbes]

In the meantime...replace the power wires to the solenoid module. Just for something to do.

[Al Skierkiewicz]

Nick,
Are you using 24 volt solenoids? Are they powered from the 24 volt output of the Power Distribution Board? Have you followed the Power Distribution diagram Rev A and checked out Team Update #2?

[nick1023]

Thanks for the quick replies guys!

We are using all 12v solenoids on the bot and power to the solenoid bumper is wired according to the 12v solenoid diagram.

I did suspect the power to the solenoid breakout module bumper. We swapped the wiring and the wago connector to the breakout module bumper.

To be clear, the sputtering still happens when I have only the cRio with only the pneumatics module hooked up running the Labview solenoid example. Nothing else is hooked up to the cRio including the pneumatics bumper. The module lights that indicate what solenoids should be energized still flutter.

[Al Skierkiewicz]

This may be a programming issue. You may also have an unusual electrical issue. Have you tried replacing the breaker that feeds the solenoids? These sometimes have a manufacturing defect that causes them to be intermittent. Also double check the wires in the Wago on the PD. They should be stripped 1/2=5/8" in order to fully insert in the connector.

[Joe Ross]

Does this also happen on other outputs? If so, this could also be a system watchdog issue. Install the new DS update released 2 days ago and see if it fixes the issue.

[nick1023]

Team member last night during troubleshooting "whats this watchdog error.."

Me - known issue, ignore it.

Watchdog error happens, my sputtering happens.

No watchdog error, no problem.

Installing the update now. Will post if it comes back again.

Thanks to all!

[waialua359]

We had the EXACT same issues for a couple of weeks which was a watchdog issue.
I kept suspecting it was a hardware thing because the symptoms seemed like poor connections somewhere. We rechecked everything 10 times over and the problem would occur sometimes more often than others.

I'll let our programmer tell you what he did exactly, but basically it was a programming fix, cleaning up a few things.

[Merle]

Does this also happen on other outputs? If so, this could also be a system watchdog issue. Install the new DS update released 2 days ago and see if it fixes the issue.

Has anyone seen the latest updates eliminate their watchdog timer issues? We are using LabView, and after updating the cRIO and dashboard we still have the intermittent watchdog issue that we saw before the latest updates.


Merle Yoder
Programming/Electrical Mentor
GRUNTS Team #3146

[Greg McKaskle]

Just before kickoff, the Comm thread and FPGA were changed to update the Diagnostic with counts of even the smallest glitches. The latest DS definitely makes changes to improve the timing and eliminate many of the glitches.

At this point, it would be helpful if details of how to reproduce the glitches were posted. Before we go there, here are some things to consider.

The print statements do not differentiate between user induced disables and watchdog timeouts. Therefore, some of the diagnostics print statements will occur naturally.

If the dashboard runs on the same computer as the DS and is misbehaving, it is feasible that it could starve the DS thread enough to cause watchdog glitches. If the robot code misbehaves, it may be able to cause occasional glitches. Some teams have also noticed a .NET conversion tool that seems to get caught in a loop comsuming lots of CPU.

If you have a repeatable situation, please post the details. It may also be useful to notice the classmate CPU usage and cRIO CPU usage.

Greg McKaskle

[waialua359]

Ok,
I just called our programmer who is out for today.
We never had that problem again after he made some changes and did some troubleshooting.
The main problem was that the dashboard was running on the classmate. We think that was the majority of it as we moved it onto our second laptop. Seems that its possible the classmate's CPU was on overload.
We also moved the functions for robot out of teleop and into the periodic functions, getting as much things out of teleop as possible.
The problem was fixed before the latest update came out.
Hope that helps for what its worth.

[apalrd]

We have never had a Watchdog issue, at all. The DashBoard has no effect whatsoever on the Watchdog, since it runs on the Classmate not the cRio. The classmate has plenty of power for DriverStation and DashBoard.

You are not confined to Teleop and Periodic Tasks, you can put code in Main. We have a separate VI for each main function called from Main, each VI handles its own loop timing and such. This puts almost nothing in Teleop, allowing it to run and feed the watchdog.

[waialua359]

We have never had a Watchdog issue, at all. The DashBoard has no effect whatsoever on the Watchdog, since it runs on the Classmate not the cRio. The classmate has plenty of power for DriverStation and DashBoard.

You are not confined to Teleop and Periodic Tasks, you can put code in Main. We have a separate VI for each main function called from Main, each VI handles its own loop timing and such. This puts almost nothing in Teleop, allowing it to run and feed the watchdog.

You're probably right.
We did both fixes at the same time and couldnt correlate whether it was due to either one or both.

[Joe Ross]

We have never had a Watchdog issue, at all. The DashBoard has no effect whatsoever on the Watchdog, since it runs on the Classmate not the cRio. The classmate has plenty of power for DriverStation and DashBoard.

There are two watchdogs. The first is the user watchdog which is entirely contained in the cRIO and controlled by the the watchdog class/vis. I think that is what you are referring too. The second is the system watchdog which is handled by the "behind the scenes" code and is dependent on the network communication with the driver station. If the driver station doesn't send data quickly enough (because it was overloaded), it would trigger the system watchdog.

[ilmondo]

We are having the exact same problem, we have done all the updates and have minimal code in our main loop, the only thing it calls is our defined function that keep the solenoid "armed" so that we can fire.

Is there any fix for this, to get around the watch dog.

Thanks in advance

Ben

[Mark McLeod]

Does your defined function have a loop as part of it?

[DanielForero]

We have the exact same problem and tried to fix it in many different ways.

Here are our results:

Disabled Camera Feed -> NO EFFECT
Hard Wired Robot (No WIFI) -> NO EFFECT
Increased Watchdog Timeout -> NO EFFECT
Installed Dashboard on Another Laptop -> NO EFFECT
Added a 10ms Wait Command in Teleop Code -> NO EFFECT
Updated Dashboard to most recent firmware -> SOME IMPROVEMENT

When we updated the dashboard, the "sputtering" did not happen as often, but it is still happening. We have spent well over 6 hours on this, to no prevail. Tomorrow we are going to try different methods of testing the network traffic, we have a feeling that the traffic is overwhelming hence the watchdog timeouts. We will keep you guys updated on our progress. Good luck to all teams!

[Merle]

[QUOTE=DanielForero;921654]We have the exact same problem and tried to fix it in many different ways.

Here are our results:

Disabled Camera Feed -> NO EFFECT
Hard Wired Robot (No WIFI) -> NO EFFECT
Increased Watchdog Timeout -> NO EFFECT
Installed Dashboard on Another Laptop -> NO EFFECT
Added a 10ms Wait Command in Teleop Code -> NO EFFECT
Updated Dashboard to most recent firmware -> SOME IMPROVEMENT
QUOTE]

How do you increase the watchdog timeout? I've tried looking lower and lower in the watchdog Vis but eventually hit a password protected block.

We have had instances of multiple watchdog timeouts, it is now occuring only with one to four timeouts per minute (noticable as the compressor spike toggles off for a fraction of a second). I've monitored our Classmate CPU usage and it is only at ~50% when the timeouts occur.

Merle Yoder
GRUNTS Team #3146
Granby, CT

[Vikesrock]

How do you increase the watchdog timeout? I've tried looking lower and lower in the watchdog Vis but eventually hit a password protected block.


The Watchdog expiration is wired in to the Watchdog Open VI. The default is .5 seconds.

[DanielForero]

Yes, default is .5 seconds ... changing that does nothing in our case. By the way, i should point out that we're using C++...

[ilmondo]

In our pneumatics code we have a case statement so its not exactly a loop, however ever since doing the updates the glitches happen even more frequently, so we have stripped everything out and are basically starting again with a new project.

The only one that seems to glitch is the solenoid we have open most of the time as it is there to "lock" our firing mechanism.

Ben

[Mark McLeod]

If the User Watchdog is giving you too many problems, it can be disabled.
But it is probably warning of a portion of your code that is too slow and you may have other problems, such as, delayed responses because of it.