NI Pneumatics Module "sputtering"
 

We've been experiencing an intermittent problem with our NI pneumatics module and or our cRio. Our module will randomly start "sputtering"; turning the engerized solenoids off and on quickly (250ms or so).

We first suspected a short somewhere in the system so we started unplugging things until we were left with only the cRio loaded with only the pneumatics module tethered to the driver station via a network cable.

I'm comfortable with Labview and the use of the solenoid vis. To be sure I was not doing something weird in programming I used the example program for testing. Using Labview and the solenoid example I can light up one light on the solenoid module when I set the front panel control to true. Since no changes have been made to the example (except the cRio IP) I'm confident the problem is not programming logic.

When the program runs, the solenoid light on the module that I have energized will light up and "flicker" intermittantly. Sometimes it goes minutes with no flicker. Sometimes it will "flicker" repeatedly over a short period. This "flicker" is whats causing our sputtering on the solenoids when we have them hooked up.

I've re-imaged the cRio. From the steps I've taken I believe the solenoid module to be flakey.

I'll be starting a support call with NI on monday morning. In the mean time...

Anyone seen similar behavior with their module or have further troubleshooting ideas?

Re: NI Pneumatics Module "sputtering"
 

In the meantime...replace the power wires to the solenoid module. Just for something to do.

Re: NI Pneumatics Module "sputtering"
 

Nick,
Are you using 24 volt solenoids? Are they powered from the 24 volt output of the Power Distribution Board? Have you followed the Power Distribution diagram Rev A and checked out Team Update #2?

Re: NI Pneumatics Module "sputtering"
 

Thanks for the quick replies guys!

We are using all 12v solenoids on the bot and power to the solenoid bumper is wired according to the 12v solenoid diagram.

I did suspect the power to the solenoid breakout module bumper. We swapped the wiring and the wago connector to the breakout module bumper.

To be clear, the sputtering still happens when I have only the cRio with only the pneumatics module hooked up running the Labview solenoid example. Nothing else is hooked up to the cRio including the pneumatics bumper. The module lights that indicate what solenoids should be energized still flutter.

Re: NI Pneumatics Module "sputtering"
 

This may be a programming issue. You may also have an unusual electrical issue. Have you tried replacing the breaker that feeds the solenoids? These sometimes have a manufacturing defect that causes them to be intermittent. Also double check the wires in the Wago on the PD. They should be stripped 1/2=5/8" in order to fully insert in the connector.

Re: NI Pneumatics Module "sputtering"
 

Does this also happen on other outputs? If so, this could also be a system watchdog issue. Install the new DS update released 2 days ago and see if it fixes the issue.

Re: NI Pneumatics Module "sputtering"
 

Team member last night during troubleshooting "whats this watchdog error.."

Me - known issue, ignore it.

Watchdog error happens, my sputtering happens.

No watchdog error, no problem.

Installing the update now. Will post if it comes back again.

Thanks to all!

Re: NI Pneumatics Module "sputtering"
 

We had the EXACT same issues for a couple of weeks which was a watchdog issue.
I kept suspecting it was a hardware thing because the symptoms seemed like poor connections somewhere. We rechecked everything 10 times over and the problem would occur sometimes more often than others.

I'll let our programmer tell you what he did exactly, but basically it was a programming fix, cleaning up a few things.

Re: NI Pneumatics Module "sputtering"
 

Has anyone seen the latest updates eliminate their watchdog timer issues? We are using LabView, and after updating the cRIO and dashboard we still have the intermittent watchdog issue that we saw before the latest updates.


Merle Yoder
Programming/Electrical Mentor
GRUNTS Team #3146

Re: NI Pneumatics Module "sputtering"
 

Just before kickoff, the Comm thread and FPGA were changed to update the Diagnostic with counts of even the smallest glitches. The latest DS definitely makes changes to improve the timing and eliminate many of the glitches.

At this point, it would be helpful if details of how to reproduce the glitches were posted. Before we go there, here are some things to consider.

The print statements do not differentiate between user induced disables and watchdog timeouts. Therefore, some of the diagnostics print statements will occur naturally.

If the dashboard runs on the same computer as the DS and is misbehaving, it is feasible that it could starve the DS thread enough to cause watchdog glitches. If the robot code misbehaves, it may be able to cause occasional glitches. Some teams have also noticed a .NET conversion tool that seems to get caught in a loop comsuming lots of CPU.

If you have a repeatable situation, please post the details. It may also be useful to notice the classmate CPU usage and cRIO CPU usage.

Greg McKaskle

Re: NI Pneumatics Module "sputtering"
 

Ok,
I just called our programmer who is out for today.
We never had that problem again after he made some changes and did some troubleshooting.
The main problem was that the dashboard was running on the classmate. We think that was the majority of it as we moved it onto our second laptop. Seems that its possible the classmate's CPU was on overload.
We also moved the functions for robot out of teleop and into the periodic functions, getting as much things out of teleop as possible.
The problem was fixed before the latest update came out.
Hope that helps for what its worth.

Re: NI Pneumatics Module "sputtering"
 

We have never had a Watchdog issue, at all. The DashBoard has no effect whatsoever on the Watchdog, since it runs on the Classmate not the cRio. The classmate has plenty of power for DriverStation and DashBoard.

You are not confined to Teleop and Periodic Tasks, you can put code in Main. We have a separate VI for each main function called from Main, each VI handles its own loop timing and such. This puts almost nothing in Teleop, allowing it to run and feed the watchdog.

Re: NI Pneumatics Module "sputtering"
 

You're probably right.
We did both fixes at the same time and couldnt correlate whether it was due to either one or both.

Re: NI Pneumatics Module "sputtering"
 

There are two watchdogs. The first is the user watchdog which is entirely contained in the cRIO and controlled by the the watchdog class/vis. I think that is what you are referring too. The second is the system watchdog which is handled by the "behind the scenes" code and is dependent on the network communication with the driver station. If the driver station doesn't send data quickly enough (because it was overloaded), it would trigger the system watchdog.

Re: NI Pneumatics Module "sputtering"
 

We are having the exact same problem, we have done all the updates and have minimal code in our main loop, the only thing it calls is our defined function that keep the solenoid "armed" so that we can fire.

Is there any fix for this, to get around the watch dog.

Thanks in advance

Ben