Still having some watchdog issues

[Lumit]

We just updated to LV2.1 and dashboard1.1, the documentation from NI said that this should solve the watchdog glitches. For us, it has helped, but not completely eliminated the problem, occasionally a solenoid will for a brief instant return to it's default position. On the dashboard we can see several watchdog timeouts.

This isn't a huge issue, it's fairly uncommon and doesn't affect the bot much. But it's a bit worrisome.

Anyone have any idea what else could be causing watchdog errors like this?

thanks

[Wolfgang]

This is a tough problem which my team had when it came to autonomous. Lots of our autonomous codes were not working because of the WatchDog. Mine was the only one that worked, and was the simplest one. Try making the code simpler.
My team's cheif programmer managed to fix one other code, but it was unclear how that was done, even to him.

[Lumit]

Originally Posted by Wolfgang

This is a tough problem which my team had when it came to autonomous. Lots of our autonomous codes were not working because of the WatchDog. Mine was the only one that worked, and was the simplest one. Try making the code simpler.
My team's cheif programmer managed to fix one other code, but it was unclear how that was done, even to him.

I think we're probably having different issues...
Ours is a problem for both autonomous and teleop, and it doesn't really cripple the code at all, it just will occasionally startle us with an unexpected solenoid twitch.

The only reason I'm at all worried about it, is that it could mess with our timing that prevents us from breaking the rules about movement outside of the bumper perimeter. (rule G30a)

[Greg McKaskle]

The update fixed the issues we knew of with the watchdog, but this is one of those where until all contributing factors are accounted for, it isn't completely fixed for everyone. I'm working with a team who still sees the watchdog glitches. If you see any patterns, please let us know. My current theory has to do with the virus scanning, but I need to show the data to some other people, then figure out various ways of addressing it, etc.

Greg McKaskle

[asbplane]

Our Team is having the same problem. We simply turn the bot on in TeleOp and it will randomly cause the compressor to turn off for a split second and simultaneously cause all of our solenoids and pistons to twitch. We thought it might have been a watchdog error because we put the robot up on blocks and drove it forward for a while and we didn't notice any "ghost fires". Im guessing that perhaps when no one is touching the joysticks, the watchdog goes off. I can't say for sure though.

[Merle]

I want to point out to teams that are experiencing occasional watchdog timeouts and think it is not a big deal as they only see a short glitch for a fraction of a second, at a competition it becomes a HUGE deal... the field control system will shut your communications down between your operator controls and the robot on just one watchdog error - you will have a dead robot for the rest of the match - we saw this in multiple matches at the Suffield Scrimmage (Suffield HS, CT) where an official FIRST field control was used. I have a suspicion that control can be regained by restarting your dashboard program, but this would take a while and is not a very good option during a match.

We were able to resolve our watchdog errors by removing all excess code (LabView) that we were not using (such as camera, gyro, and solenoids that were programed, but attachments not yet made on our robot) and ran for a few matches without errors.

Greg - thanks for the lead on the virus scanning, I think we had it turned off, but I will look into this. For us it was usually System Watchdog errors, and I've monitored Classmate CPU usage to be 50-80% when errors occurred, and have seen it at 100% (while shutting down LabView when dashboard was running) but did not have errors occur then.

Merle Yoder
GRUNTS Team #3146

[diviney]

For the record, we too are struggling with what appear to be watchdog issues. They manifest themselves as lags both with control assertion and de-assertion. We have the latest updates applied, but it did not address the problems. We have stripped our code down and sprinkled watchdog triggers in the code. After reading the previous post, we are concerned that this will become a fatal problem when we get to see a real Field Control system in competition.

Any thoughts would be appreciated

[Tom Line]

Our situation:

We have 2 crios. One on the practice bot, one on the comp bot. We have been using the practice bot with no problems (no disables when we get the watchdog messages).

Today we wired up the comp board and put it on the comp bot. Before we connected the victors to the drive motors, we powered everything up and downloaded the code to confirm function.

Every time we see a new "watchdog expired" message, all the victors "blink" for a second. The odd part is, they don't blink to disabled. Somestime they blink to red, sometimes to green... the blink is 100% irregular. We tried a new battery, rewired all the power wires to confirm no clamping on insultation, wiggled every circuit on the board to attempt to isolate the problem, unplugged every pwm except one and still had the problem on that one.

We just went through and reimaged to the newest image last night, plus the newest DS, and we're going to try tomorrow again before ship. However, I found it very odd that with the same code and the same DS the Crio's are acting differently, one blinking the victors during the watchdog expires and one not.

[Lumit]

Originally Posted by Merle

I want to point out to teams that are experiencing occasional watchdog timeouts and think it is not a big deal as they only see a short glitch for a fraction of a second, at a competition it becomes a HUGE deal... the field control system will shut your communications down between your operator controls and the robot on just one watchdog error - you will have a dead robot for the rest of the match - we saw this in multiple matches at the Suffield Scrimmage (Suffield HS, CT) where an official FIRST field control was used. I have a suspicion that control can be regained by restarting your dashboard program, but this would take a while and is not a very good option during a match.

NOT GOOD!
After Greg's post, we just decided we'd have to live with it. Tomorrow is ship day, we have no testbed.
I guess we'll have to pray for a miracle on unpack/practive day! :/

Maybe it's the cRio version? We just imaged the only version that showed up on the list of options, are there more?

EDIT: As for dumping bits of the code, we are using everything except the periodic tasks loop... Do you have any idea what part of the code it is? Maybe the camera feed is causing a delay or something?

[Greg McKaskle]

I do not believe that a watchdog occurrence will cause a robot to be disabled for remainder of a match. The FMS really doesn't even know whether the robot is watchdogged or not. Can anyone confirm this? Were there other circumstances such as an FTA disabling the robots?

Greg McKaskle

[JoeyTNT280]

Still having the same issues. Even with a code without any timing. This error is causing our robot to disable pretty rapily sometimes and then slowly others. We believe that this error will affect our on field piloting.

[Greg McKaskle]

If you are still having issues with watchdogs, please unplug the I/O module and determine what effect this has. We are still investigating the cause, but it appears that in some setups, the I/O board can cause excessive driver activity which can delay the loop execution and lead to a system watchdog.

Please post one way or the other as to whether this has an impact. Also, it may be useful to test with a different I/O module, different USB cable, etc.

Greg McKaskle

[kws4000]

we are also having watchdog issues. we have two fisher-prices on 2 seperate relays, both 20 amp. when tring to run them with a heavy load, we either blow the spike fuse, or we get a watchdog error. the compressor is also stuttering, also because of watchdog.

under diagnostics, DS:
"Watchdog Expiration: System XX, User YY" where XX and YY are disconnected numbers that only go up or stay the same.

[Lumit]

Greg,

If you are still having issues with watchdogs, please unplug the I/O module and determine what effect this has.

You are referring to the circuit-board like thing that is plugged into the classmate with a usb cable, correct? Our team decided not to use it at all.

I do not believe that a watchdog occurrence will cause a robot to be disabled for remainder of a match. The FMS really doesn't even know whether the robot is watchdogged or not. Can anyone confirm this? Were there other circumstances such as an FTA disabling the robots?

Could you possibly find out for 100% sure? My team certainly can't afford to be caught off-guard by this, it would take us out of the competition and waste all of our hard work.
Where can we learn more about the FMS? (that is the name for the central control?)

[Mark McLeod]

Originally Posted by Lumit

Could you possibly find out for 100% sure? My team certainly can't afford to be caught off-guard by this, it would take us out of the competition and waste all of our hard work.
Where can we learn more about the FMS? (that is the name for the central control?)

The FMS won't disable you whenever your User Watchdog goes off. It doesn't even know when it happens, nor care. That's a "user issue" to it.

What FMS responds to is that Stop button. On the real field the E-stop button is quite a bit more robust and hitting that WILL disable your robot for the remainder of a match.