Feeding the Watchdog in "Periodic Tasks"

[Austin3324]

Hello all,

During the Buckeye Regionals, I was set to the task of trying to create an autonomous program in three days time when I had absolutely no LabView experience to speak of. (I was the backup programmer and the other programmers neither showed up nor commented their code.) In the end, we had no autonomous program.

I had some help from the programming mentor of team 1629 (I hope I got that right) and when we worked on the autonomous program, he fed the watchdog in the 100 ms loop of the Periodic Tasks vi. I currently don't have the team computer and am programming a newer program on my own laptop and am having troubles recreating what we did at the competition. If anyone can help, I would be most appreciative and if you need anything, please, just tell me.

Thanks,

Austin

[Greg McKaskle]

Redoing what the other mentor did is fine, but more important is to understand why it was done.

Watchdogs are there to verify that motors, pneumatics, and other potentially dangerous outputs are being controlled safely. Any place in the program where you are updating an output, you should consider what would happen if this code were to break. Also, how would you know if it broke and weren't running?

A tool for the second element is to have the loop update, or feed, the watchdog. Then set the update requirements of the watchdog to be related to the output rate. If the mentor fed the watchdog within the periodic task loop running at 100ms, it is probably because the loop does pneumatics or other outputs.

Does that make sense?

Greg McKaskle

[Mark McLeod]

Personally, I wouldn't put Watchdog handling in a Periodic task unless you really understand everywhere the Watchdog is being used and why.
Periodic tasks do not typically require a Watchdog at all.

The main error I see is that using the Watchdog in a Periodic Task generally invalidates all other uses of the Watchdog in the parallel Auto or Teleop tasks.
Parallel Watchdogs just tend to preempt each other.

Generally, you want the Watchdog to be handled in a single critical path.

[Ether]

Originally Posted by Mark McLeod

Personally, I wouldn't put Watchdog handling in a Periodic task unless you really understand everywhere the Watchdog is being used and why.
Periodic tasks do not typically require a Watchdog at all.

The main error I see is that using the Watchdog in a Periodic Task generally invalidates all other uses of the Watchdog in the parallel Auto or Teleop tasks.
Parallel Watchdogs just tend to preempt each other.

Generally, you want the Watchdog to be handled in a single critical path.

I agree with Mark on this one.

If you are concerned about the safety of a periodic task, then monitor its health from a task (like TeleOP or Auto) which is protected by watchdog.

For example, increment a global variable in the periodic task, and test the variable in TeleOP (or Auto). If the variable stops incrementing or is not incrementing at the expected rate, then trip the watchdog within TeleOp (or Auto).


~

[The Lucas]

Personally, I always just disable the User Watchdog. Putting it the Periodic task is the closest thing to disabling it (see Marks post) so just disable it. I know there are good reasons to have it, but I am not satisfied with how FMS handles a Watchdog error so I will not use it in competition (but I use it during code development). If you throw a Watchdog at FMS there seems to be no way to gracefully recover from it and you are going to have communication problems for the rest of the match even once you move past the problematic code.

I first observed this last year in the Philly elims. We ran the wrong auto which was still in the debug stages (had printf statements). The code would work if run with our own router in the shop since the delay for printf is low in that case. Of course that delay is longer on the field so it throws watchdogs and stops during auto. Thats fine and I would expect that, but it should run teleop fine right? No, there is a long delay until we can run teleop (we are using a modified IterativeRobot there is no way to be stuck in auto). We spend the whole rest of the match going in and out of comm (more time out of comm but we did get to run telelop for like 20 continuous sec at a time). Now if we had disabled watchdog before that match instead of for every match afterward, we would have likely run a slow unresponsive auto followed by a perfectly fine teleop. I have seen this this with some teams that throw watchdog errors in auto and have periodic comm problems for the rest of match. Star by disabling watchdog, and the comm problems go away but auto still doesnt work right. Fix auto bug after some more debugging time and everything is good.

In short, IMHO throwing a Watchdog error at FMS is about the worst thing you can do with your code. You can cause yourself to lose comms for the rest of the match, and some people claim it can cause others to lose comms (I am not sure if I believe this myself).

[Ether]

Originally Posted by The Lucas

If you throw a Watchdog at FMS

Could you please explain what you mean by "throw a watchdog at FMS" ?

It was my understanding that FMS has no knowledge of when a user watchdog error has occurred.


~

[The Lucas]

Originally Posted by Ether

Could you please explain what you mean by "throw a watchdog at FMS" ?

It was my understanding that FMS has no knowledge of when a user watchdog error has occurred.


~

There are better people to answer this on this forum as I have never worked with FMS (I volunteer as an inspector). While FMS doesn't really monitor user code closely it does watch watchdog errors as per my understanding. "Watchdog not Fed" showing up as state your FMS connected Classmate should be evidence of this as FMS controls state.

[Austin3324]

Well, I must thank you all for your quick responses.

So, the general consensus seems to be to not put watchdog into the Periodic Tasks vi or to just "disable" it all together. Considered what you all said Watchdog does, I would prefer not to disable it.

We currently have watchdog working fine in our Teleop vi and it's just been a problem with getting watchdog to work in the Autonomous Independent vi. The way I see it is to throw in a second while loop, separate from the main loop, and just tell that to feed the watchdog something like every 20 ms. (The reason it has to be separate is because the main loop will only run anywhere from 1-3 times before it exits.)

If you need it, I've included a copy of the vi that currently houses my code for the Autonomous Independent vi.

Thanks again for your responses.

Austin

[Ether]

Originally Posted by Austin3324

We currently have watchdog working fine in our Teleop vi and it's just been a problem with getting watchdog to work in the Autonomous Independent vi. The way I see it is to throw in a second while loop, separate from the main loop, and just tell that to feed the watchdog something like every 20 ms.

Adding a concurrent "second while loop" separate from the "main loop" which does nothing but feed the watchdog defeats the purpose of the watchdog. The watchdog is not protecting your "main loop".

Originally Posted by Austin3324

(The reason it has to be separate is because the main loop will only run anywhere from 1-3 times before it exits.)

Are you getting watchdog errors in the auto independent state only after your "main loop" exits but before the state changes to TeleOp? If so, add a watchdog delay and feed after the main loop exits, to keep the watchdog fed until the state changes to TeleOp.

Or, are you getting watchdog errors while your "main loop" is running? If so, you probably have delays in your "main loop" code which are not feeding the watchdog.



~

[Greg McKaskle]

If the periodic task doesn't have an output that is the primary thing you are concerned about, then definitely do not feed it there.

What I'd like for the WD to evolve into is actually named entities associated with allocated outputs. You then can have as many watchdogs as you like and you can have a name as to the dogs that haven't been fed. At this point, you'd have an overview of the timing of the system. All enabled WDs would be anded together and affect the overall robot status. We'll see if we get around to doing anything like that in the future.

Greg McKaskle

[Ether]

Originally Posted by The Lucas

There are better people to answer this on this forum as I have never worked with FMS (I volunteer as an inspector). While FMS doesn't really monitor user code closely it does watch watchdog errors as per my understanding. "Watchdog not Fed" showing up as state your FMS connected Classmate should be evidence of this as FMS controls state.

http://www.chiefdelphi.com/forums/sh...6&postcount=15

"The FMS won't disable you whenever your User Watchdog goes off. It doesn't even know when it happens, nor care. That's a "user issue" to it."

~

[The Lucas]

Originally Posted by Ether

http://www.chiefdelphi.com/forums/sh...6&postcount=15

"The FMS won't disable you whenever your User Watchdog goes off. It doesn't even know when it happens, nor care. That's a "user issue" to it."

~

I wasn't implying that it disabled you, just that it leads to comm problems. All the info I have on FMS is second hand so I don't know if FMS sees the watchdog. I am not really saying this is FMS's fault. If your code messes up and throws a watchdog, it is hard to recover while connected to FMS.

[Mark McLeod]

Originally Posted by The Lucas

If your code messes up and throws a watchdog, it is hard to recover while connected to FMS.

This isn't true with FMS in my experience.
Most of the teams at SBPLI and NJ, the two events I volunteered at, went to Watchdog not Fed by the end of Autonomous simply when their drive code ran out early or when they had no autonomous code. There's nothing unusual with that. It's perfectly fine.

The user code on the cRIO continues to run. It isn't interrupted at all. Just the cRIO control outputs, e.g., PWM, are disabled by an unfed Watchdog.
Once you feed the watchdog again, the Watchdog re-enables the control outputs. That's why robots experience the momentary stalls when the Watchdog times out only for a second, but then come back again. There's nothing the Watchdog is doing that slows or otherwise blocks your code, and FMS isn't involved. The Watchdog has to work if FMS goes offline after all. It wouldn't be real useful if FMS or the field access point crashed and all the robots went bonkers...

One possibility is slow or marginally slow code. FMS does monitor the packet send times and there are quite a few instances I've seen or heard of where the robot packet times were 10 times slower than the other robots on the field. The cases where I was able to investigate turned out to be slow code or unbalanced parallel tasks.

Not directed at Brian, because I know he writes good code, but to the general CD readership...
It doesn't help to boost the Watchdog delay period. That's like turning up the car radio as a fix for that odd engine noise your car is making.
If you need to do that then something in your code needs to be streamlined. The Watchdog is there to help you discover those kinds of problems.

All that said, I agree with Brian about using the Watchdog for development and debugging purposes, not for competition use.
Misused, it causes more grief than it saves robots, and there's always the E-stop button which is an FMS control.

[Mark McLeod]

Originally Posted by Austin3324

We currently have watchdog working fine in our Teleop vi and it's just been a problem with getting watchdog to work in the Autonomous Independent vi. The way I see it is to throw in a second while loop, separate from the main loop, and just tell that to feed the watchdog something like every 20 ms. (The reason it has to be separate is because the main loop will only run anywhere from 1-3 times before it exits.)

Sorry I haven't responded directly to your question till now.
I've had a chance to look at your auto code and you need Watchdog Feeds in each of those interior While loops where you wait around for the Ball Sensor to trip or the Kicker Limit Switch to say you've completed your kick. You also need a Watchdog Delay & Feed wherever you do a timer wait, e.g., the one for 100ms.

In fact you probably should throw delays into each of those interior While loops to free up the CPU a bit to allow parallel tasks (such as the Watchdog) time to execute too. That'd mean doing Delay & Feeds rather than Feeds there.
A Delay & Feed in a loop or sequence frame like you're using does the action of a timer delay, so you can actually replace the timer delay with the Delay & Feed to get the same result.

The rest of the sequence code operates fast enough that you don't need Watchdog feeds everywhere. Placing Feeds wherever you take a lot of time will satisfy the rest of the sequence as well.

[Austin3324]

Originally Posted by Mark McLeod

I've had a chance to look at your auto code and you need Watchdog Feeds in each of those interior While loops where you wait around for the Ball Sensor to trip or the Kicker Limit Switch to say you've completed your kick. You also need a Watchdog Delay & Feed wherever you do a timer wait, e.g., the one for 100ms.

In fact you probably should throw delays into each of those interior While loops to free up the CPU a bit to allow parallel tasks (such as the Watchdog) time to execute too. That'd mean doing Delay & Feeds rather than Feeds there.

Thanks Mark.

I hope that this all works. I should be able to test it out on the robot in a few days so I'll definitely keep you all posted if there are any errors. Also, as a final check, here's a copy of the revised vi. (I took your advice in using Delay and Feeds as opposed to just Feeds.)

Again, thanks for the help and hopefully all will work well!

Austin