Watchdog!?!?!?!

[big1boom]

We only experienced problems when on the field. Tethered in the pits we had autonomous, on the field we had not autonomous, and had no teleoperated except our manipulator. Thank you Simbotics for finding our problem.

[keehun]

Originally Posted by Wayne TenBrink

Our programming mentor has about 2 months worth of experience with LabView.

Don't worry, I was the main programmer and that's about what I had, too.

Originally Posted by Wayne TenBrink

When you all describe "feeding the watchdog", what actually does that? From your description, I assume that its some built-in, behind the scenes, function of the code that just happens without any special "instruction" from our programmer. And if the code is too busy or gets stuck in a "while" loop, it doesn't feed the watchdog.

Yes. There is a WPI library block which gets input a boolean(?) which I think is tied to some low level vxWorks call. When vxWorks doesn't see it, then it probably stops the .rtexe and decides to tell the DS that the Watchdog has sadly starved to death. (figuratively)

Originally Posted by Wayne TenBrink

What would make that happen only intermittently?

I highly doubt it is anything but the code..

Originally Posted by Wayne TenBrink

We would like to think it was field/system related, but we really don't know.

This, I highly doubt, especially if it was only your robot with this problem.

Originally Posted by Wayne TenBrink

Is there a credible scenario where static electricity could be involved?

What I think is possible is that your battery was low (6~7 volts) and cRio had problems. This "symptom" would be different, so I doubt it.

Originally Posted by Wayne TenBrink

What about startup sequencing? I've read where some teams had communication problems when they powered up the robot before the DS. We generally powered up the DS first, but probably not always. I wouldn't think that would necessarily have anything to do with watchdog.

This would be more of an issue with "No Comms" error message, which is worse.

Originally Posted by Wayne TenBrink

We generally like the new control system and LabView, but are anxious to find out where the bugs are hiding out and get rid of them.

If you compete in another regional, maybe you can upload your code and we can look it over for any unending loops?

Good luck!
Keehun
Team 2502

[StephenB]

Originally Posted by keehun

Yes. There is a WPI library block which gets input a boolean(?) which I think is tied to some low level vxWorks call. When vxWorks doesn't see it, then it probably stops the .rtexe and decides to tell the DS that the Watchdog has sadly starved to death. (figuratively)

The watchdogs are actually implemented on the FPGA. So when the cRIO gets a new TCP packet it feeds the FPGA system watchdog. And if you enable the user watchdog and call the feed subVI... you are feeding the FPGA section for that watchdog. Again, this is explained in more detail at: http://decibel.ni.com/content/docs/DOC-2957

I highly doubt it is anything but the code..
This, I highly doubt, especially if it was only your robot with this problem.

Yes exactly. If you say... set up your user watchdog to have a 0.5s time out... and you only feed the watchdog every 1s. Then your motors will cut out for half of every second.


The .rtexe doesn't get stopped. Your code keeps running just like before, but the FPGA gets set to a fail safe state, where no outputs are usable. You can still read in anything and run like normal, but when either watchdog is tripped no outputs are available.

[Arthur S]

This happened to us(team 4) about 3 years ago. Our lead programmer was using interrupts while trying to program the gear tooth sensor we had on our robot. This did not work out to well because the code had to finish within 23.6 milliseconds(if i remember correctly), but due to the interrupts it made the code exceed that time limit. This is what tripped the watchdog and caused a code error. You might wanna check any of your code that uses interrupts and make sure that no interrupts are interrupting interrupts(boy does that sound confusing). Hope i helped in any way.

[PinionTwister]

We did not experience the problem when tethered or when working with our own wireless system. Once connected to the field management system we experienced the same problem ("WATCHDOG" appearing in the Driver Station).

Check the wait timers in all your loops. Make sure you have wait timers in all loops!
Don't make them ALL the same value.
Feed the watchdog into the fastest loop.

[professorX]

Originally Posted by Wayne TenBrink

...
Two of the match failures occurred at the start of the match, resulting in no autonomous motion, and nothing after it either (until we learned how to reset the cRio from the DS (thanks 494)). One of our failures occurred after a successful autonomous. I don't know for sure if we got the "watchdog" message that time, but one side of the drive train (Jaguar/CIM) didn't work after autonomous. After the match, we found no problems and the problem didn't repeat.

Can you please share with us how to reset the cRIO from the DS?

I appreciate it.

[Tom Line]

To try to clarify this (and get rid of the cute "feed" term):

Feeding the watchdog is nothing more than letting the watchdog code run once in a while. This tells the system that the robot is still "under control".

A VERY easy way to not allow the watchdog code to run is to put your camera code in the same loop as the drive code. The camera code is processor hungry and will run as often as it can - and not allow the watchdog to run.

As other people have said - infinite loops, looping with no wait statement, and a couple other items can cause the watchdog to kick off. We struggled with it earlier when we put our camera code in our drive code as an experiment. It was a good learning experience!

[EricVanWyk]

Originally Posted by Stuart

ok just to be proactive. ( I have not run in to this problem with our own robot or any other that Ive messed with, but then Murphy's law n such.)

if you take out all the delay and feeds in auto mode. and replace them with your own way of delaying, AND place a parallel 100ms feed loop, everything will be OK?

The parallel loop effectively kills the usefulness of the watchdog - even if your important code hangs, this artificial loop will keep it fed. At that point, just disable the dog and be done with it (I don't recommend this).

[Wayne TenBrink]

Regarding our robot, apparently we do have some camera code in the program even though we aren't using the camera. We had previously planned to use it, and our programmer had made some progress with tracking software. We later decided to focus on defensive maneuvers and enter teleoperation with a loaded hopper.

Any code for driving the motors (or servos?) is disabled, but apparently the code is still trying to collect images from the camera. We will remove all of that. I don't know if it is (was) in the same loop as the drive code.

We will also verify all the device setup (IP addressing, etc.).

To professorX: To reset the cRio from the DS, simultaneously press all three white buttons on the DS and hold for a second or so. A new menu will come up and you can "select" reset. It takes about 20 seconds, but thats a lot better than sitting idle for the entire match. I recommend you test/practice in the pit on tether.

Thanks to all for your help on this.

[NC GEARS]

Yes, thanks to everyone for the help. Hopefully we can resolve our issues with your support. Thanks again to everyone.

[Kingofl337]

I think the driver station should be updated so that it reads either
"System Watch Dog" or "User Watch Dog" that would help teams who
are not sure which is active.

I also wish they would change the DS so it says "Status: No Code" instead of "Battery: No Code". Because people keep asking, "Why does the battery need code?"

[StephenB]

I think the driver station should be updated so that it reads either
"System Watch Dog" or "User Watch Dog" that would help teams who
are not sure which is active.

since the system watchdog is tripped entirely by communication loss, that event is already covered. 'No comms' means no network connection and therfore the system watchdog has been tripped. 'Watchdog' means the user.

[Mike Bennett]

Now that we have learned that we probably need a timer delay, the question has been raised as to "what is a reasonable delay?"
For our robot during autonomous mode the only thing that might be time critical is the reading of the gyro to test if we have turned the number of degrees that we want.

It would seem that we don't want to tie up the CPU any more than necessary.

[Tom Line]

In the past, all examples I have seen suggested something on the order of 5-10 milliseconds.

[professorX]

Originally Posted by Wayne TenBrink

Regarding our robot, apparently we do have some camera code in the program even though we aren't using the camera. We had previously planned to use it, and our programmer had made some progress with tracking software. We later decided to focus on defensive maneuvers and enter teleoperation with a loaded hopper.

Any code for driving the motors (or servos?) is disabled, but apparently the code is still trying to collect images from the camera. We will remove all of that. I don't know if it is (was) in the same loop as the drive code.

We will also verify all the device setup (IP addressing, etc.).

To professorX: To reset the cRio from the DS, simultaneously press all three white buttons on the DS and hold for a second or so. A new menu will come up and you can "select" reset. It takes about 20 seconds, but thats a lot better than sitting idle for the entire match. I recommend you test/practice in the pit on tether.

Thanks to all for your help on this.

Thank you for the help.