Chief Delphi - runtime thread execution monitoring (identify cause of hung code)

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Programming (http://www.chiefdelphi.com/forums/forumdisplay.php?f=51)

- - runtime thread execution monitoring (identify cause of hung code) (http://www.chiefdelphi.com/forums/showthread.php?t=107740)

Ether

09-08-2012 21:47

runtime thread execution monitoring (identify cause of hung code)

See following posts for some interesting ideas.

Todd	09-08-2012 22:34

Re: runtime thread execution monitoring (identify cause of hung code)

The method that a lot of embedded systems watchdogs I've come across is the following:

High priority task maintains a collection of bits for other periodic tasks.

High priority task (at the rate that other tasks are expected to run) sets bits false.

Other tasks set the bits true when they run.

If the High priority task, when it goes to set the bits notices that they haven't been flipped back, then it knows that the low priority task has not run in that timeframe.

Once it realizes that a certain number of frames have passed without the other task 'reporting in', it knows something is wrong. You don't really need mutual exclusivity or interrupts to maintain this, so long as you can ensure that the 'task monitor' thread is of elevated priority.

There's no metrics available on 'how much' of the other tasks available cycle was consumed before it was completed, but that could be added fairly trivially.

I work with several systems like this, if the 'tried and true' notion is of any value. :P

Ether

09-08-2012 23:39

Re: runtime thread execution monitoring (identify cause of hung code)

Quote:

Originally Posted by Todd (Post 1181118)

If the High priority task, when it goes to set the bits notices that they haven't been flipped back, then it knows that the low priority task has not run in that timeframe.

Once it realizes that a certain number of frames have passed without the other task 'reporting in', it knows something is wrong.

I like that approach Todd.

Has anybody else used this or something similar, and if so, did it prove helpful?

Greg McKaskle

10-08-2012 07:23

Re: runtime thread execution monitoring (identify cause of hung code)

This bit system is similar to how the LV diagram does execution hilighting. The bits are cleared by the diagram before node code begins, and nodes flip their bit as they execute. But this data structure is problematic on multi-core architecture performance since you get cache coherency issues as each node sprinkle bits as a side-effect of their run. It is still correct, but the overhead of this tracing goes up significantly with parallel cores and especially parallel packages.

Also, can you give more detail on why you don't need a mutex? From your description I can see folks writing a read/modify/write piece of code with no mutex, and getting very confused. Are you using something like __sync_fetch_and_or()?

Greg McKaskle

Ether

10-08-2012 08:47

Re: runtime thread execution monitoring (identify cause of hung code)

Quote:

Originally Posted by Greg McKaskle (Post 1181139)

Also, can you give more detail on why you don't need a mutex? From your description I can see folks writing a read/modify/write piece of code with no mutex,

Yeah, I saw that too but forgot to comment about it.

Asserting a bit in a word is not an atomic operation. You have to read the word, or it with a mask, and write it back. You could get interrupted in the middle of the sequence.

Since memory is not an issue (for FRC), instead of flipping bits you could use an array of words*. Then there'd be no contention among the threads.

* the length of which is whatever is atomic for the processor and most efficiently handled by it

Ether

10-08-2012 18:22

Re: runtime thread execution monitoring (identify cause of hung code)

How about using a counter instead of a flag?

Code:

* start thread *

push_flags();

disable_interrupts();

threadID++;

pop_flags();

* put thread code here *

* end thread *

Monitoring thread reads and resets (zeros) counters. Keeps track of when each counter was last reset. Uses counter value and elapsed time (since last reset) to monitor thread health.

Todd	10-08-2012 19:13

You're both right, an atomic fetch and set is required at a minimum. Once you accept that, and expand to fetch and increment and fetch and decrement operations, there's a wide variety of metrics that can be prepared and transmitted later on to a dashboard or logged in a file.

I'm imagining pursuing this with functional global variables in labview with our team this year.

brian.axelrod

11-08-2012 21:01

Re: runtime thread execution monitoring (identify cause of hung code)

If you're using C++, vxworks has a built in watchdog that can be useful.
You can use it as such.

You need to include wdLib.h

Code:

WDOG_ID watchDog = wdCreate();

wdStart(watchDog, ticks, OvertimeCallback, param);

Once the time specified in the second parameter (in ticks) expires the function specified by the OvertimeCallback is called. You can get the number of ticks per second with the
sysClkRateGet() function. If your code completes in time you cancel the call with

Code:

wdCancel(watchDog);

You can find the documentation for the vxworks watchdog here http://www-kryo.desy.de/documents/vx...ref/wdLib.html

There may be equivalents for the other programming languages.

If you are looking for more tools for evaluating profiling you can see the profiler we wrote. Our code is open source and posted on our website lynbrookrobotics.com

All times are GMT -5. The time now is 00:48.