Go to Post I am not a moderator, but I am John V-Neun. - Billfred [more]
Home
Go Back   Chief Delphi > Technical > Programming
CD-Events   CD-Media   CD-Spy   FRC-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Closed Thread
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 08-09-2012, 08:47 PM
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 5,851
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
runtime thread execution monitoring (identify cause of hung code)


See following posts for some interesting ideas.


Last edited by Ether : 08-10-2012 at 05:11 PM. Reason: better ideas posted
  #2   Spotlight this post!  
Unread 08-09-2012, 09:34 PM
Todd's Avatar
Todd Todd is offline
Software Engineer
FRC #1071 (Team Max)
Team Role: Mentor
 
Join Date: Feb 2005
Rookie Year: 2004
Location: Connecticut, Wolcott
Posts: 51
Todd is just really niceTodd is just really niceTodd is just really niceTodd is just really niceTodd is just really nice
Re: runtime thread execution monitoring (identify cause of hung code)

The method that a lot of embedded systems watchdogs I've come across is the following:

High priority task maintains a collection of bits for other periodic tasks.

High priority task (at the rate that other tasks are expected to run) sets bits false.

Other tasks set the bits true when they run.

If the High priority task, when it goes to set the bits notices that they haven't been flipped back, then it knows that the low priority task has not run in that timeframe.

Once it realizes that a certain number of frames have passed without the other task 'reporting in', it knows something is wrong. You don't really need mutual exclusivity or interrupts to maintain this, so long as you can ensure that the 'task monitor' thread is of elevated priority.

There's no metrics available on 'how much' of the other tasks available cycle was consumed before it was completed, but that could be added fairly trivially.

I work with several systems like this, if the 'tried and true' notion is of any value.
  #3   Spotlight this post!  
Unread 08-09-2012, 10:39 PM
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 5,851
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: runtime thread execution monitoring (identify cause of hung code)

Quote:
Originally Posted by Todd View Post
If the High priority task, when it goes to set the bits notices that they haven't been flipped back, then it knows that the low priority task has not run in that timeframe.

Once it realizes that a certain number of frames have passed without the other task 'reporting in', it knows something is wrong.
I like that approach Todd.

Has anybody else used this or something similar, and if so, did it prove helpful?


  #4   Spotlight this post!  
Unread 08-10-2012, 06:23 AM
Greg McKaskle Greg McKaskle is offline
Registered User
no team (Team NI)
 
Join Date: Apr 2008
Rookie Year: 2008
Location: Austin, TX
Posts: 3,892
Greg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond reputeGreg McKaskle has a reputation beyond repute
Re: runtime thread execution monitoring (identify cause of hung code)

This bit system is similar to how the LV diagram does execution hilighting. The bits are cleared by the diagram before node code begins, and nodes flip their bit as they execute. But this data structure is problematic on multi-core architecture performance since you get cache coherency issues as each node sprinkle bits as a side-effect of their run. It is still correct, but the overhead of this tracing goes up significantly with parallel cores and especially parallel packages.

Also, can you give more detail on why you don't need a mutex? From your description I can see folks writing a read/modify/write piece of code with no mutex, and getting very confused. Are you using something like __sync_fetch_and_or()?

Greg McKaskle
  #5   Spotlight this post!  
Unread 08-10-2012, 07:47 AM
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 5,851
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: runtime thread execution monitoring (identify cause of hung code)

Quote:
Originally Posted by Greg McKaskle View Post
Also, can you give more detail on why you don't need a mutex? From your description I can see folks writing a read/modify/write piece of code with no mutex,
Yeah, I saw that too but forgot to comment about it.

Asserting a bit in a word is not an atomic operation. You have to read the word, or it with a mask, and write it back. You could get interrupted in the middle of the sequence.

Since memory is not an issue (for FRC), instead of flipping bits you could use an array of words*. Then there'd be no contention among the threads.


* the length of which is whatever is atomic for the processor and most efficiently handled by it



Last edited by Ether : 08-10-2012 at 08:06 AM.
  #6   Spotlight this post!  
Unread 08-10-2012, 05:22 PM
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 5,851
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: runtime thread execution monitoring (identify cause of hung code)


How about using a counter instead of a flag?

Code:
* start thread *
push_flags();
disable_interrupts();
threadID++;
pop_flags();
* put thread code here *
* end thread *
Monitoring thread reads and resets (zeros) counters. Keeps track of when each counter was last reset. Uses counter value and elapsed time (since last reset) to monitor thread health.


  #7   Spotlight this post!  
Unread 08-10-2012, 06:13 PM
Todd's Avatar
Todd Todd is offline
Software Engineer
FRC #1071 (Team Max)
Team Role: Mentor
 
Join Date: Feb 2005
Rookie Year: 2004
Location: Connecticut, Wolcott
Posts: 51
Todd is just really niceTodd is just really niceTodd is just really niceTodd is just really niceTodd is just really nice
You're both right, an atomic fetch and set is required at a minimum. Once you accept that, and expand to fetch and increment and fetch and decrement operations, there's a wide variety of metrics that can be prepared and transmitted later on to a dashboard or logged in a file.

I'm imagining pursuing this with functional global variables in labview with our team this year.
__________________
Mike Todd
Team MAX 1071
Sikorsky Aircraft Corporation - Embedded Software Engineer
Guy that does stuff
  #8   Spotlight this post!  
Unread 08-11-2012, 08:01 PM
brian.axelrod brian.axelrod is offline
Registered User
FRC #0846
 
Join Date: Oct 2010
Location: US
Posts: 9
brian.axelrod is an unknown quantity at this point
Re: runtime thread execution monitoring (identify cause of hung code)

If you're using C++, vxworks has a built in watchdog that can be useful.
You can use it as such.

You need to include wdLib.h
Code:
WDOG_ID watchDog = wdCreate();
wdStart(watchDog, ticks, OvertimeCallback, param);
Once the time specified in the second parameter (in ticks) expires the function specified by the OvertimeCallback is called. You can get the number of ticks per second with the
sysClkRateGet() function. If your code completes in time you cancel the call with
Code:
wdCancel(watchDog);
You can find the documentation for the vxworks watchdog here http://www-kryo.desy.de/documents/vx...ref/wdLib.html

There may be equivalents for the other programming languages.

If you are looking for more tools for evaluating profiling you can see the profiler we wrote. Our code is open source and posted on our website lynbrookrobotics.com
Closed Thread


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 11:14 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi