realtime runtime code throughput margin monitoring

[Ether]

I'm wondering how many teams add runtime throughput and scheduling monitoring in each thread (or parallel loop) of their practice or even competition deployed code.

Something like the following* thread start *
t0=system_microseconds();
if (t0-t1)>TBD1 err1++; //scheduling monitor
t1=t0;
* put thread code here *
if (system_microseconds()-t0)>TBD2 err2++; //thread throughput margin monitor
* thread end *"err1" and 'err2" could be displayed on the dashboard for each thread.

[Todd]

I'm wondering how many teams add runtime throughput and scheduling monitoring in each thread (or parallel loop) of their practice or even competition deployed code.

I'm planning on going through this exercise with our students next year. I meant to this year, but we fell short of time.

As far as I see it's the next step toward system stability after we work out a few more of our selective failure cases.

[plnyyanks]

We had this in our code, to a certain extent. We sent loop execution times to our custom dashboard, where they would be monitored and used to check that everything was running okay. If our main loop ever ran much over 20 ms per iteration, then we'd be worried (the vision loop ideally would be at about 100 ms). Code for reference. (http://www.chiefdelphi.com/media/papers/2679)

[Ether]

We sent loop execution times to our custom dashboard, where they would be monitored and used to check that everything was running okay.

Were they monitored by software in the dashboard which counted and flagged individual outliers, or did you mean they were monitored by the driver (whenever he had a chance to glance at the screen)?

[Greg McKaskle]

The LV template project contains a folder called Support Code. In it is a VI called Elapsed TImes.vi. Placing it into any loop or recurring code will result in a table of elapsed times being updated to calculate and show the period of the calls. The easiest way to view the times is to open the Elapsed Times panel. This is typically used for debugging and later deleted, but could be transmitted back to dashboard if a team wanted. I have no idea how many times it was used.

Greg McKaskle

[plnyyanks]

Were they monitored by software in the dashboard which counted and flagged individual outliers, or did you mean they were monitored by the driver (whenever he had a chance to glance at the screen)?


The latter. Although, while debugging, writing, and tuning code, they were watched very closely and we didn't let the code go into "production" until functionality was there, as well as low loop times/CPU usage.

Thankfully, it was never necessary to actively monitor these, although having software do it is a good idea for next year.

[Ether]

The LV template project contains a folder called Support Code. In it is a VI called Elapsed TImes.vi. Placing it into any loop or recurring code will result in a table of elapsed times being updated to calculate and show the period of the calls. The easiest way to view the times is to open the Elapsed Times panel. This is typically used for debugging and later deleted, but could be transmitted back to dashboard if a team wanted. I have no idea how many times it was used.

Greg, do you know roughly how much runtime overhead is incurred by this? e.g. Could it reasonably be put into all the parallel tasks in an FRC robot project?

[Ether]

while debugging, writing, and tuning code, they were watched very closely

How often was your dashboard screen display updated?

[plnyyanks]

How often was your dashboard screen display updated?


Not all that often (I don't remember, but maybe ~200 ms). Which, of course, now that I think about it, doesn't really help catch situations like you're thinking of... Anyway, it's a good idea to implement next year.

[Greg McKaskle]

I don't have an overhead number, but I have seen it raise the CPU. The code is in about six loops in the Driver Station and I saw no need to delete it. I decided not to leave it in the default robot code loops and chose to let teams do it for themselves. I hope to get around to adding some high-water mark and stats to it and perhaps I can find a way to leave it in all loops. Time will tell.

Greg McKaskle

[JamesTerm]

I'm wondering how many teams add runtime throughput and scheduling monitoring in each thread (or parallel loop) of their practice or even competition deployed code.
[/I]


I've taken a low-tech solution to this type of task... where I use printf statements that can be turned on by ftp of lua script... I get roughly 10-11ms iterations of a single thread (I cannot see the need for any more). All functions work with a delta time (double dTime_s) parameter to deal with uneven slices of time... this way I need not do anything fancy with the Sleep(10).

For the numbers I find interesting I use a separate homemade app to read the text and translate into a bmp file graph. So it's paritally real-time... partially post process.

[apalrd]

Greg, do you know roughly how much runtime overhead is incurred by this? e.g. Could it reasonably be put into all the parallel tasks in an FRC robot project?




We ran a slightly modified Elapsed Times in our robot code, and it wasn't the biggest cause of CPU usage (sorta...)

The biggest cause of CPU usage while not permanently deployed seems to be updating VI front panels. If you have enough inclusions of Elapsed Times, it seems to update the front panel array every time it changes (Every tick of every loop). If you create a slower loop somewhere (we ran ours at 75ms I think) to read the array of times from Elapsed Times, it is significantly less CPU hungry than simply opening Elapsed Times. We used the same loop to read the CPU load cluster and print it to the same debug screen, and that information (CPU usage by thread priority) was also extremely useful.

It was a very useful VI. We had it in about 10 or so tasks.

[Greg McKaskle]

If you create a slower loop ...

Good to know. The panel was the simplest way to see the data, but it will transmit all modifications to to the host each call, and the array is constantly changing. Maybe something like that will allow it to be in the framework by default.

Thanks.
Greg McKaskle