Monitor cRIO CPU level

[keehun]

Is there any way of monitoring the CPU usage? I wouldn't mind any methodology as long as I could read it while driving it.

Just out of curiosity I guess, and planning how much processing "room" I have left or if I need to start optimizing something. It would hate to have developed really quality things by week 5 but then run out of "processing room" and rush to get that in the clock cycle and everything else gets screwed up.

[keen101]

are you really worried about it? It's a huge upgrade from the previous microcontrollers. I mean come on, the cRIO is a (300Mhz?) Power PC based computer! Heck it could run embedded Linux! I actually think it runs vxworks on UNIX.

In my opinion you shouldn't really have to worry about it. It's practically a computer.

But, even though it's probably not needed at all, it would still be cool.

[keehun]

It runs vxworks on UNIX.

Yeah, before the season I was joking how we should run OSX on this thing. ;-)

In my opinion you shouldn't really have to worry about it. It's practically a computer. But, even though it's probably not needed at all, it would still be cool.

Perhaps it's not needed. But I definitely want it for the cool-factor. But seriously I just want to watch it for any spikes, repeated patterns, etc. Who knows if that while loop is just killing the CPU

[synth3tk]

Just to reinforce what keen stated, the old controllers were significantly "weaker" than the cRIOs.

As far as I know, people didn't have issues with processing limits on the old IFIs, so I don't think these will be much of a problem.

[nathanww]

The quick way(and the only way I can think off off the top of my head) is to create a thread that basically just monitors how many times it can loop in a given mount of time. Less should mean more CPU load

[wireties]

It runs vxWorks and VxWorks is an industrial real-time operating system. It does not run "VxWorks on Unix". VxWorks has a task-based profiler called 'spy'. I have not checked to see if FIRST/NI/WPI included it in the kernel image they gave us. If this feature is in the kernel you can see the spy output from the command line or from Workbench.

HTH

[Greg McKaskle]

spy and similar serial terminal tools will give you a performance dump.

If using LV, it is pretty easy to open the Tools>>RealTime>>Real Time System Monitor. Unfortunately, its default settings have lots of overhead, so hit the other tab and turn of VI state. On the original tab, turn on performance, either leave disk on or off, doesn't matter much. Now hit the start button and observe the chart.

Greg McKaskle

[Tom Line]

Thank you! We were looking at doing some additional and rather extensive image processing, and I wasn't sure how the CPU would hold up.

It there a threshold of CPU usage we should strive to stay below?

[Alan Anderson]

It there a threshold of CPU usage we should strive to stay below?

Don't go over 100%. ;)

Seriously, unused CPU cycles are of no value. There's no benefit to limiting yourself to only a fraction of the available processing power.

[AustinSchuh]

Don't go over 100%. ;)

Seriously, unused CPU cycles are of no value. There's no benefit to limiting yourself to only a fraction of the available processing power.

I'm not sure how good the vxWorks scheduler is, but wouldn't you want to leave a couple of percent left so that the processes that need to be responsive, like the one that talks to the DS, can be scheduled easier and on time?

[Mike Soukup]

I'm not sure how good the vxWorks scheduler is, but wouldn't you want to leave a couple of percent left so that the processes that need to be responsive, like the one that talks to the DS, can be scheduled easier and on time?
VxWorks has an excellent scheduler. If the controller was designed properly (and I assume it was, I just haven't looked at the relative priorities of the tasks) the critical tasks that we rely on will all have appropriate priorities. VxWorks guarantees that if a higher priority task wants to run, it will run before a lower priority task.

[Uberbots]

are you really worried about it? It's a huge upgrade from the previous microcontrollers. I mean come on, the cRIO is a (300Mhz?) Power PC based computer! Heck it could run embedded Linux! I actually think it runs vxworks on UNIX.

In my opinion you shouldn't really have to worry about it. It's practically a computer.

But, even though it's probably not needed at all, it would still be cool.

its 400Mhz

but that also means we can throw much more complex tasks (image processing, array crunching, super-linear-time operations over big data sets) to choke the processor with.

[Kingofl337]

I think it's very important to know the available processing power. Currently vision in Windriver is being done at 10FPS if we have more available processing power when the robot is programmed, we could increase the FPS rate for more fluid tracking. The fact the cRio is 400MHz is moot because it's doing alot more work already then the old system. It's running an OS, it's running a network stack, communications protocol, and doing vision.

[Tanner]

Lab View has a option to monitor the CPU usage and memory usage on the cRIO.

Tools -> Real-Time Module -> System Manager

You can do it, but there's really not much of a reason to do it as stated by others.

-Tanner

[wireties]

"The fact the cRio is 400MHz is moot" - Well, its not exactly moot. The cRIO is doing a lot more work but it is many many times faster. Its a 32 bit machine with primary and secondary caches running at 400Mhz with a floating point co-processor. It could be nearly 200X faster under some circumstances, more if you need floating point.

Running VxWorks is not like running windoze, its an industrial real-time operating system. The network stack is an extra load but the old processor was still handling application level comms with the driver station. And the work done by the network stack in VxWorks can be prioritized relative to any other activity (but interrupts).

The vision work is this game is a pretty simple algorithm. I don't think it will begin to stress the cpu bandwidth of the cRIO.

[RyanCahoon]

Its a 32 bit machine with primary and secondary caches running at 400Mhz with a floating point co-processor. It could be nearly 200X faster under some circumstances, more if you need floating point.


Just a question from the curious: does anyone know if the cRios has a vector (http://en.wikipedia.org/wiki/Vector_processor)/SIMD (http://en.wikipedia.org/wiki/SIMD) ISA (http://en.wikipedia.org/wiki/Instruction_set)?

[Greg McKaskle]

The PPC is an MPC5200, part of the 603e family. That means no altivec, no vector, just a workhorse industrial temp PPC with an FPU.

As for monitoring CPU utilization, I tend to peek at it after any major changes. CPU usage is not as important as correct operation, crashes, or bugs, but if you never look at it, you will almost certainly be surprised at how things creep in. I treat it like a thermometer. If you glance at it now and then you learn a bit more about how things contribute, and how you want to balance the system. And one of the implications of RT programming is that if it isn't well timed, it is wrong. In a batch system, it doesn't really matter when the results arrive. If you are controlling something, timing is far more important.

To put things into perspective, it is actually pretty easy to use up 100% of any processor, just write a loop with nothing to throttle it, and before you know it the CPU is racing full speed. This is usually fine, in fact it is what you want. The point where this isn't good is when there are tasks the CPU isn't getting to fast enough.

As an example, if you are processing images, processing DS packets, running control loops for motors and manipulator, and sending data back to the dashboard, you will probably find that it all works fast enough. BUT, if your control loops don't have enough oomph, or if your vision frame rate is too low, it may be time to look at performance and tune things up a bit. I'm going to talk about this from the LV perspective because that is what I know best, but the approach is generally the same for C/C++, but the tools and steps will be different.

The first thing I do is open up the System Monitor from Tools>>Real Time>>System Monitor. As with any monitor, there is overhead to measuring the CPU, and in order to get a better measurement, it is pretty important to click on the VIs tab and turn off the Track VI States option. Because there are so many VIs in WPILib, this will add lots of artificial overhead when monitoring, and all this will tell you is that three of four VIs are running in parallel, sort of like tasks. So you can look at this if curious, but you should generally turn it off. Now switch back to the Resources tab, turn on the Track CPU Usage option and Start.

At this point you may get a message about turning on the server. If you've used the FRC template, this should already be done for you, otherwise follow the directions to enable it, reboot the controller, then try again.

The upper trace for disk usage doesn't cost much, but won't be a concern unless you are storing LOTS of data on the disk/flash. The lower trace will take one second samples and show you the average usage. You can see the overhead before you run your program, you can see the usage of your disabled robot, your autonomous, your different teleop operations. As long as the operation is relatively similar for one or two seconds, you can get a feel for the expense of various tasks. I generally jot a handful of numbers in my notebook along with explanations and dates/times so later when things are different I can more easily refer to how they used to run.

As mentioned, 100% CPU usage isn't necessarily bad. If all of the tasks are being handled well enough, that is the important thing. What 100% means is that you probably can't add many more CPU intensive tasks without taking away from the current things.

Sooner or later you will want to speed stuff up, and the System monitor doesn't give enough info. I won't go into the tools for that because this post is already huge. I'll wait for someone to ask.

Greg McKaskle

[buchanan]

> I'll wait for someone to ask.

OK, I'll ask :)

We're interested in tracking CPU load because we'd like to apply any otherwise unoccupied CPU horsepower to sampling and smoothing noisy sensor inputs. It's easy enough to write a Thread (we're using Java) to do this as fast as it can, but we don't want to risk starving more important tasks.

It's a lot easier to tune things like this if there's an easy way to watch CPU loads during development. Any hints?

[Greg McKaskle]

If the noisy sensors can be made nonnoisy with better wiring or contacts, that is probably preferable. A second option is to use the averaging on the Analog object -- that way it is done by the FPGA and doesn't affect the CPU at all.

If you want something more fancy, you can do the conditioning when read and you probably don't need a thread running in parallel.

If you still use the thread, just make sure to add a delay to it so that it doesn't run hundreds of times faster than needed.

Greg McKaskle