Worried about high CPU usage in CRIO

[Joe Ross]

Originally Posted by Chris_Elston

Ok, I think I have attracted all the "power users" of Labview at CD in one post. "ya'll" are scaring me with your comments....

You should get really worried when Mark McLeod shows up.

Originally Posted by billbo911

Agreed 100%!

If the basic code is already running at 65%, couldn't that indicate there is a flaw in the basic control code?
I don't recall any previous years' code acting this way.
Don't we have tools that could point us to where the biggest users of CPU cycles are? Something akin to "Task Manager" in Windows?

We've struggled with high usage since the beginning. I wonder how much of this is now it's much easier to see the CPU% on the driver station, so now more people are noticing it. Also, all the extra monitoring unfortunately also takes more processor time. Unfortunately, I didn't look at the utilization for the default project in previous years, and now that the LabVIEW license expired, I'm not sure it's possible.

There's a few things you can do to track down high usage. In the Default project, there is a VI called Elapsed Times. You can drop it into each loop and wire in a name, and it will keep track of how long it takes between calls of that VI. This can help track down slow loops. You can also go to Tools -> Profile -> Performance and Memory for NI's equivalent of Task Manager.

[Chris_Elston]

Originally Posted by Joe Ross

I wonder how much of this is now it's much easier to see the CPU% on the driver station, so now more people are noticing it.

You may have an excellent point there, except the watch dog errors have always been there. In the case where 100% CPU is seen, the watch dog error fill the driver station diagnostic window. If in the past, without the CPU chart, we would have seen the same watch dog errors fill the screen I assume due to high CPU usage, even though we could not see the CPU usage in year previous.

My logic is, if we did not see any watch dog errors in 2009, 2010, and 2011, then I assume maybe we did not have a MAX'ed out CPU Usage. This year, we see the CPU usage hit 100%, then we see the watch dog errors fill the screen, and teleop disables, and the robot shudders to a stop when we have too much loaded in the CRIO. (Which bty the way isn't much code at all... compared to the past robots. And if you wanna see the past robots, click the link at the bottom of the screen to the repository. The only time we used vision was 2009 and that robot was fine.

BTY, that's a great tool who ever designed this CPU, latency and charts, thanks for giving us the chart trends to see this information....who ever you are. Thank you.

[wireties]

There is a function called "spy" which one can run on the cRIO console. It will print CPU usage by task every 10 seconds. There is a problem with the spy utility though, it uses the auxiliary clock library in the OS to profile the system and I'm not sure if NI uses that timer/clock library for anything else - next time I am in our lab I will check. There is also a remote display of the nearly the same information when using Workbench in debug mode.

The watchdog goes off if it does not get "petted" regularly and the FRC comms code interprets this as a dangerous condition (thus the disabling of motors etc). 100% CPU usage is not a good sign but it does not automatically mean something is wrong. If the watchdog is going off you could being doing too much work serially (one right after another) in between messages from the DS. Try parallelizing your activities and prioritizing the comms with the DS. The watchdog alarms should go away and you'll be giving the camera all the "left over" time. Then slow down and/or simplify the camera code till utilization drops just below 100%.

HTH

[Ether]

Originally Posted by wireties

The watchdog goes off if it does not get "petted" regularly

ASPCA would be happy to know you are petting your watchdog rather than kicking it.

[DonRotolo]

Originally Posted by Chris_Elston

Ok, I think I have attracted all the "power users" of Labview at CD in one post.

My role here is to balance that equation. What's the opposite of "power user?"

Originally Posted by DominickC

Wait, so you can string together multiple cRIO II's to get more processing power?

Yes, just as you can put a laptop (or even a desktop, I suppose) onto the robot as an auxiliary processor, under [R65]. But remember [R52] applies here as well.

[Thad House]

One thing i have noticed when running the code is that sending the data to the dashboard using the default code takes up alot of CPU resources. It the past when i have deleted that it freed up between 10 and 20% of the cpu resources. I havent tried it with this years code or the new Crio's. I will try this tomorrow when i get access to our robots.

[wireties]

Originally Posted by sst.thad

One thing i have noticed when running the code is that sending the data to the dashboard using the default code takes up alot of CPU resources. It the past when i have deleted that it freed up between 10 and 20% of the cpu resources. I havent tried it with this years code or the new Crio's. I will try this tomorrow when i get access to our robots.

We take over that default operation with our own dashboard sending code and send the data less often, more often than 10Hz is of questionable value

[Thad House]

Originally Posted by wireties

We take over that default operation with our own dashboard sending code and send the data less often, more often than 10Hz is of questionable value

Last year we had so little code we sent it every teleop period, and it worked, but this year i know that will not be possible because of the amount of stuff we have to have. We will have to test how often we can send our dashboard, or we might not have it at all.

[SuperS_5]

Adding some code to my test project, when I was demonstrating for a local team, I kept killing the cRIO. (BTW the DoS bug in the network stack still exists.) I had to add some careful performance controls in my code to keep the CPU utilization down. (Partially my fault to begin with.) I was running between 65-75% CPU on the cRIO with nearly default code.

Previous years have not be a whole lot better, and normally saw these utilization numbers for most LabVIEW projects. The Vision loop was the worse, normally consuming whatever was remaining of the cRIO. The performance monitoring in LabVIEW is very useful in tracking down problems. A built project, running at start-up should take a bit less resources then just hitting the run button, since it is not running in debug mode.

PS. During my testing the other night, I saw some interesting metrics. I will have to dive into it tonight.

[DominickC]

Now, I've placed this code within Periodic Tasks.vi. There are two more case structures similar to the two visible, and I've moved the axis value to Teleop.vi. Today, we were getting some infrequent watchdog errors which shut down our comms with the cRIO.

Just by looking at this image, is there a way to streamline the code? I had tried to do something similar to the second image with just the single case structure visible, wired to a joystick button. When the structure was false, it set motor outputs to 0. No matter what I did, it wouldn't work. (I followed it in debug mode, and it appeared that the command to set the motor output to -1 was being triggered, however nothing happened on our jags.)

Is there a conflict between the 500 millisecond timing and the 100 millisecond timing within the while loop?

[Greg McKaskle]

Now, I've placed this code within ...

The image shows a single sleep of 100 and then in parallel, two loops that do unlimited I/O calls.

I'm assuming that the code shown is in the Periodic task with 100ms sleep. The issue is that when one of those buttons are pressed, the inner loop goes for 500ms with no sleep. That would likely cause a watchdog or other issue with CPU usage. After the inner loop completes, things would go back to normal.

If you place a 20ms delay within the loop, that will improve the CPU usage, but you will still have an issue in that for 500ms, the outer loop cannot run. I'd think that you can make a loop that starts on a message, like a notifier, then runs for 500ms and waits again and is independent of the others. You can then send the notifier from the teleop or other loops.

Greg McKaskle

[Greg McKaskle]

I was able to look at the usage of the framework code a bit today.

I built a new framework, ran the code from the run button, and watched the CPU usage from the DS tab.

The disabled robot default usage on an 8-slot was about 25%. I ran the default teleop and the usage did not change. I ran the default auto, and the usage jumped to just under 50%. Odd, but it turns out that when the auto doesn't update the motors, they go into safety update, and I'm assuming the error messaging eats the CPU. I need to take more measurements tomorrow to be sure that the errors are the issue. But if the auto sets the motors, even setting them to 0 every 20 ms or so, the CPU usage stays at 25%.

Since this was default code, this was without video enabled, with no panels open except for Robot Main. The 8-slot was formatted without net console or CAN enabled.

Can anyone with higher CPU usage provide differences or code to go on?

Greg McKaskle

[Ether]

@DominickC: If you haven't already seen these two posts, they're worth reading:

http://www.chiefdelphi.com/forums/sh...96&postcount=3

http://www.chiefdelphi.com/forums/sh...21&postcount=6

[EricVanWyk]

Having a high usage isn't necessarily bad. Well written code can consume 100% of the CPU but be squishy enough to not get in the way. For example, let vision consume whatever cycles happen to be left over, but prioritize it correctly so it doesn't consume cycles that aren't left over.

Also, it is really easy to rail the processor - any while loop without a delay will take care of your spare cycles immediately.

Do any of the power users have a favorite way to implement Dominick's code? If it doesn't need to run in parallel with itself, I typically do that with a single (Enum)Notifier with multiple loops.

Let me temper that with an admission: I haven't gone into serious depth with FRC LabVIEW since I started working at NI. Most of my FIRST LV experience is in FTC/FLL, and most of that was so far under the hood that I didn't know what road we were driving on...

Originally Posted by DominickC

Now, I've placed this code within Periodic Tasks.vi. There are two more case structures similar to the two visible, and I've moved the axis value to Teleop.vi. Today, we were getting some infrequent watchdog errors which shut down our comms with the cRIO.

Just by looking at this image, is there a way to streamline the code? I had tried to do something similar to the second image with just the single case structure visible, wired to a joystick button. When the structure was false, it set motor outputs to 0. No matter what I did, it wouldn't work. (I followed it in debug mode, and it appeared that the command to set the motor output to -1 was being triggered, however nothing happened on our jags.)

Is there a conflict between the 500 millisecond timing and the 100 millisecond timing within the while loop?

I found a few spots in your code that you might want to look at. Once you fix the first error, I think the others might not be relevant. I mention them just in case.

Both loops are busy loops - they will consume all available CPU time to check to see if 500 milliseconds have elapsed and to write the output. If there isn't anything else contending, you could just write once, wait 500, write again.

Both loops are fully blocking - the owning VI has to wait 500 milliseconds to continue. Was this intended?

Both loops pull the motor reference every iteration. If you are using the same reference often, you don't have to pull it each time.

[DominickC]

@Greg - Ah, now I understand. I'll see what I can do here.

@Ether - I have read the first post before, however reading it again now (with a deeper knowledge of LV) I feel like I understand it better. If I wanted to make a "state machine", I would get the Joystick button values and set a local/global variable, and call such a variable within the code requiring the value from the variable, yes?

@EricVanWyk - Is there a way within LV to prioritize code?

I did not intend for my loops to be blocking (at least not in this situation). How could I write a value (such as a motor output) then rewrite a value 500ms later without it becoming a blocking loop?

I'm going to try to write the code here from scratch.