Worried about high CPU usage in CRIO

[billbo911]

Originally Posted by DonRotolo

A 400 MHz Power PC and we're maxing it out. Incredible.

Agreed 100%!

If the basic code is already running at 65%, couldn't that indicate there is a flaw in the basic control code?
I don't recall any previous years' code acting this way.
Don't we have tools that could point us to where the biggest users of CPU cycles are? Something akin to "Task Manager" in Windows?

[Chris_Elston]

Ok, I think I have attracted all the "power users" of Labview at CD in one post. "ya'll" are scaring me with your comments....

I just got back helping a rookie team with a new CRIO-II. We downloaded the default code in that CRIO-II and it was running 40-45% with the default code. We only have two of the CRIO-I. Maybe I'll order a CRIO-II tomorrow just to gain another 20%...

Maybe I should post this in the NI community and hook up with a Labview engineer over there and understand if we are doing something wrong or if this is fact of the matter...

[DominickC]

Originally Posted by Chris_Elston

We only have two of the CRIO-I. Maybe I'll order a CRIO-II tomorrow just to gain another 20%...

Wait, so you can string together multiple cRIO II's to get more processing power?

[DonRotolo]

One man's flaw is another man's feature.

You know, I asked our programming mentor that exact question Friday. He said that if I could find the Ctl, Alt and Del keys on the cRio I could access the Program Manager...

[Joe Ross]

Originally Posted by Chris_Elston

Ok, I think I have attracted all the "power users" of Labview at CD in one post. "ya'll" are scaring me with your comments....

You should get really worried when Mark McLeod shows up.

Originally Posted by billbo911

Agreed 100%!

If the basic code is already running at 65%, couldn't that indicate there is a flaw in the basic control code?
I don't recall any previous years' code acting this way.
Don't we have tools that could point us to where the biggest users of CPU cycles are? Something akin to "Task Manager" in Windows?

We've struggled with high usage since the beginning. I wonder how much of this is now it's much easier to see the CPU% on the driver station, so now more people are noticing it. Also, all the extra monitoring unfortunately also takes more processor time. Unfortunately, I didn't look at the utilization for the default project in previous years, and now that the LabVIEW license expired, I'm not sure it's possible.

There's a few things you can do to track down high usage. In the Default project, there is a VI called Elapsed Times. You can drop it into each loop and wire in a name, and it will keep track of how long it takes between calls of that VI. This can help track down slow loops. You can also go to Tools -> Profile -> Performance and Memory for NI's equivalent of Task Manager.

[Chris_Elston]

Originally Posted by Joe Ross

I wonder how much of this is now it's much easier to see the CPU% on the driver station, so now more people are noticing it.

You may have an excellent point there, except the watch dog errors have always been there. In the case where 100% CPU is seen, the watch dog error fill the driver station diagnostic window. If in the past, without the CPU chart, we would have seen the same watch dog errors fill the screen I assume due to high CPU usage, even though we could not see the CPU usage in year previous.

My logic is, if we did not see any watch dog errors in 2009, 2010, and 2011, then I assume maybe we did not have a MAX'ed out CPU Usage. This year, we see the CPU usage hit 100%, then we see the watch dog errors fill the screen, and teleop disables, and the robot shudders to a stop when we have too much loaded in the CRIO. (Which bty the way isn't much code at all... compared to the past robots. And if you wanna see the past robots, click the link at the bottom of the screen to the repository. The only time we used vision was 2009 and that robot was fine.

BTY, that's a great tool who ever designed this CPU, latency and charts, thanks for giving us the chart trends to see this information....who ever you are. Thank you.

[wireties]

There is a function called "spy" which one can run on the cRIO console. It will print CPU usage by task every 10 seconds. There is a problem with the spy utility though, it uses the auxiliary clock library in the OS to profile the system and I'm not sure if NI uses that timer/clock library for anything else - next time I am in our lab I will check. There is also a remote display of the nearly the same information when using Workbench in debug mode.

The watchdog goes off if it does not get "petted" regularly and the FRC comms code interprets this as a dangerous condition (thus the disabling of motors etc). 100% CPU usage is not a good sign but it does not automatically mean something is wrong. If the watchdog is going off you could being doing too much work serially (one right after another) in between messages from the DS. Try parallelizing your activities and prioritizing the comms with the DS. The watchdog alarms should go away and you'll be giving the camera all the "left over" time. Then slow down and/or simplify the camera code till utilization drops just below 100%.

HTH

[Ether]

Originally Posted by wireties

The watchdog goes off if it does not get "petted" regularly

ASPCA would be happy to know you are petting your watchdog rather than kicking it.

[DonRotolo]

Originally Posted by Chris_Elston

Ok, I think I have attracted all the "power users" of Labview at CD in one post.

My role here is to balance that equation. What's the opposite of "power user?"

Originally Posted by DominickC

Wait, so you can string together multiple cRIO II's to get more processing power?

Yes, just as you can put a laptop (or even a desktop, I suppose) onto the robot as an auxiliary processor, under [R65]. But remember [R52] applies here as well.

[Thad House]

One thing i have noticed when running the code is that sending the data to the dashboard using the default code takes up alot of CPU resources. It the past when i have deleted that it freed up between 10 and 20% of the cpu resources. I havent tried it with this years code or the new Crio's. I will try this tomorrow when i get access to our robots.

[wireties]

Originally Posted by sst.thad

One thing i have noticed when running the code is that sending the data to the dashboard using the default code takes up alot of CPU resources. It the past when i have deleted that it freed up between 10 and 20% of the cpu resources. I havent tried it with this years code or the new Crio's. I will try this tomorrow when i get access to our robots.

We take over that default operation with our own dashboard sending code and send the data less often, more often than 10Hz is of questionable value

[Thad House]

Originally Posted by wireties

We take over that default operation with our own dashboard sending code and send the data less often, more often than 10Hz is of questionable value

Last year we had so little code we sent it every teleop period, and it worked, but this year i know that will not be possible because of the amount of stuff we have to have. We will have to test how often we can send our dashboard, or we might not have it at all.

[SuperS_5]

Adding some code to my test project, when I was demonstrating for a local team, I kept killing the cRIO. (BTW the DoS bug in the network stack still exists.) I had to add some careful performance controls in my code to keep the CPU utilization down. (Partially my fault to begin with.) I was running between 65-75% CPU on the cRIO with nearly default code.

Previous years have not be a whole lot better, and normally saw these utilization numbers for most LabVIEW projects. The Vision loop was the worse, normally consuming whatever was remaining of the cRIO. The performance monitoring in LabVIEW is very useful in tracking down problems. A built project, running at start-up should take a bit less resources then just hitting the run button, since it is not running in debug mode.

PS. During my testing the other night, I saw some interesting metrics. I will have to dive into it tonight.

[DominickC]

Now, I've placed this code within Periodic Tasks.vi. There are two more case structures similar to the two visible, and I've moved the axis value to Teleop.vi. Today, we were getting some infrequent watchdog errors which shut down our comms with the cRIO.

Just by looking at this image, is there a way to streamline the code? I had tried to do something similar to the second image with just the single case structure visible, wired to a joystick button. When the structure was false, it set motor outputs to 0. No matter what I did, it wouldn't work. (I followed it in debug mode, and it appeared that the command to set the motor output to -1 was being triggered, however nothing happened on our jags.)

Is there a conflict between the 500 millisecond timing and the 100 millisecond timing within the while loop?

[Greg McKaskle]

Now, I've placed this code within ...

The image shows a single sleep of 100 and then in parallel, two loops that do unlimited I/O calls.

I'm assuming that the code shown is in the Periodic task with 100ms sleep. The issue is that when one of those buttons are pressed, the inner loop goes for 500ms with no sleep. That would likely cause a watchdog or other issue with CPU usage. After the inner loop completes, things would go back to normal.

If you place a 20ms delay within the loop, that will improve the CPU usage, but you will still have an issue in that for 500ms, the outer loop cannot run. I'd think that you can make a loop that starts on a message, like a notifier, then runs for 500ms and waits again and is independent of the others. You can then send the notifier from the teleop or other loops.

Greg McKaskle