For teams having trouble downloading code

[Greg McKaskle]

I have never personally had to use the No App switch. If it is helping you to get things deployed, then by all means use it.

On the other hand, I suspect the need for using this indicates that the code is using so much CPU that the protocol methods are being starved.

If using LV, you should turn off the global for disk logging of errors. Doug Norman posted good directions for doing this. It involves changing one global variable and it is a very good thing for LV teams to do.

The next thing a LV team might want to do is to open Tools>>Real Time>>System Manager. This is basically the task manager for the LV cRIO. To get an accurate reading of CPU usage, click to the VI tab and turn off the Track VI States. Then click back to the first tab and Start. The bottom chart shows CPU usage and you should help understand what the cRIO is having to perform. Using this, you can run different modes, turn off different features, even comment out some code, and use the task manager to learn what the CPU cost is of different features.

Feel free to post questions if you have issues shrinking the CPU usage. My assumption is that by cutting the CPU usage even a bit, you will no longer need to use the No App button.

Sorry I don't have as much detail for nonLV teams. The profiling tools exist, I'm just not as familiar with them. I suspect that the download issues would be caused by the same sorts of issues.

Greg McKaskle

[apalrd]

@rwood359
That is common. There is just an old version of code on the cRio.

@Greg:
So, you are saying the cRio isn't powerful enough? I will turn off the global logging and see if it helps. I'm not even using any camera-related things (no camera on dashboard or target tracking) so I can't imagine how many problems teams using it must be having. It would have been nice to know about this back in week 2 when we mounted the cRio way in the belly of the robot..

@Joe:
We must run as startup, since it has to load the code when we go on to the field. That LabVIEW RT dialog that says "Waiting for Real-time target" with the "stop waiting and disconnect" box during Run and Deploy had me thinking it had failed - when it hadn't. However, once I let it wait, sometimes it will actually fail and I will have to find a little screwdriver, reach way into the belly of the robot, flip the No App switch, try again, and flip the switch back.

I do not believe my code is overly processor-hungry, it has two threads processing PID loops running with 50hz (slightly faster than IFI processor) and a few other threads with open-loop controls running at 50hz or 20hz. My autonomous code has a single loop at 100hz (same speed as Victor updates), although it is not running when I am downloading. I have checked, and none of my VI's are running without waits. Last year we never had this problem. What has changed to cause this?

[rwood359]

Originally Posted by apalrd

@rwood359
That is common. There is just an old version of code on the cRio.

I understand that. The problem is that the new code will not load without re-imaging the cRIO or using the no app startup.

[Greg McKaskle]

Powerful is relative. As I said, I've never needed to use the switch. If the download fails, my assumption is that it is because the helper task that does the RT protocol, including downloading code, is not given enough CPU and is timing out. It could also be that it is failing for a reason unrelated to CPU.

Based upon the other posts you have made, it seems like something odd is definitely happening. Have you run the System Manager yet? I'm curious to see what it says, and in fact, you should be able to leave it up while you try to download. Even more informative would be to use the serial cable and the command line profiling.

I'd also be interested in running your project, looking at the performance, and determining why you are getting the dialog. I'll be leaving for the Dallas event tomorrow, so I may be slow getting to you, but you can post it here or PM and I'll give an email.

As for the processor power. Think of it as the immovable object/irresistible force conundrum. Computers do what you tell them, and the RT System Manager will help you figure out what it is being asked to do.

Greg McKaskle

[ayeckley]

Originally Posted by Greg McKaskle

The next thing a LV team might want to do is to open Tools>>Real Time>>System Manager. This is basically the task manager for the LV cRIO. To get an accurate reading of CPU usage, click to the VI tab and turn off the Track VI States. Then click back to the first tab and Start. The bottom chart shows CPU usage and you should help understand what the cRIO is having to perform. Using this, you can run different modes, turn off different features, even comment out some code, and use the task manager to learn what the CPU cost is of different features.

Does anyone have any benchmark data for what a "normal" processor load is? I've found that running the default code puts the cRIO CPU load at about 92%. I've found that running our "full featured" code (ten motors, four encoders, two analog inputs, lots of global variable use) the processor load is about 95%. Even shutting down most of the VIs doesn't seem to have a significant impact on processor load. With that level of insensitivity, I don't see a whole lot to be gained by using the System Manager as an optimization tool. Are there a lot of housekeeping tasks going on in the background that drive the CPU load so high?

[Greg McKaskle]

The System Manager is indeed a very high level tool, very easy to use, but it is a single number, a single speedometer. Before going any further, make sure to go to the VIs tab and turn off the Monitor VI State feature. This is also useful to see what is running, but with so many VIs on the cRIO, this monitoring will raise the CPU number by quite a bit.

If/when you are ready for more detail, try using the Performance Profiler. The basics are to open it using Tools>>Profile>>Performance and Memory. You probably don't need to profile what is on the PC, so you can change the Targets to profile if you want.

The most trustworthy way to use it is to Start the Profiler, then run your app, then stop your app, then stop the profiler. Once you know what you are doing, you can use the profiler while your app is running, but until you know what is slow, this is the best way to start.

Once you stop the profile, you can click on a column to sort, and you can double click on a row to see what subVIs were called by and contributed to the VI time. Feel free to post profile images or data if you want some input.

Note that the profiler only reports VIs that started and finished while the profiler was on. Also note that the profiler works at the level of the VI. It will not tell you which loop or node within the VI took time unless that node is a VI.

Greg McKaskle

[tsa256]

My team has also had the exact same problem's regarding the downloading and deploying. And we are currently using the dip switch method at the moment. But during our two days of troubleshooting the cRIO way got a vast amount of data points, but due to the inconsistency have no idea what to do.

These data points include all the above symptoms described, but in addition we noticed some very inconsistent code speed. Sometimes we reboot the robot and things will just lag, and this may not seem like anything, but when we power cycled it the code ran at a normal speed. Also, when attempting to run the code as startup while there is currently code on the cRIO it fails about 99% of the time. When it fails and the robot reboots, the status lights on the cRIO blink orange 4 times. Further research on the NI website showed that this can be due to "The controller software has crashed twice without rebooting or cycling power between crashes. This usually occurs when the controller runs out of memory. Review your RT VI and check the controller memory usage. Modify the VI as necessary to solve the memory usage issue." Then thought that is was a coding problem. As a troubleshooting step we re-imaged and then attempted to run as startup the FIRST default code. As we expected the default code deployed the target settings successfully, and that worked flawlessly. Then we attempted to download our code. This failed, which surprised me because if something on the cRIO code related was using too much memory during the deployment why would it fail when deploying out code over the FIRST default code. Our isn't that modified from the FRC structure. Basically what we've done is read drive encoders in a 20ms loop then send those to another 50ms loop. The 50ms loop executes some calculations depending on the global variables sent into and out of the loop. e.g joystick, I/O and encoder values. It then drives the motors within the same 20ms loop. Furthermore, more complex code has been downloaded to the robot with 100% success about 2 weeks earlier. Which would point away from a code problem. Also when running the default FIRST code we get constant watchdog errors when disabling and enabling the robot. Another instance of these bugs are sometimes one of the PWM outputs being controlled by a PID loop will drop out. But this only happens when the process variable of the encoder reaches a specific point. To diagnose the problem I coded a emulation of that control loop, there was indeed one point on the code were the PID loop seamed to just stop outputting updated error. After further investigation this point where it drops out is also a cross over point for the encoder, e.g. The encoder jumps from -2.5 to 2.5. I'm am unsure if this would cause an error, or the PID loop to just stop working. The final data point we've gotten is that sometimes changing one number by .1 or some small value will allow successful deployment.

Also, I was wondering if a MAX reformat would be of order. As far as I know this would allow us to reconfigure the cRIO from it's default off the shelf state. Would there be any repercussions, besides having to reconfigure the cRIO just like it was just shipped? If not, we could try running the default code twice. Build run as startup, and then determine if it truly is the code or something else?

Finally, I've heard of the idea to turn of the FTP, HTTP server, or the error logging on the cRIO. Does anyone have any wisdom about this. I've heard conflicting views, and does it actually make any noticeable difference?


Unfortunately at this point to my knowledge it's like finding hay in a needle stack. Any help would be greatly appreciated.

[Greg McKaskle]

You indeed have lots of observations, and I can't make sense of all of them either.

I'd start by turning off the error logging -- if using LV. To do so, open ErrorsGlobal.vi, drop the global icon into the diagram of Begin.vi and set it to FALSE. Turning off error logging will not hurt anything, as unhandled errors are still being sent to the DS and logged. This global should have been off all along and was an oversight.

While your cRIO is running your code, open the Tools>>Real Time>>System Manager and observe the memory usage over time and the CPU usage over time.

Another thing to point out is that the VIs tab is another way to selectively terminate VIs. It is likely to be faster to kill the Robot Main and deploy rather than flip the DIP, reboot, deploy, flip, reboot.

Greg McKaskle

[tsa256]

Thanks I appreciate the procedure for turning off the error logging.We currently have the dip switch procedure down but I feel like it just bandaging the problem, and for future knowledge I'd like to know exactly what the problem is so we can avoid or quickly fix it. I'm currently unable to test the error logging off until tonight but I'll post back with the results of the deployment and if possible get a CPU and memory usage. Does anybody know what the standard CPU usage is on a consistently working machine?

In addition would anyone advise a full system format using the MAX tool. This would provide a low level format and a clean slate. Am I correct?


One last thing, I doubt this would affect it but I run out of places to look, would changing the loop timings in periodic tasks to both 20ms have any effect on the thread scheduling, or watchdog, perhaps it's using to much memory or periodically stopping a process to run another deterministic process. Anybody have any ideas? If it is a processing issue, any tips on making code more efficient?

Once again thanks.

[ayeckley]

Originally Posted by tsa256

Does anybody know what the standard CPU usage is on a consistently working machine?

In my previous post, I reported 92-95% regardless of the complexity of the code. I'm not currently having any download issues (at present) although I do still see the occasional "Invoke Node" problem with BUILD (as discussed in the "Labview Build Errors" thread, and still a very open issue from my perspective).

Originally Posted by tsa256

One last thing, I doubt this would affect it but I run out of places to look, would changing the loop timings in periodic tasks to both 20ms have any effect on the thread scheduling, or watchdog, perhaps it's using to much memory or periodically stopping a process to run another deterministic process.

I've got all of our asynchronous loops (we don't use "periodic tasks.vi" per-se) throttled at 50ms, FWIW.

[Team#2057-Vegas]

AH! Yes! Thank you for finding this problem. We sent our robot with everything but the cRIO and rebuilt a robot within 2 hrs. It worked great and then all of a sudden problems formed. We have been trying to fix it for 3 weeks and the pressure is building with competition around the corner. We will test this tomorrow and post our results.

Thanks!

[Team#2057-Vegas]

Originally Posted by Team#2057-Vegas

AH! Yes! Thank you for finding this problem. We sent our robot with everything but the cRIO and rebuilt a robot within 2 hrs. It worked great and then all of a sudden problems formed. We have been trying to fix it for 3 weeks and the pressure is building with competition around the corner. We will test this tomorrow and post our results.

Thanks!

We have tried testing the possible fix 4 or 5 times so far today and nothing is working, we are going to continue trying different variations and things but anyone else have any ideas? It says we still have "No Robot Code"

We are working with Java this year and it worked fine up until a few weeks ago. Once again anyone have any ideas?

Here's what we've done:

We changed the battery, reimaged/formatted sucessfully, restarted, deployed successfully after flipping the switch to on, flipped it off, restarted and recycled the power.

Help ):

[Team#2057-Vegas]

Problem solved! We redid all the network configuration and found out our bridge was broken. We have purchased a new bridge and things are looking up and just in time! Las Vegas Regional 2010 starts in 2 days! Thank you all for your suggestions.