For teams having trouble downloading code

[Greg McKaskle]

The System Manager is indeed a very high level tool, very easy to use, but it is a single number, a single speedometer. Before going any further, make sure to go to the VIs tab and turn off the Monitor VI State feature. This is also useful to see what is running, but with so many VIs on the cRIO, this monitoring will raise the CPU number by quite a bit.

If/when you are ready for more detail, try using the Performance Profiler. The basics are to open it using Tools>>Profile>>Performance and Memory. You probably don't need to profile what is on the PC, so you can change the Targets to profile if you want.

The most trustworthy way to use it is to Start the Profiler, then run your app, then stop your app, then stop the profiler. Once you know what you are doing, you can use the profiler while your app is running, but until you know what is slow, this is the best way to start.

Once you stop the profile, you can click on a column to sort, and you can double click on a row to see what subVIs were called by and contributed to the VI time. Feel free to post profile images or data if you want some input.

Note that the profiler only reports VIs that started and finished while the profiler was on. Also note that the profiler works at the level of the VI. It will not tell you which loop or node within the VI took time unless that node is a VI.

Greg McKaskle

[tsa256]

My team has also had the exact same problem's regarding the downloading and deploying. And we are currently using the dip switch method at the moment. But during our two days of troubleshooting the cRIO way got a vast amount of data points, but due to the inconsistency have no idea what to do.

These data points include all the above symptoms described, but in addition we noticed some very inconsistent code speed. Sometimes we reboot the robot and things will just lag, and this may not seem like anything, but when we power cycled it the code ran at a normal speed. Also, when attempting to run the code as startup while there is currently code on the cRIO it fails about 99% of the time. When it fails and the robot reboots, the status lights on the cRIO blink orange 4 times. Further research on the NI website showed that this can be due to "The controller software has crashed twice without rebooting or cycling power between crashes. This usually occurs when the controller runs out of memory. Review your RT VI and check the controller memory usage. Modify the VI as necessary to solve the memory usage issue." Then thought that is was a coding problem. As a troubleshooting step we re-imaged and then attempted to run as startup the FIRST default code. As we expected the default code deployed the target settings successfully, and that worked flawlessly. Then we attempted to download our code. This failed, which surprised me because if something on the cRIO code related was using too much memory during the deployment why would it fail when deploying out code over the FIRST default code. Our isn't that modified from the FRC structure. Basically what we've done is read drive encoders in a 20ms loop then send those to another 50ms loop. The 50ms loop executes some calculations depending on the global variables sent into and out of the loop. e.g joystick, I/O and encoder values. It then drives the motors within the same 20ms loop. Furthermore, more complex code has been downloaded to the robot with 100% success about 2 weeks earlier. Which would point away from a code problem. Also when running the default FIRST code we get constant watchdog errors when disabling and enabling the robot. Another instance of these bugs are sometimes one of the PWM outputs being controlled by a PID loop will drop out. But this only happens when the process variable of the encoder reaches a specific point. To diagnose the problem I coded a emulation of that control loop, there was indeed one point on the code were the PID loop seamed to just stop outputting updated error. After further investigation this point where it drops out is also a cross over point for the encoder, e.g. The encoder jumps from -2.5 to 2.5. I'm am unsure if this would cause an error, or the PID loop to just stop working. The final data point we've gotten is that sometimes changing one number by .1 or some small value will allow successful deployment.

Also, I was wondering if a MAX reformat would be of order. As far as I know this would allow us to reconfigure the cRIO from it's default off the shelf state. Would there be any repercussions, besides having to reconfigure the cRIO just like it was just shipped? If not, we could try running the default code twice. Build run as startup, and then determine if it truly is the code or something else?

Finally, I've heard of the idea to turn of the FTP, HTTP server, or the error logging on the cRIO. Does anyone have any wisdom about this. I've heard conflicting views, and does it actually make any noticeable difference?


Unfortunately at this point to my knowledge it's like finding hay in a needle stack. Any help would be greatly appreciated.

[Greg McKaskle]

You indeed have lots of observations, and I can't make sense of all of them either.

I'd start by turning off the error logging -- if using LV. To do so, open ErrorsGlobal.vi, drop the global icon into the diagram of Begin.vi and set it to FALSE. Turning off error logging will not hurt anything, as unhandled errors are still being sent to the DS and logged. This global should have been off all along and was an oversight.

While your cRIO is running your code, open the Tools>>Real Time>>System Manager and observe the memory usage over time and the CPU usage over time.

Another thing to point out is that the VIs tab is another way to selectively terminate VIs. It is likely to be faster to kill the Robot Main and deploy rather than flip the DIP, reboot, deploy, flip, reboot.

Greg McKaskle

[tsa256]

Thanks I appreciate the procedure for turning off the error logging.We currently have the dip switch procedure down but I feel like it just bandaging the problem, and for future knowledge I'd like to know exactly what the problem is so we can avoid or quickly fix it. I'm currently unable to test the error logging off until tonight but I'll post back with the results of the deployment and if possible get a CPU and memory usage. Does anybody know what the standard CPU usage is on a consistently working machine?

In addition would anyone advise a full system format using the MAX tool. This would provide a low level format and a clean slate. Am I correct?


One last thing, I doubt this would affect it but I run out of places to look, would changing the loop timings in periodic tasks to both 20ms have any effect on the thread scheduling, or watchdog, perhaps it's using to much memory or periodically stopping a process to run another deterministic process. Anybody have any ideas? If it is a processing issue, any tips on making code more efficient?

Once again thanks.