For teams having trouble downloading code

[lscime]

Toward the end of build season we started having trouble connecting to our cRIO. It turned out that when our cRIO booted and ran its deployed code it took up too much processor speed to run the code and connect to our programming computer properly. So we started flipping the "no app" dip switch on the cRIO every time we wished to download code to the cRIO and all of our connection issues were solved! Essentially, flipping this dip switch prevents the cRIO from running code when it boots preventing connection issues. (Of course you'll want to reset the switch before you go to play a match). I didn't want to post this until we had been to a competition and I was sure there wasn't anything wrong with our code.

If you are having similar issues this following procedure may help.
Good Luck FRC!

Luke

[rwood359]

Originally Posted by lscime

Toward the end of build season we started having trouble connecting to our cRIO. It turned out that when our cRIO booted and ran its deployed code it took up too much processor speed to run the code and connect to our programming computer properly. So we started flipping the "no app" dip switch on the cRIO every time we wished to download code to the cRIO and all of our connection issues were solved! Essentially, flipping this dip switch prevents the cRIO from running code when it boots preventing connection issues. (Of course you'll want to reset the switch before you go to play a match). I didn't want to post this until we had been to a competition and I was sure there wasn't anything wrong with our code.

We were having problems doing a "run as startup". I can't remember the error message, but we were getting several lines that said something to the effect that it couldn't complete with the current dialog. Clicking NEXT for each error cleared the message and sometimes would let the load complete. But usually it would not succeed with an error about failing to download configuration settings. The only way that we found to get around the problem was to re-image the cRIO.
Doug Norman from NI was at the AZ regional, looked at the problem and couldn't come up with a solution. He showed us the "no app" switch procedure. We now Set the "no app" switch, press "reset" on the cRIO, build, run as startup, reset the no app switch, and reset the cRIO. I made sure that all of the drive team knows to look for the switch in the wrong position and the coach carries a small screwdriver in the programming team forgets to reset the switch.

[Tom Line]

While we're talking about odd bugs - we get one where when we try to run (not permanently depoy) code, it will not do so. It will get to the point where it is trying to run the project file, and we will lose connection to the crio.

Our solution to this has been to format and leave the crio clean. Then we can run to our heart's content, and when we finally want to deploy permanently, we can do so.

The next time we want to "run", we do a reimage of the crio so that the code is all wiped, and go back to "running" the code rather than permanently deploying it.

[jhersh]

Originally Posted by Tom Line

Our solution to this has been to format and leave the crio clean. Then we can run to our heart's content, and when we finally want to deploy permanently, we can do so.

The next time we want to "run", we do a reimage of the crio so that the code is all wiped, and go back to "running" the code rather than permanently deploying it.

That seems like a pretty drastic approach, but it does seem like it should work. I would think there are other, faster ways to do this such as the no app switch. Also, you could set it to not run at startup (as long as you didn't tell the imaging tool to "always run as startup"). You could also FTP to the cRIO and delete the startup application.

I never have this problem anyway... perhaps the code you are deploying is overly processor hungry? Perhaps you are cancelling when you don't need to (there are a number of poorly worded dialogs in LabVIEW RT that can mislead you into thinking there is a problem and luring you into aborting an operation in progress that would complete successfully if you don't read them carefully)?

-Joe

[Alan Anderson]

Originally Posted by rwood359

We were having problems doing a "run as startup". I can't remember the error message, but we were getting several lines that said something to the effect that it couldn't complete with the current dialog. Clicking NEXT for each error cleared the message and sometimes would let the load complete.

That sounds very much like the informational message one expects when about to overwrite a program that is already running on the cRIO. It's not an error; it's just letting you know that if you continue, the existing program will be halted. Since you're trying to put a new program in place, halting the old one is exactly what you want to do.

[Ziaholic]

Doug gave us the same advice. We were having random sucess with our deployment, more failures than successes. IMO, the NoApp switch is a better solution than a cRIO re-image for a few reasons ... with the primary reason being the chance of lobotomizing the cRIO if the re-image goes bad.

Here are some steps I posted in the "Deployment Issues" thread ...

1) on the cRIO, set the No App dipswitch to TRUE (this prevents user code from running when the cRIO boots, regardless of the SW settings)

2) reset the cRIO and feel free to deploy/run/build/set-as-startup, whatever ... As long as that dip-switch is set, it'll never run the startup apps when it boots, and the deployents (or just hitting RUN in the RobotMain.VI) works every time.

3) when you are done debugging and/or troubleshooting and want to try it out in a "realistic" fashion, do a final build and Set as Startup.

4) when the upload to the cRIO is complete, there's a window to close and then it asks you to reboot the cRIO. Before you click the Reboot button, UNSET that dip-switch

He said that they are looking into why this is occurring, but apparently they've got a pretty good workaround until we get a patch.

After I learned this trick, I never saw those errors again, and we were slinging code onto our 'bot all day today (Practice day at the AZ regionals)

[apalrd]

We have this problem often, but only when fully deploying (not running main). Our solution is to reformat, as the cRio was not put in a position that is easy to get to to flip the No App switch. We have never had a problem in the past that required us to press the buttons on the RC or cRio, so we burried it in a place where there is hardly enough room to get a tool in there. It works fine, and we can plug things into it, we just can't flip the switch every other time we have to download. The errors seem to occur after a day of not downloading code, so formatting once will get us a day of downloading happily. We found that the No App switch works after talking to 27 at Kettering, seems they have a similar problem sometimes. We actually spend 1/2 hour of our unbag time trying to download code. (then we spend 45 minutes finding out why Arcade Drive in auton wasn't driving, then found out it scales inputs without telling us). I always assumed this bug was just another LabVIEW bug since I have found several others (probing while the code loads will crash LabVIEW, this bug involving not downloading, loss of cRio while downloading, etc.)

[lscime]

Originally Posted by Tom Line

While we're talking about odd bugs - we get one where when we try to run (not permanently depoy) code, it will not do so. It will get to the point where it is trying to run the project file, and we will lose connection to the crio.

This is exactly the problem we had. Setting your cRIO to "no app" before booting will allow you to download and run code on it. Without having to worry about formatting it

[apalrd]

Today I tried doing a Run from Main on the practice robot. It worked fine yesterday, and it worked fine Saturday, but it didn't work today. While downloading, it would hit Buzz15.lvproj, wait for a while, then say "Waiting for Real-time target" with the option to "stop waiting and disconnect". It will eventually timeout and say "Lost connection to real-time target", and when I look at the run box the last thing it tried was "RT CompactRIO target (failed to deploy target settings) (newline) Deploy completed with errors. Its really really annoying. I tried everything. New battery, tether, reboot several times, then flipped the No App switch and booted it again. All was well. Its really annoying to have to reach deep into the robot where the cRio is. And this is the practice robot - the real robot with bumpers makes it even harder since its much harder to see without the bumpers. If we would have known this during week 1, we would have put the cRio somewhere else so at least we could flip the No App switch when we needed to. It's really annoying to have to flip the No App switch.

[Tom Line]

That's the /exact/ issue we see, on the same file apalrd.

Since formatting takes just slightly longer than building if you start both at the same time, we simply format before uploading each build, then format again before we run code.

We have a standard "hands off" policy regarding the Crio. Things do NOT get plugged and unplugged - everything goes through an adapter. Dip switches do NOT get changed, and no one touches it.

The last thing on earth I want is some over-zealous screwdriver-jockey to put one of those dip switches through the back of the crio because they don't understand how fragile some things are. (Nothing against screwdriver-jockies of course....) Even worse would be to screw up the ethernet port to the point where we can't program at all.

That's just our way of doing it. It sounds like the dipswitches work well for lots of folks, and that's cool!

[Greg McKaskle]

I have never personally had to use the No App switch. If it is helping you to get things deployed, then by all means use it.

On the other hand, I suspect the need for using this indicates that the code is using so much CPU that the protocol methods are being starved.

If using LV, you should turn off the global for disk logging of errors. Doug Norman posted good directions for doing this. It involves changing one global variable and it is a very good thing for LV teams to do.

The next thing a LV team might want to do is to open Tools>>Real Time>>System Manager. This is basically the task manager for the LV cRIO. To get an accurate reading of CPU usage, click to the VI tab and turn off the Track VI States. Then click back to the first tab and Start. The bottom chart shows CPU usage and you should help understand what the cRIO is having to perform. Using this, you can run different modes, turn off different features, even comment out some code, and use the task manager to learn what the CPU cost is of different features.

Feel free to post questions if you have issues shrinking the CPU usage. My assumption is that by cutting the CPU usage even a bit, you will no longer need to use the No App button.

Sorry I don't have as much detail for nonLV teams. The profiling tools exist, I'm just not as familiar with them. I suspect that the download issues would be caused by the same sorts of issues.

Greg McKaskle

[apalrd]

@rwood359
That is common. There is just an old version of code on the cRio.

@Greg:
So, you are saying the cRio isn't powerful enough? I will turn off the global logging and see if it helps. I'm not even using any camera-related things (no camera on dashboard or target tracking) so I can't imagine how many problems teams using it must be having. It would have been nice to know about this back in week 2 when we mounted the cRio way in the belly of the robot..

@Joe:
We must run as startup, since it has to load the code when we go on to the field. That LabVIEW RT dialog that says "Waiting for Real-time target" with the "stop waiting and disconnect" box during Run and Deploy had me thinking it had failed - when it hadn't. However, once I let it wait, sometimes it will actually fail and I will have to find a little screwdriver, reach way into the belly of the robot, flip the No App switch, try again, and flip the switch back.

I do not believe my code is overly processor-hungry, it has two threads processing PID loops running with 50hz (slightly faster than IFI processor) and a few other threads with open-loop controls running at 50hz or 20hz. My autonomous code has a single loop at 100hz (same speed as Victor updates), although it is not running when I am downloading. I have checked, and none of my VI's are running without waits. Last year we never had this problem. What has changed to cause this?

[rwood359]

Originally Posted by apalrd

@rwood359
That is common. There is just an old version of code on the cRio.

I understand that. The problem is that the new code will not load without re-imaging the cRIO or using the no app startup.

[Greg McKaskle]

Powerful is relative. As I said, I've never needed to use the switch. If the download fails, my assumption is that it is because the helper task that does the RT protocol, including downloading code, is not given enough CPU and is timing out. It could also be that it is failing for a reason unrelated to CPU.

Based upon the other posts you have made, it seems like something odd is definitely happening. Have you run the System Manager yet? I'm curious to see what it says, and in fact, you should be able to leave it up while you try to download. Even more informative would be to use the serial cable and the command line profiling.

I'd also be interested in running your project, looking at the performance, and determining why you are getting the dialog. I'll be leaving for the Dallas event tomorrow, so I may be slow getting to you, but you can post it here or PM and I'll give an email.

As for the processor power. Think of it as the immovable object/irresistible force conundrum. Computers do what you tell them, and the RT System Manager will help you figure out what it is being asked to do.

Greg McKaskle

[ayeckley]

Originally Posted by Greg McKaskle

The next thing a LV team might want to do is to open Tools>>Real Time>>System Manager. This is basically the task manager for the LV cRIO. To get an accurate reading of CPU usage, click to the VI tab and turn off the Track VI States. Then click back to the first tab and Start. The bottom chart shows CPU usage and you should help understand what the cRIO is having to perform. Using this, you can run different modes, turn off different features, even comment out some code, and use the task manager to learn what the CPU cost is of different features.

Does anyone have any benchmark data for what a "normal" processor load is? I've found that running the default code puts the cRIO CPU load at about 92%. I've found that running our "full featured" code (ten motors, four encoders, two analog inputs, lots of global variable use) the processor load is about 95%. Even shutting down most of the VIs doesn't seem to have a significant impact on processor load. With that level of insensitivity, I don't see a whole lot to be gained by using the System Manager as an optimization tool. Are there a lot of housekeeping tasks going on in the background that drive the CPU load so high?