Death of our cRIO, and some suggestions for NI

[Chris Hibner]

Today at the Detroit district event our cRIO suffered a slow decline to its ultimate death. Actually, I don't think it's completely dead, but at least the power connector doesn't work leaving it dead for all intents and purposes. The slow decline (intermittent problems) caused us to sit stationary for 20-30 seconds in a match or two, and the death caused us to sit still for the entire match after our first kick (the shock inflicted the final blow).

I've been very supportive of NI and the new control system, but I have some constructive criticism here. I don't want this to become a "piling on" thread, but I have one suggestion that would have saved us from missing a match.

Once we decided we needed to swap the cRIO, we had to get a loaner, which means we had to re-image the cRIO to set it up with our team number. Issues with the imaging process cost us about 40 minutes and we finally decided that the loaner cRIO wasn't working, so we had to go to a SECOND loaner. All of this wasted time caused us to miss a match altogether. With some simple changes, we could have easily saved 30 minutes and we wouldn't have missed the match.

Sorry for the long backgroud, here's the rub:

The cRIO Imaging Tool needs a status bar

if that wasn't clear,

THE CRIO IMAGING TOOL NEEDS A STATUS BAR

During the re-imaging process, the tool got to the "Formatting the cRIO" part, and sat there saying that for a long time. Without some sort of status bar, we had no idea if the process was still working, it was hung up, the cRIO crashed, etc. If we could see a) a status bar that stopped moving, or b) some type of "still working" feedback like what you get when a web page is loading, we would have known that the process was stuck.

Considering how long it takes to reformat the cRIO (and how much time pressure we were under), we didn't want to cancel the process and start over, especially if it was this close to being done. But without a status bar, we had NO IDEA.

We finally wound up canceling the operation, trying the safe mode thing, and that hung up as well (at least we assume it hung up, but we had no way of telling without a status bar (does anyone see the common theme here)).

Anyway, after 40 minutes of frustration, we finally decided that it was time to get ANOTHER cRIO. That one imaged without issue.

Once again, here's my beef: if we had a status bar, we probably would've noticed it frozen within the first 2 minutes. We could've tried it again, noticed it frozen, then got another cRIO. That would've saved us at least 30 minutes, and we wouldn't have missed a match.

I know that the tone may seem a little angry, but it really isn't. I'm actually keeping fairly light-hearted about the whole deal, all things considered. So, if you came away thinking I'm furious, re-read this post with a sarcastic yet light-hearted mindset.

Anyway: Status bar - PLEASE.

[EricVanWyk]

Good idea. I've made sure the right people have seen your request.

[The Lucas]

I realise if the power connector is dead it is unusable and you actually needed a new one. However, I have to ask:
Did you try puting the cRIO in Safe Mode and reimaging it?
I have seen the same problems and I was always able to fix the cRIO by putting it in Safe Mode (flip DIP switch), restarting and imaging (3 different cRIOs, 2 ours, 1 another team at MD). Of course after you format it in safe mode you have to turn safe mode off and format it again. This whole long process is made worse because you are staring at a screen with no status bar and everyone keeps asking "when is it going to be done?" and you keep telling them you have no clue, but it will probably take a long time However, once this excruciatingly painful process is done the cRIO usually works as good as it ever had. Unfortunately, then I have to tell everyone I need another minute to reinstall the CAN plugin and download the code

[Chris Hibner]

Originally Posted by The Lucas

I realise if the power connector is dead it is unusable and you actually needed a new one. However, I have to ask:
Did you try puting the cRIO in Safe Mode and reimaging it?
I have seen the same problems and I was always able to fix the cRIO by putting it in Safe Mode (flip DIP switch), restarting and imaging (3 different cRIOs, 2 ours, 1 another team at MD). Of course after you format it in safe mode you have to turn safe mode off and format it again. Of course this whole long process is made worse because you are staring at a screen with no status bar and everyone keeps asking "when is it going to be done?" and you keep telling them you have no clue, but it will probably take a long time However, once this excruciatingly painful process is done the cRIO usually works as good as it ever had. Unfortunately, then I have to tell everyone I need another minute to reinstall the CAN plugin and download the code

Our cRIO was dead via the power connector (no lights whatsoever on the cRIO).

For the first replacement cRIO, we did do the safe-mode thing (I've helped other teams do it at Kettering) - no dice.

[DjMaddius]

Im extremely glad you guys got another cRIO before match 46. You guys basically boosted up back into the top 10. Thanks

[Travis Hoffman]

Originally Posted by EricVanWyk

Good idea. I've made sure the right people have seen your request.

Without hijacking the thread, since this is along the same lines, can you ask the same people to add a "pause" feature to the otherwise-great NetConsole?

The old IFILoader terminal had a nice pause feature that was useful during debug.

[Alan Anderson]

Originally Posted by Chris Hibner

Today at the Detroit district event our cRIO suffered a slow decline to its ultimate death. Actually, I don't think it's completely dead, but at least the power connector doesn't work leaving it dead for all intents and purposes. The slow decline (intermittent problems) caused us to sit stationary for 20-30 seconds in a match or two, and the death caused us to sit still for the entire match after our first kick (the shock inflicted the final blow).

That sounds familiar. A team at the Boilermaker Regional had pretty much the same symptoms. After they replaced their cRIO, I took a close look at the "bad" one with one of their students. There was a small crumb of aluminum in one of the unused module connectors, underneath the plastic cap. I don't know if they got around to testing it again, but I suspect it now will work fine.

Suggestion: look for debris that might be shorting a power connection.

[Chris Hibner]

Originally Posted by Alan Anderson

That sounds familiar. A team at the Boilermaker Regional had pretty much the same symptoms. After they replaced their cRIO, I took a close look at the "bad" one with one of their students. There was a small crumb of aluminum in one of the unused module connectors, underneath the plastic cap. I don't know if they got around to testing it again, but I suspect it now will work fine.

Suggestion: look for debris that might be shorting a power connection.

I'll call the school and have them look into that. Thanks.

[Travis Hoffman]

Originally Posted by Alan Anderson

That sounds familiar. A team at the Boilermaker Regional had pretty much the same symptoms. After they replaced their cRIO, I took a close look at the "bad" one with one of their students. There was a small crumb of aluminum in one of the unused module connectors, underneath the plastic cap. I don't know if they got around to testing it again, but I suspect it now will work fine.

Suggestion: look for debris that might be shorting a power connection.

Sounds like yet another reason to mount the cRIO vertically, in a "panel mount" orientation. This minimizes the risk of the cRIO chassis turning into a robot debris wastebasket.

[EricVanWyk]

Originally Posted by Travis Hoffman

Without hijacking the thread, since this is along the same lines, can you ask the same people to add a "pause" feature to the otherwise-great NetConsole?

The old IFILoader terminal had a nice pause feature that was useful during debug.

Sure. I have access to the source, I'll try to make the change this week (schedule providing).

[Chris Hibner]

Originally Posted by Alan Anderson

That sounds familiar. A team at the Boilermaker Regional had pretty much the same symptoms. After they replaced their cRIO, I took a close look at the "bad" one with one of their students. There was a small crumb of aluminum in one of the unused module connectors, underneath the plastic cap. I don't know if they got around to testing it again, but I suspect it now will work fine.

Suggestion: look for debris that might be shorting a power connection.

As it turns out, that wasn't the issue. I know we always had our empty connectors covered, but I wasn't sure if any of them came uncovered. The people at the school said they were still covered and everything was checked to be clean when the cRIO was pulled out of the robot.

[Bob Bonczyk]

At West Michigan there were four teams that had crios go out that I knew about. We fixed two of the four crios by replacing the internal fuse.

[Russ Beavis]

Bob,
Did you really mean to say that you opened the cRIOs and replaced the fuses or did you open up the PD (aka Power Distribution) and replace devices within it?

Neither resolution is obviously preferable but there is a big difference.

Russ

[Bob Bonczyk]

We opened the crios that ere out of warranty.

[EricVanWyk]

Originally Posted by Bob Bonczyk

We opened the crios that ere out of warranty.

It is my understanding that they don't go out of warranty... What made you believe this was the case?