View Full Version : cRIO interrupted during reimaging
We were reimaging the cRIO device and our network cable came loose during the "Formatting CompactRIO device..." phase.
After restoring the network link, we can ping the DS (10.0.0.5), but cannot ping the cRIO (10.0.0.2).
On the cRIO the Status light is blinking orange and the power light is green.
Restarting the cRIO imaging tool says that No CompactRIO devices were found. Verify the network connection.
Looking for advice on how to proceed...
Thanks
What exactly does the blinking look like? The pattern can tell you what error the cRIO is encountering.
From the cRIO documentation (http://decibel.ni.com/content/docs/DOC-2632):
Table 3. Status LED Indications
Number of Flashes Indication
1 The chassis is unconfigured. Use MAX to configure the chassis. Refer
to the Measurement & Automation Explorer Help for information about
configuring the chassis.
2 The chassis has detected an error in its software. This usually occurs
when an attempt to upgrade the software is interrupted. Reinstall
software on the chassis. Refer to the Measurement & Automation
Explorer Help for information about installing software on the chassis.
3 The chassis is in safe mode because the SAFE MODE DIP switch is in
the ON position. Refer to the Configuring DIP Switches section for
information about the Safe Mode DIP switch.
4 The software has crashed twice without rebooting or cycling power
between crashes. This usually occurs when the chassis runs out of
memory. Review your RT VI and check the memory usage. Modify
the VI as necessary to solve the memory usage issue.
Continuous flashing
or solid
The device may be configured for DHCP but unable to get an IP address
because of a problem with the DHCP server. Check the network
connection and try again. If the problem persists, contact National
Instruments.
One thing you could try is to set the IP RESET dip switch to on, and powercycling the cRIO. It should now have the IP 0.0.0.0
From the cRIO documentation (http://decibel.ni.com/content/docs/DOC-2632):
Push the IP RESET switch to the ON position and reboot the chassis to
reset the IP address to 0.0.0.0. If the chassis is on your local subnet and
the IP RESET switch is in the ON position, the chassis appears in MAX
with IP address 0.0.0.0. You can configure a new IP address for the
chassis in MAX. Refer to the Resetting the Network Configuration of the
cRIO-FRC section for more information about resetting the IP address.
You also can push this switch to the ON position to unlock a chassis that
was previously locked in MAX.
After resetting the IP, try to ping the cRIO. If it responds, see if you can reimage it now.
The flash is continous.
We are trying the Reset IP suggestion...
After the IP Reset, I still get a continous blinking status light.
The ethernet port on the cRIO shows a green light on the 10V/100 side, and a solid orange light on the link side.
Both MAX and the Imaging Tool are unable to find the cRIO.
We are linked to the DS throught ETH2 all lights are green, which is linked to the cRIO through ETH1 with a green light on the DS and the lights on the cRIO described above.
EHaskins
20-12-2008, 14:04
Assuming you changed your team number while re-flashing, you should change your PC's ip address to 10.24.93.6, and then try to ping 10.24.93.2. But that probably won't matter now that you reset the IP address.
Thanks Eric, just tried that and the request timed out.
Anyone know if NI provides phone support on Saturdays? And if so, what is the number to call?
After the IP Reset, the documentation says the IP address is reset to the factory default. I assume this is 10.0.0.2?
OK, powered down the cRIO, powered it back up and now the orange link light on the cRIO ethernet port is blinking, so I am assuming we have communication taking place. That is a step in the right direction.
The Status light on the cRIO is still doing a slow continous blink.
Still wired from the computer to the DS to the cRIO. We temporarily tried using the crossover cable which also shows blinking orange link led.
a ping of 10.0.0.2 and a ping of 10.24.93.2 (with the computer ip set appropriately) are now returning Destination Host Unreachable
EHaskins
20-12-2008, 15:17
Try connecting the cRio and your PC to a router with DHCP enabled. Then check the router DHCP table and see if it got an IP address from DHCP. Its possible when it was reset it changed to DHCP configuration.
EHaskins
20-12-2008, 15:26
You could also connect your PC to the console serial port on the cRio. On about line 30 the IP address/subnet mask information will be displayed.
To do this you need to configure your cRio and PC as described in section 5.4.1 of the control system manual. If you're looking for another terminal client for Windows PuTTY is what I'm using.
I'll have to go out and find a NULL Modem serial cable to try that, but in the mean time.....
I did get MAX to talk to the cRIO and was able to reset the IP address from 0.0.0.0 to 10.0.0.2.
I then was able to reflash the cRIO to a point. It got to the point where it was formatting, and then I received a UDP Unexpected Error 60 (sorry for the paraphrase, I didn't get to write the text of the message).
So now I am back to trying to reset the IP again.
Oh yeah, when I turned off the firewall on the computer, MAX has a much easier time finding the cRIO
Back to reimaging.... keep your fingers crossed.....
OK, so twice now we get this message during the Format phase of the reimage process:
Unable to complete operation because of unexpected error 60: UDP Open in repFPC open connection.vi->SendRebootCommand.vi->RebootSystem.vi->SetTargetConfig.vi->SetIPAddress.vi->ConfigureUtility.lvlib:SetIPAddrAndGetInfo.vi->ConfigUtility.lvlib:ReImageTarget.vi->ConfigUtilty.lvlib.Dialog.vi
Any thoughts?
Greg McKaskle
20-12-2008, 16:58
Error 60 is listed as network address is currently in use. Troubleshooting the network stuff isn't my forte, but maybe that helps you.
Here are other procedural things that have helped other teams in the past.
Hook the cRIO directly to the PC, not through the DS, router or any sort of switch.
Turn off firewalls.
Turn off other enet cards, wifi cards, and other things that could confuse the routing as the cRIO IP moves around during the formatting process.
Try the formatting on anther computer (typically the failure was due to one of the previous two, and sometimes this is easier).
Pay careful attention to the DIP switches. Flipping them on to get the cRIO IP reset may then cause problems as you try to reimage.
Greg McKaskle
Hallelujah!
We have success.
Here is what we did.
1. Do an IP Reset
set the IP Reset DIP switch on the cRIO to ON
push reset button on cRIO
Set the IP Reset DIP switch on the cRIO to OFF
2. Power off the cRIO, then Power it back on
2a. Turn off your computer's firewall
3. Connect to computer to the cRIO using the crossover cable
IP Address on computer 10.0.0.6 SubNet 255.0.0.0
4. Run MAX (Measurement and Automation) to reset IP address (it is now 0.0.0.0) on the cRIO
Tools\NI-Serial\Ethernet Device Configuration
This should find the cRIO and show the MAC address
5. Click Properties and set the IP Address to 10.0.0.2 SubNet 255.0.0.0
6. In MAX, click Remote Systems and then F5 to refresh and you should now see the cRIO.
7. Expand the cRIO under Remote Systems,click on Software, and click on Add/Remove software
Reload the NI-RIO 2.4.1 software
At this point you should be able to return to the FRC cRIO Imaging Tool and flash the cRIO
99. Turn your firewall back on
Oh yeah, and one final step that seemed to work, we switched computers. When we did that everything worked fine.
And they say water boarding is torture. Ha, it ain't got nothing on flashing a cRIO
Alan Anderson
20-12-2008, 21:34
The "Error 60" makes it sound like your computer was set to the IP address the cRIO wanted to use. It might be worth checking the network configuration on the first computer to see if that was your problem.
We did so many things to get this working including:
IP Resets
Loading the base NI-RIO software
Formatting the cRIO in Safe Mode (scary thing to do)
Running directly to the cRIO with the crossover cable
All kinds of address configurations (0.0.0.0, 10.0.0.2, 10.24.93.2)
The computer was always 10.0.0.6 or 10.0.0.10 (or subbing 24.93 for the zeros)
Using a different computer
Turning off the firewall
Disabling all other network connections on the computer
Clearing all COM channels (paraphrased I think) using MAX
Switching to another computer finally seemed to clear things up and make things work.
I think that was all the things we tried. It took us at least 5 hours today to finally get it figured out.
I did my best to document what I think were the relevant steps in an earlier post, hopefully to help other teams.
I sure wish I knew this morning what I know now about loading the cRIO image, things would have gone much easier
We did so many things to get this working including:
IP Resets
Loading the base NI-RIO software
Formatting the cRIO in Safe Mode (scary thing to do)
Running directly to the cRIO with the crossover cable
All kinds of address configurations (0.0.0.0, 10.0.0.2, 10.24.93.2)
The computer was always 10.0.0.6 or 10.0.0.10 (or subbing 24.93 for the zeros)
Using a different computer
Turning off the firewall
Disabling all other network connections on the computer
Clearing all COM channels (paraphrased I think) using MAX
Switching to another computer finally seemed to clear things up and make things work.
I think that was all the things we tried. It took us at least 5 hours today to finally get it figured out.
I did my best to document what I think were the relevant steps in an earlier post, hopefully to help other teams.
I sure wish I knew this morning what I know now about loading the cRIO image, things would have gone much easier
Although NI doesn't offer phone support on the weekends (and not till Jan. 5 for FRC) we do have a large number of knowledge base articles including topics such as network communication issues with the cRIO. I recommend searching for the operating instructions for the cRIO-9074 for more information.
It also doesn't hurt to ask questions like this on ni.com/FIRST so we can identify the gaps in our documentation (this being one of them).
One thing I would like to point out that Formatting the cRIO shouldn't be a scary thing to do. Its more robust than you think.
Sure thing, I'll repost the relevant stuff from this thread over at NI.
What makes reformatting the cRIO scary was the stories I have heard about how carefully the benchtest procedures have to be followed or you could brick your system. We were way off the beaten path, and it was our first time with the cRIO, so we had no idea if we were going to be able to load software onto the cRIO after a format in safe mode. Then the format ran for 20 minutes without any type of feedback whether it was making progress or not. We finally had to close the "formatting" message box without having any idea if it was still working or complete. Fortunately there were no ill effects from doing that.
I know it is tough to document everything or cover every use case. Some improvements that would have helped:
A progress bar on the formatting process to let us know how far it has progressed.
Perhaps the message box that pops up when the format in safe mode is kicked off could give an idea of how to proceed once the format is complete, i.e., point to the documentation on how to reload software. I had no idea if the cRIO was going to be wiped to the point I would not be able to load any of the base software.
Why is the Ethernet device search on the NI-Serial menu? I just happened to stumble across it while trying to figure out what to do.
It would be nice to be able to cut and paste from the error messages, especially when there is a long list of VIs in them.
After the IP Reset, the documentation says the IP address is reset to the factory default. I assume this is 10.0.0.2?
Factory default is 0.0.0.0.
Hallelujah!
4. Run MAX (Measurement and Automation) to reset IP address (it is now 0.0.0.0) on the cRIO
Tools\NI-Serial\Ethernet Device Configuration
This should find the cRIO and show the MAC address
5. Click Properties and set the IP Address to 10.0.0.2 SubNet 255.0.0.0
These steps shouldn't be necessary. The same protocol is used by MAX and the imaging tool to find RT devices, so if MAX found it at 0.0.0.0 then the imaging tool should too. In fact, during the imaging process the cRIO will temporarily have a 0.0.0.0 IP.
Formatting the cRIO in Safe Mode (scary thing to do)
This shouldn't be that scary, notice the check box for format cRIO? It's doing the same thing.
I think all your problems could have been resolved using a partially-documented feature of the imaging tool. In the event you have a power/network failure in the middle of imaging and your controller is left in an unusable state, you should be able to toggle the "SAFE MODE" dipswitch on, reboot, and then from the imaging tool restore the device back to a functional factory default setting. You can then toggle "SAFE MODE" off, reboot, and run the imaging tool again to put the correct FRC_2009 image on the device. On Monday morning I'll double check that all works, then see if we can't get a last-minute addendum to the documentation (I definitely owe our tech writers donuts).
Alan Anderson
21-12-2008, 21:11
What makes reformatting the cRIO scary was the stories I have heard about how carefully the benchtest procedures have to be followed or you could brick your system.
Try not to get too worried about what you've "heard". Pay attention to things you've read or experienced yourself, and verify anything else before you take it too seriously. You'll eventually learn which sources of second-hand information you can trust, but until that time, it might be a good idea to treat everything you "hear" as unverified rumor.
In this specific case, you've misunderstood two things. First, the "brick" problem relates to the Driver Station, not the cRIO. Second, the procedure that must be followed carefully to avoid bricking it is the DS firmware update, which is not supposed to be done until after the bench test has been performed. It is probably theoretically possible to misprogram the cRIO beyond easy recovery, but I think someone would have to do it intentionally.
Ah, if I only knew then what I know now :)
I don't know if this already exists, but it sure would be nice to have a FAQ for when things go wrong, especially pointing to relevant documentation.
Also thanks for the clarification between the DS and the cRIO as far as bricking is concerned.
This has definitely been a good learning experience for the team and I hope others can benefit from it as well.
Thanks everyone for your help!
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.