cRio Kernel Error/Reboots

[Marc P.]

Hello all!

I'm here asking for advice on behalf of Team Beta's programmers. The other night they were attempting to write some simple code to experiment with an encoder, with some success. Things seemed to be going well until the cRio inexplicably started rebooting itself.

The cRio powers on, boots up, runs the deployed code for just long enough for the classmate/driver station status to show "Teleoperated Disabled" for a split second before showing "No Robot Communication" as the cRio status light blinks on for a second, indicating it restarted.

The programmers are using C++/Wind River, the cRio is running image version 25, WPIlib and the build environments are all updated to the most recent downloadable versions.

My first thought was to format/re-image the cRio. While in the "not functioning properly" state, the cRio imaging tool can not find the cRio for re-imaging. After powering off the main breaker, I flipped the Safe Mode DIP to "on" and turned the breaker back on. The cRio then booted up, and was detected by the imaging tool. From there we were able to successfully format the cRio, and the imaging tool then said to power down, turn off the safe mode switch, and power back on. After doing so, the imaging tool was again able to detect the cRio, and successfully set the IP address and applied the v25 image. The cRio will then boot up fine, the classmate/driver station is able to connect, and the status shows "No Robot Code." It will sit like that for extended periods of time without issue.

The programmers then proceeded to compile and deploy basic sample code with seemingly no issue. Once the deployment finishes, the cRio is rebooted via the "Robot Reboot" button on the classmate/driver station. It again reconnects to the classmate/driver station just long enough for the status to show "Teleoperated Disabled" for a split second before changing back to "No Robot Communication."

With Wind River still open after deploying code, the following error pops up:

Target Exception

VxWorks6x_10.28.36.2: Exception in Kernel Task FRC-RobotTask:0x12816e0 at pc=0x218682C

Vector 0x300 : Data Access addr=0x992611E8 status=0x21867C0

The System has stopped.  Do you want to attach the debugger now?

(yes)   (no)

Clicking yes opens a tab with scrolling lines saying:

deadbeef:  Read access not allowed
deadbef0:  Read access not allowed
deadbef1:  Read access not allowed
deadbef2:  Read access not allowed
deadbef3:  Read access not allowed
deadbef4:  Read access not allowed
deadbef5:  Read access not allowed
...and so on

The cRio then continues in it's rebooting loop, after multiple cold boots/power cycles, and the only thing that seems to stop it is booting in Safe Mode, formatting, rebooting in normal mode, and re-imaging. But even that only works until any code is deployed, after which the cycle begins again.

Steps we have tried:

Full power cycles (turning off breaker, unplugging battery, reconnecting, turning breaker back on)
Removing all cRio modules
Formatting the cRio in safe mode
Re-imaging the cRio in normal mode
Blowing out the cRio with compressed air
Deploying default/sample code
Fresh battery

All to no avail. Is there anything else we can try before contacting NI?

Thank you all for any advice or assistance!

[Alan Anderson]

I don't have time to give a considered reply, so I'll just suggest you look at the code for an uninitialized pointer.

[Mark McLeod]

Next time try using the NO APP switch/cRIO reset instead of all that reformatting.
It just sounds like your code is going south and freezing up the cRIO.

[Greg McKaskle]

To add to what Mark and Alan are saying, the benefit of C++ is ultimate control over the cRIO. The downside is, bad code results in a reboot. Use NoApp to recover from bad code.

Greg McKaskle

[Marc P.]

Thanks guys! The NoApp switch tip will help save us some time troubleshooting.

The programmers mentioned they tried deploying the default non-modified code, as well as code that was known to work earlier in the week with the same result. I'll try a clean installation of Wind River on a separate laptop with the default/sample code when we can get back in to the school tomorrow. We've lost 4 days in the last 2 weeks to snow closures, and have some catching up to do!

Thanks again!
-Marc

[Marc P.]

The plot thickens...

At our meeting tonight the first thing we tried was booting the cRio with the NoApp switch toggled, and it still rebooted. We powered up the circuit breaker, after 15-20 seconds the classmate status showed "No Robot Code," and a split second later changed to "No Robot Communication." Just before "No Robot Code" appeared, the Robot Signal Light blinked 5 times, then stayed off as the classmate status changed to "No Robot Communication."

After doing the safe mode/format/reboot/re-image/reboot cycle again, we tried loading default/stock tank drive code from a fresh install of Wind River on a different laptop, and the result was the same- code loaded fine, but after the first reboot the same problem occurred, even cold booting with the NoApp switch.

Is there anything else we can try?

[Alan Anderson]

Originally Posted by Marc P.

Is there anything else we can try?

At this point, I suggest you try a different cRIO.

Before that, though, check the 24v power and look carefully for metallic debris in the cRIO connectors and inside the chassis. You said you used compressed air, but that won't necessarily clean out sharp crumbs of aluminum.

[Marc P.]

Good news! It turns out it was a code problem, and the cRIO is fine.

The programmers were using the "FRC Sample Simple C Template" in Wind River as the default program, which we found out, the hard way, doesn't work. We're not sure why it's still included, even in fresh installs of Wind River/WPIlib.

After formatting/re-imaging the cRIO again, we created and compiled a new project using the "FRC Simple Robot Template" and it worked perfectly.

Also, in reading the "crio-frc_Operating_Instructions" PDF published by NI-

"The NO APP switch does not affect C/C++ startup applications. You can
use the Undeploy option in Wind River Workbench to disable a C/C++
application."

...which explains why it would continue to reboot while toggled.

Everything is back to normal now. Thank you all!

[Mark McLeod]

Sorry I'd forgotten the No-App switch didn't apply to C++.

[Alan Anderson]

Thank you for letting us know what you found. The information might help another team with the same problem if they find this thread.