cRio Constantly Rebooting

[KayyPii]

Hello everyone, my team have been struggling with an issue with our cRio losing communication for the past few days. When the robot loses communication, the status light on the cRio will flash for a second indicting a reboot. We have updated the image to v28 and redeployed the code several times.

At first we suspected that the motors were drawing too much current through the power distribution, but even changing from victors to jaguars did not solve the problem because it was not tripping the jaguars over current fault mode. Furthermore, we used the kit bot set up, so there shouldn't be any reason that the motors are drawing too much current.

Event:
When we turn the robot on, everything is well and we are able to gain communication to start driving. But after some amount of time driving, the cRio will just reboot itself. This can happen anywhere between 2 seconds to the full 2 minutes

Mechanical set up:
1. drive train is the basic kitbot set up with the CIMple box, 2 CIMs on each, 12 : 26 sprocket ratio.
2. The drive motors are powered by 4 Jaguars (we have also tried victors but it was the same result.)
3. Control system has the basic set up with the radio connected on port 1 of the cRio
4. There are also 3 additional window motors on the robot but they are not used before the robot loses communication

Specifics:
1. The issue happens after some random amount of time driving. The lowest recorded voltage I saw on the dashboard recorded at 8 volts.
2. The program is done in Java. The code is also very simple.

We have searched the forum many times but was not able to find any other posts with this issue and quite frankly, we are stuck

Hopefully some one out there have the answers =S

[cabbagekid2]

We are having the same issue. After sometime of driving randomly the crio will cutout (reboot i guess) and then the driver station will loose communication and code. Then it will restart again.

We are using Labview and firmware version 28.


We also noticed before (v27) that when our grabber comes down and hits the robot frame the crio reboots. I thought the Crio could handle some massive Gs but I guess it can't WHILE running? We're starting to think something is wrong with our crio.

[Greg McKaskle]

The G's probably aren't the problem.

You should probably use a multimeter to measure the voltage difference between the grabber and frame. You may have a frayed wire or something else shorting when these touch.

Another test is to wiggle various cables -- similar to how they will be disturbed when the arm hits the frame. If wiggling a battery, PD, Digital SideCar, radio, or cRIO power cable causes the reboot, fix the loose connection.

If it isn't the cables, but a tap on the cRIO causes a reboot, I'd recommend following the cleaning procedure posted on ni.com/frc, and apply the gaskets if they aren't already. Jostling metallic debris around inside the cRIO can cause a reboot, or blow a fuse.

For the original poster -- look at the Diagnostics tab about errors, and then look at netconsole output to see if it is a SW crash.

Greg McKaskle

[Al Skierkiewicz]

Guys,
The Crio chassis is connected to battery common and requires it to be insulated from the frame of the robot. If it is not and you have another motor short like some are experiencing with RS775 motors, you may be driving motor current through the chassis of the Crio and into the power wiring. This will cause the Crio to lose at least half of it's power supply which will cause a reboot. That is assuming you are connected to the dedicated +24 volt regulated power supply on the PD.

[Jared Russell]

We had this issue crop up consistently throughout the Florida Regional starting on Friday afternoon and continuing into the playoffs. At some point in the match, the cRIO would reboot and we would lose comms for at least 45 seconds. The FTA was able to tell that our battery voltage was dropping to <7.5 volts or so just prior us rebooting.

We practiced with our robot (very aggressively - moreso than in a real match) for a week+ prior to ship and then completed practice day with no power issues cropping up whatsoever. The robot has 6 drive motors (4 CIMs, 2 RS-775s), a compressor, a FP motor running an arm, and a RS-775 running a roller claw. That is a lot, but none of the motors even get warm during a 2 minute match, so I am disposed to believe that the problem is electrical rather than mechanical in nature (or perhaps a hybrid of the two). Each time when we died on the field, we were not in a pushing match or anything - we were simply maneuvering around the field at a moderate speed, sometimes while manipulating a tube.

We pulled our electronics off the bot prior to crating, and will be able to try and reproduce/diagnose the problem in our shop over the next few weeks.

What I absolutely dread, however, is the possibility of not being able to reproduce/diagnose the issue at home, and then going into our next event with zero confidence that we're not going to have the same problem crop up again...

We are running Java v28 with no vision processing, but a lot of SmartDashboard I/O. I will use NetConsole in the coming weeks to make sure it isn't a code issue.

Any other suggestions on what to look for/try?

EDIT: Things we have checked already:
-Replaced 120A main breaker (immediately prior to cRIO reboots happening, the robot popped its main breaker during a match. The breaker itself looked suspect so we swapped out. After the switch, no breaker tripping, but cRIO brownouts started to happen.)

-BaneBots RS-775 motors with case shorts were identified and removed, but the issue still happened.

-Replaced our compressor.

-Re-did wiring for main breaker, PD board, cRIO power.

[Al Skierkiewicz]

Jared,
A picture would help greatly.

[ATH1RSTYM00SE]

our team was having a similar problem and at our regional we talked to some guy from NI. He said that it was a problem on their end because labview doesnt prioritize com over the code. What he told us to do was make sure our code was as simple as possible and wasn't running a bunch of things in parallel because that would cause the crio to give up com to try to run the code

[boomergeek]

If you had case shorts on your RS-775, then I would suggest to look to possibly swapping out the motor controller that had been attached to the case shorts.

[Al Skierkiewicz]

Jared,
After witnessing a variety of faults over the weekend, I would recommend you disconnect the Banebots motors from the outputs of your speed controllers and remove the breakers that feed those controllers. Try running like that and see what happens. It would seem that most teams that report this problem are also reporting that they are using the six motor drive. Seems like a coincidence doesn't it?!?

[Jared Russell]

Al,

We will certainly try that (as soon as our tool crate returns - hopefully in a day or two). We did try removing the RS-775s from the drive, but not disconnecting the speed controllers' (we use Victors) breakers. Thanks for the advice!

[Al Skierkiewicz]

Jared,
I don't think this an issue that would be solved with either the Victor or Jaquar.

[Greg McKaskle]

because labview doesnt prioritize com over the code

This explanations is a possibility, and I've asked teams before if they are boosting priorities and possibly doing this. I've yet to find a team whose priorities or code caused this. Also, this is just as likely in all languages.

From my observations, a 45 second reboot is quite a bit longer than a cRIO warm or cold reboot. That time is closest to the boot time of the dlink. If you can see that the orange light is not blinking, the dlink bridge is not up. Make sure that the dlink power is the boosted supply on the PD and obviously wiggle and squeeze the wires. In NJ, we saw several robots with shorts at the PD end of the cable and several with loose barrel connectors at the dlink end.

Greg McKaskle

[scooperman]

Perhaps this is related:

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

[Jared Russell]

Originally Posted by scooperman

Perhaps this is related:

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

Thanks for the link. We are using the power cable included with the DLink, but I will make sure that the connection is secure and robust.

Hopefully the issue is that simple. The fact that the field could tell that our battery voltage had dropped precipitously immediately before losing comms, however, makes that unlikely in my opinion.

[Travis Hoffman]

We had several robot resets in Pittsburgh that were very similar to what 341 experienced.

We have no RS-775's on our robot (thankfully). 4-CIM drivetrain, 2 RS-540's in the roller claw, 1 RS-550 for the arm.

The FTA reported the same low voltage reading to us at one point that was reported to 341.

Our first reset was after a hard impact with a tower. Most others occurred during normal operations, with no contact or other anomalous activity occurring at the time of reset. There were a few impacts later on in the competition that did not interrupt robot function at all.

We use C++. Others report using Java.

cRIO is isolated from the frame.

It appears that reports of this occurrence are getting more and more widespread, and the robots experiencing them more and more diverse in design and function. What are the common hardware and software ties that bind these robots and reboots together?

If any other teams have experienced similar issues, please share them.

Originally Posted by Jared341

We had this issue crop up consistently throughout the Florida Regional starting on Friday afternoon and continuing into the playoffs. At some point in the match, the cRIO would reboot and we would lose comms for at least 45 seconds. The FTA was able to tell that our battery voltage was dropping to <7.5 volts or so just prior us rebooting.

We practiced with our robot (very aggressively - moreso than in a real match) for a week+ prior to ship and then completed practice day with no power issues cropping up whatsoever. The robot has 6 drive motors (4 CIMs, 2 RS-775s), a compressor, a FP motor running an arm, and a RS-775 running a roller claw. That is a lot, but none of the motors even get warm during a 2 minute match, so I am disposed to believe that the problem is electrical rather than mechanical in nature (or perhaps a hybrid of the two). Each time when we died on the field, we were not in a pushing match or anything - we were simply maneuvering around the field at a moderate speed, sometimes while manipulating a tube.

We pulled our electronics off the bot prior to crating, and will be able to try and reproduce/diagnose the problem in our shop over the next few weeks.

What I absolutely dread, however, is the possibility of not being able to reproduce/diagnose the issue at home, and then going into our next event with zero confidence that we're not going to have the same problem crop up again...

We are running Java v28 with no vision processing, but a lot of SmartDashboard I/O. I will use NetConsole in the coming weeks to make sure it isn't a code issue.

Any other suggestions on what to look for/try?

EDIT: Things we have checked already:
-Replaced 120A main breaker (immediately prior to cRIO reboots happening, the robot popped its main breaker during a match. The breaker itself looked suspect so we swapped out. After the switch, no breaker tripping, but cRIO brownouts started to happen.)

-BaneBots RS-775 motors with case shorts were identified and removed, but the issue still happened.

-Replaced our compressor.

-Re-did wiring for main breaker, PD board, cRIO power.