cRio Constantly Rebooting

[Jared Russell]

We had this issue crop up consistently throughout the Florida Regional starting on Friday afternoon and continuing into the playoffs. At some point in the match, the cRIO would reboot and we would lose comms for at least 45 seconds. The FTA was able to tell that our battery voltage was dropping to <7.5 volts or so just prior us rebooting.

We practiced with our robot (very aggressively - moreso than in a real match) for a week+ prior to ship and then completed practice day with no power issues cropping up whatsoever. The robot has 6 drive motors (4 CIMs, 2 RS-775s), a compressor, a FP motor running an arm, and a RS-775 running a roller claw. That is a lot, but none of the motors even get warm during a 2 minute match, so I am disposed to believe that the problem is electrical rather than mechanical in nature (or perhaps a hybrid of the two). Each time when we died on the field, we were not in a pushing match or anything - we were simply maneuvering around the field at a moderate speed, sometimes while manipulating a tube.

We pulled our electronics off the bot prior to crating, and will be able to try and reproduce/diagnose the problem in our shop over the next few weeks.

What I absolutely dread, however, is the possibility of not being able to reproduce/diagnose the issue at home, and then going into our next event with zero confidence that we're not going to have the same problem crop up again...

We are running Java v28 with no vision processing, but a lot of SmartDashboard I/O. I will use NetConsole in the coming weeks to make sure it isn't a code issue.

Any other suggestions on what to look for/try?

EDIT: Things we have checked already:
-Replaced 120A main breaker (immediately prior to cRIO reboots happening, the robot popped its main breaker during a match. The breaker itself looked suspect so we swapped out. After the switch, no breaker tripping, but cRIO brownouts started to happen.)

-BaneBots RS-775 motors with case shorts were identified and removed, but the issue still happened.

-Replaced our compressor.

-Re-did wiring for main breaker, PD board, cRIO power.

[Al Skierkiewicz]

Jared,
A picture would help greatly.

[ATH1RSTYM00SE]

our team was having a similar problem and at our regional we talked to some guy from NI. He said that it was a problem on their end because labview doesnt prioritize com over the code. What he told us to do was make sure our code was as simple as possible and wasn't running a bunch of things in parallel because that would cause the crio to give up com to try to run the code

[boomergeek]

If you had case shorts on your RS-775, then I would suggest to look to possibly swapping out the motor controller that had been attached to the case shorts.

[Al Skierkiewicz]

Jared,
After witnessing a variety of faults over the weekend, I would recommend you disconnect the Banebots motors from the outputs of your speed controllers and remove the breakers that feed those controllers. Try running like that and see what happens. It would seem that most teams that report this problem are also reporting that they are using the six motor drive. Seems like a coincidence doesn't it?!?

[Jared Russell]

Al,

We will certainly try that (as soon as our tool crate returns - hopefully in a day or two). We did try removing the RS-775s from the drive, but not disconnecting the speed controllers' (we use Victors) breakers. Thanks for the advice!

[Al Skierkiewicz]

Jared,
I don't think this an issue that would be solved with either the Victor or Jaquar.

[Greg McKaskle]

because labview doesnt prioritize com over the code

This explanations is a possibility, and I've asked teams before if they are boosting priorities and possibly doing this. I've yet to find a team whose priorities or code caused this. Also, this is just as likely in all languages.

From my observations, a 45 second reboot is quite a bit longer than a cRIO warm or cold reboot. That time is closest to the boot time of the dlink. If you can see that the orange light is not blinking, the dlink bridge is not up. Make sure that the dlink power is the boosted supply on the PD and obviously wiggle and squeeze the wires. In NJ, we saw several robots with shorts at the PD end of the cable and several with loose barrel connectors at the dlink end.

Greg McKaskle

[scooperman]

Perhaps this is related:

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

[Jared Russell]

Originally Posted by scooperman

Perhaps this is related:

http://www.chiefdelphi.com/forums/sh...ad.php?t=93657

Thanks for the link. We are using the power cable included with the DLink, but I will make sure that the connection is secure and robust.

Hopefully the issue is that simple. The fact that the field could tell that our battery voltage had dropped precipitously immediately before losing comms, however, makes that unlikely in my opinion.

[Travis Hoffman]

We had several robot resets in Pittsburgh that were very similar to what 341 experienced.

We have no RS-775's on our robot (thankfully). 4-CIM drivetrain, 2 RS-540's in the roller claw, 1 RS-550 for the arm.

The FTA reported the same low voltage reading to us at one point that was reported to 341.

Our first reset was after a hard impact with a tower. Most others occurred during normal operations, with no contact or other anomalous activity occurring at the time of reset. There were a few impacts later on in the competition that did not interrupt robot function at all.

We use C++. Others report using Java.

cRIO is isolated from the frame.

It appears that reports of this occurrence are getting more and more widespread, and the robots experiencing them more and more diverse in design and function. What are the common hardware and software ties that bind these robots and reboots together?

If any other teams have experienced similar issues, please share them.

Originally Posted by Jared341

We had this issue crop up consistently throughout the Florida Regional starting on Friday afternoon and continuing into the playoffs. At some point in the match, the cRIO would reboot and we would lose comms for at least 45 seconds. The FTA was able to tell that our battery voltage was dropping to <7.5 volts or so just prior us rebooting.

We practiced with our robot (very aggressively - moreso than in a real match) for a week+ prior to ship and then completed practice day with no power issues cropping up whatsoever. The robot has 6 drive motors (4 CIMs, 2 RS-775s), a compressor, a FP motor running an arm, and a RS-775 running a roller claw. That is a lot, but none of the motors even get warm during a 2 minute match, so I am disposed to believe that the problem is electrical rather than mechanical in nature (or perhaps a hybrid of the two). Each time when we died on the field, we were not in a pushing match or anything - we were simply maneuvering around the field at a moderate speed, sometimes while manipulating a tube.

We pulled our electronics off the bot prior to crating, and will be able to try and reproduce/diagnose the problem in our shop over the next few weeks.

What I absolutely dread, however, is the possibility of not being able to reproduce/diagnose the issue at home, and then going into our next event with zero confidence that we're not going to have the same problem crop up again...

We are running Java v28 with no vision processing, but a lot of SmartDashboard I/O. I will use NetConsole in the coming weeks to make sure it isn't a code issue.

Any other suggestions on what to look for/try?

EDIT: Things we have checked already:
-Replaced 120A main breaker (immediately prior to cRIO reboots happening, the robot popped its main breaker during a match. The breaker itself looked suspect so we swapped out. After the switch, no breaker tripping, but cRIO brownouts started to happen.)

-BaneBots RS-775 motors with case shorts were identified and removed, but the issue still happened.

-Replaced our compressor.

-Re-did wiring for main breaker, PD board, cRIO power.

[Al Skierkiewicz]

Travis,
A small number of main breakers have a manufacturing defect that makes them intermittent. If you tap the red button and the robot lights flicker, this is likely the cause. Replacing the breaker is the only fix. Other sources of low voltage can be loose connections on the battery terminals. I recommend that a star washer be placed between the terminals before the hardware is inserted. This not only breaks through any surface crud, it also locks the terminals together so that they can't loosen. Any loose connection or crimp in the robot primary wiring can also cause this brown out condition that takes out the Crio and radio power. Although it doesn't occur often, some teams add #6 wire to the provided Anderson plug to extend the distance between battery and PD. This is the one place on the robot that wiring should be kept to minimum length as all robot current flows through this wire.

[Travis Hoffman]

Originally Posted by Al Skierkiewicz

Travis,
A small number of main breakers have a manufacturing defect that makes them intermittent. If you tap the red button and the robot lights flicker, this is likely the cause. Replacing the breaker is the only fix. Other sources of low voltage can be loose connections on the battery terminals. I recommend that a star washer be placed between the terminals before the hardware is inserted. This not only breaks through any surface crud, it also locks the terminals together so that they can't loosen. Any loose connection or crimp in the robot primary wiring can also cause this brown out condition that takes out the Crio and radio power. Although it doesn't occur often, some teams add #6 wire to the provided Anderson plug to extend the distance between battery and PD. This is the one place on the robot that wiring should be kept to minimum length as all robot current flows through this wire.

Al:

Thanks - we will check these items out first thing in Tennessee - after the rush of the elimination rounds, I actually thought to check how secure the robot's primary power wiring was...after we bagged the robot. Usually, that issue is never a problem for us.

I do know all the battery leads were securely bolted to the terminals.

I will likely check and replace the main breaker too, just in case. We've got plenty of those handy. I'll grab a few star washers for good measure.

I also know our radio power adapter is secured and has good strain relief at both ends, but I'll likely add a dab of hot glue to the barrel connector just to be safe.

[Chris is me]

I am willing to bet it's the 6 motor drive, combined with the additional current draw of a drivetrain being pushed against and pushing. We did some magic math at the beginning of the season and found that heavily loading all of those motors would lead to pretty quick battery drains (1:30 or so into a match with some other mechanisms running).

Talking to some friends with similar issues, changing the 6 motor drive's gearing to a lower speed (10 FPS is apparently a good sweet spot) made the issue go away entirely. From what I can recall, you guys are running at 13 FPS, which is much heavier on the motors.

[Jared Russell]

It's not out of the question, but a week+ of practicing (often for longer and/or harder than during a real match) and then perfect operation on practice day suggests, to me, that the issue is elsewhere. If it's not electrical, or somehow code/firmware related, then the only other possibility is that our bolted frame is losing its stiffness and causing the motors to work harder during turns (though the robot doesn't jump or otherwise seem to struggle at all to turn).

Besides, I don't believe that Travis is running a 6 motor drive, nor are either of our teams' issues happening as an obvious result of pushing matches (it happened to us less than a minute into a match where we hadn't even touched another robot). We also ran one match with just the 4 CIMs (still geared at 13 fps, though) and the issue didn't go away.