cRio Constantly Rebooting

Technical Control System
1

Hello everyone, my team have been struggling with an issue with our cRio losing communication for the past few days. When the robot loses communication, the status light on the cRio will flash for a second indicting a reboot. We have updated the image to v28 and redeployed the code several times.

At first we suspected that the motors were drawing too much current through the power distribution, but even changing from victors to jaguars did not solve the problem because it was not tripping the jaguars over current fault mode. Furthermore, we used the kit bot set up, so there shouldn’t be any reason that the motors are drawing too much current.

Event:
When we turn the robot on, everything is well and we are able to gain communication to start driving. But after some amount of time driving, the cRio will just reboot itself. This can happen anywhere between 2 seconds to the full 2 minutes

Mechanical set up:

  1. drive train is the basic kitbot set up with the CIMple box, 2 CIMs on each, 12 : 26 sprocket ratio.
  2. The drive motors are powered by 4 Jaguars (we have also tried victors but it was the same result.)
  3. Control system has the basic set up with the radio connected on port 1 of the cRio
  4. There are also 3 additional window motors on the robot but they are not used before the robot loses communication

Specifics:

  1. The issue happens after some random amount of time driving. The lowest recorded voltage I saw on the dashboard recorded at 8 volts.
  2. The program is done in Java. The code is also very simple.

We have searched the forum many times but was not able to find any other posts with this issue and quite frankly, we are stuck :confused:

Hopefully some one out there have the answers =S

2

We are having the same issue. After sometime of driving randomly the crio will cutout (reboot i guess) and then the driver station will loose communication and code. Then it will restart again.

We are using Labview and firmware version 28.

We also noticed before (v27) that when our grabber comes down and hits the robot frame the crio reboots. I thought the Crio could handle some massive Gs but I guess it can’t WHILE running? We’re starting to think something is wrong with our crio.

3

The G’s probably aren’t the problem.

You should probably use a multimeter to measure the voltage difference between the grabber and frame. You may have a frayed wire or something else shorting when these touch.

Another test is to wiggle various cables – similar to how they will be disturbed when the arm hits the frame. If wiggling a battery, PD, Digital SideCar, radio, or cRIO power cable causes the reboot, fix the loose connection.

If it isn’t the cables, but a tap on the cRIO causes a reboot, I’d recommend following the cleaning procedure posted on ni.com/frc, and apply the gaskets if they aren’t already. Jostling metallic debris around inside the cRIO can cause a reboot, or blow a fuse.

For the original poster – look at the Diagnostics tab about errors, and then look at netconsole output to see if it is a SW crash.

Greg McKaskle

4

Guys,
The Crio chassis is connected to battery common and requires it to be insulated from the frame of the robot. If it is not and you have another motor short like some are experiencing with RS775 motors, you may be driving motor current through the chassis of the Crio and into the power wiring. This will cause the Crio to lose at least half of it’s power supply which will cause a reboot. That is assuming you are connected to the dedicated +24 volt regulated power supply on the PD.

5

We had this issue crop up consistently throughout the Florida Regional starting on Friday afternoon and continuing into the playoffs. At some point in the match, the cRIO would reboot and we would lose comms for at least 45 seconds. The FTA was able to tell that our battery voltage was dropping to <7.5 volts or so just prior us rebooting.

We practiced with our robot (very aggressively - moreso than in a real match) for a week+ prior to ship and then completed practice day with no power issues cropping up whatsoever. The robot has 6 drive motors (4 CIMs, 2 RS-775s), a compressor, a FP motor running an arm, and a RS-775 running a roller claw. That is a lot, but none of the motors even get warm during a 2 minute match, so I am disposed to believe that the problem is electrical rather than mechanical in nature (or perhaps a hybrid of the two). Each time when we died on the field, we were not in a pushing match or anything - we were simply maneuvering around the field at a moderate speed, sometimes while manipulating a tube.

We pulled our electronics off the bot prior to crating, and will be able to try and reproduce/diagnose the problem in our shop over the next few weeks.

What I absolutely dread, however, is the possibility of not being able to reproduce/diagnose the issue at home, and then going into our next event with zero confidence that we’re not going to have the same problem crop up again…

We are running Java v28 with no vision processing, but a lot of SmartDashboard I/O. I will use NetConsole in the coming weeks to make sure it isn’t a code issue.

Any other suggestions on what to look for/try?

EDIT: Things we have checked already:
-Replaced 120A main breaker (immediately prior to cRIO reboots happening, the robot popped its main breaker during a match. The breaker itself looked suspect so we swapped out. After the switch, no breaker tripping, but cRIO brownouts started to happen.)

-BaneBots RS-775 motors with case shorts were identified and removed, but the issue still happened.

-Replaced our compressor.

-Re-did wiring for main breaker, PD board, cRIO power.

6

Jared,
A picture would help greatly.

7

our team was having a similar problem and at our regional we talked to some guy from NI. He said that it was a problem on their end because labview doesnt prioritize com over the code. What he told us to do was make sure our code was as simple as possible and wasn’t running a bunch of things in parallel because that would cause the crio to give up com to try to run the code

8

If you had case shorts on your RS-775, then I would suggest to look to possibly swapping out the motor controller that had been attached to the case shorts.

9

Jared,
After witnessing a variety of faults over the weekend, I would recommend you disconnect the Banebots motors from the outputs of your speed controllers and remove the breakers that feed those controllers. Try running like that and see what happens. It would seem that most teams that report this problem are also reporting that they are using the six motor drive. Seems like a coincidence doesn’t it?!?

10

Al,

We will certainly try that (as soon as our tool crate returns - hopefully in a day or two). We did try removing the RS-775s from the drive, but not disconnecting the speed controllers’ (we use Victors) breakers. Thanks for the advice!

11

Jared,
I don’t think this an issue that would be solved with either the Victor or Jaquar.

12

because labview doesnt prioritize com over the code

This explanations is a possibility, and I’ve asked teams before if they are boosting priorities and possibly doing this. I’ve yet to find a team whose priorities or code caused this. Also, this is just as likely in all languages.

From my observations, a 45 second reboot is quite a bit longer than a cRIO warm or cold reboot. That time is closest to the boot time of the dlink. If you can see that the orange light is not blinking, the dlink bridge is not up. Make sure that the dlink power is the boosted supply on the PD and obviously wiggle and squeeze the wires. In NJ, we saw several robots with shorts at the PD end of the cable and several with loose barrel connectors at the dlink end.

Greg McKaskle

13

Perhaps this is related:

http://www.chiefdelphi.com/forums/showthread.php?t=93657

14

Thanks for the link. We are using the power cable included with the DLink, but I will make sure that the connection is secure and robust.

Hopefully the issue is that simple. The fact that the field could tell that our battery voltage had dropped precipitously immediately before losing comms, however, makes that unlikely in my opinion.

15

We had several robot resets in Pittsburgh that were very similar to what 341 experienced.

We have no RS-775’s on our robot (thankfully). 4-CIM drivetrain, 2 RS-540’s in the roller claw, 1 RS-550 for the arm.

The FTA reported the same low voltage reading to us at one point that was reported to 341.

Our first reset was after a hard impact with a tower. Most others occurred during normal operations, with no contact or other anomalous activity occurring at the time of reset. There were a few impacts later on in the competition that did not interrupt robot function at all.

We use C++. Others report using Java.

cRIO is isolated from the frame.

It appears that reports of this occurrence are getting more and more widespread, and the robots experiencing them more and more diverse in design and function. What are the common hardware and software ties that bind these robots and reboots together?

If any other teams have experienced similar issues, please share them.

16

Travis,
A small number of main breakers have a manufacturing defect that makes them intermittent. If you tap the red button and the robot lights flicker, this is likely the cause. Replacing the breaker is the only fix. Other sources of low voltage can be loose connections on the battery terminals. I recommend that a star washer be placed between the terminals before the hardware is inserted. This not only breaks through any surface crud, it also locks the terminals together so that they can’t loosen. Any loose connection or crimp in the robot primary wiring can also cause this brown out condition that takes out the Crio and radio power. Although it doesn’t occur often, some teams add #6 wire to the provided Anderson plug to extend the distance between battery and PD. This is the one place on the robot that wiring should be kept to minimum length as all robot current flows through this wire.

17

Al:

Thanks - we will check these items out first thing in Tennessee - after the rush of the elimination rounds, I actually thought to check how secure the robot’s primary power wiring was…after we bagged the robot. :rolleyes: Usually, that issue is never a problem for us.

I do know all the battery leads were securely bolted to the terminals.

I will likely check and replace the main breaker too, just in case. We’ve got plenty of those handy. I’ll grab a few star washers for good measure.

I also know our radio power adapter is secured and has good strain relief at both ends, but I’ll likely add a dab of hot glue to the barrel connector just to be safe.

18

I am willing to bet it’s the 6 motor drive, combined with the additional current draw of a drivetrain being pushed against and pushing. We did some magic math at the beginning of the season and found that heavily loading all of those motors would lead to pretty quick battery drains (1:30 or so into a match with some other mechanisms running).

Talking to some friends with similar issues, changing the 6 motor drive’s gearing to a lower speed (10 FPS is apparently a good sweet spot) made the issue go away entirely. From what I can recall, you guys are running at 13 FPS, which is much heavier on the motors.

19

It’s not out of the question, but a week+ of practicing (often for longer and/or harder than during a real match) and then perfect operation on practice day suggests, to me, that the issue is elsewhere. If it’s not electrical, or somehow code/firmware related, then the only other possibility is that our bolted frame is losing its stiffness and causing the motors to work harder during turns (though the robot doesn’t jump or otherwise seem to struggle at all to turn).

Besides, I don’t believe that Travis is running a 6 motor drive, nor are either of our teams’ issues happening as an obvious result of pushing matches (it happened to us less than a minute into a match where we hadn’t even touched another robot). We also ran one match with just the 4 CIMs (still geared at 13 fps, though) and the issue didn’t go away.

20

4 motor drive: 2 CIM’s and a SuperShifter per side.

4" wheels - 4 traction, 2 omni, no drop center