We were having similar problems with our radio/cRIO rebooting. We aren’t exactly sure what happened, but usually right after auto or half way through teleop, we would loose comms for 15 or so seconds. The radio power light would go red, and then turn back blue. Upon looking at the driver station log, just before loosing comms, there were a bunch of errors relating to the camera. Simply unplugging the camera ethernet camera resolved the issue. Somehow we must have a memory leak in our vision code causing the cRIO to reboot. but its weird that the radio would reboot as well, as it would not have lost power.
Thanks for the great advice guys, I think during one or two of the matches we did have a loose battery lug, and we also had a loose anderson, we found this out by shaking them while it was on and causing it to cut out. I will try this week to give yall the information. We use labview, I’m not sure what the code exactly looks like because I don’t program, but I will get that info as well. The big question that has me stumped is how most of these problems that yall talked about wouldn’t occur off the field, even when we ran into the wall vigorously a lot? We only verified a D-link reboot twice, and that was on matches that we had definite battery problems. By no means am I an expert on any of this so please explain why it makes sense. Thanks for the input!
One of your mentors asked me that several times, and my answer is still the same: running into the wall is a jolt in one direction. Being bumped by another robot from the side, or from behind, is another direction. If there’s a loose connection that causes problems related to being hit, it won’t necessarily show up with a hard impact to only the front of the robot.
The one time I was close enough to see the speed controllers’ LEDs during one of the on-field episodes, the robot was definitely completely without power immediately after being bumped from behind. Some ten seconds later, a second gentle bump on the corner coincided with the power coming back on. Nothing in software or the field communication can cause that kind of failure.
We only verified a D-link reboot twice, and that was on matches that we had definite battery problems. By no means am I an expert on any of this so please explain why it makes sense. Thanks for the input!
The D-Link rebooting was only verified visually twice (it was mounted on the robot such that the lights are only visible from a narrow angle of view). The real-time field data monitor showed a loss of radio communication every time I was able to watch, and the FTA says it happened every time your robot had a problem on the field, though the logs I was able to review were only clear about the cRIO reboot and were inconclusive about the D-Link (the Driver Station apparently doesn’t ping the bridge during enabled mode, and the FMS stops recording when the cRIO is not responding).
The one “smoking gun” we identified was the intermittent SB50 connection. Right after that penultimate match, it was definitely easy to cause the robot to lose power by pushing on the red wire in the right direction. I understood that the connector was going to be replaced. Afterwards, when the robot had yet another verified radio reboot with associated logs suggesting a complete power dropout, I noted the same loose feel to the red wire’s terminal. I didn’t think to ask whether it was the entire connector and its wiring that had been replaced, or if it was just the housing, or if it was just the terminals and wires. It is possible that the red SB50 shell was reused, and that the underlying fault is that its “spring” isn’t holding the terminal hard enough to the matching battery connector. Or it is possible that the terminal itself wasn’t changed and has a bad surface that doesn’t provide a secure contact.
Being unable to help 4490 correct the repeatedly-appearing problem is definitely something I feel bad about.
Try replacing the buck converter that’s connected to your radio and make sure that the converter case is isolated from the rest of your robot. We saw quite a bit of that on the field at Hub City, as well. We verified that all the connections were done correctly, that the battery was properly secured, and replaced the radio (just in case).
The FMS cannot cause your battery to fall out.
There is a good chance your wiring was pulled unnoticeably loose and caused a slight loss of power during some particular hits.
Possible causes: loose lugs somewhere between battery and PD board, loose/improper crimps/WAGO connections
Just because it looks correct, doesn’t mean it is.
The FMS doesn’t directly communicates with the robot.
The FMS will tell your DS computer when to be disabled, auto, and tele modes, but it is your DS computer that sends all packets to the robot via the wifi radio on the field. The Practice match sends the same info. The code on the cRIO runs the same whether on field or in pits.
If your battery is able to fall out once, it is likely that it was loose at other times. When not held firmly in place, the battery is a 12 lb. sledge hammer banging into your other components. It easily knocks wires out of Wago terminals, knocks ribbon cables off, knocks wires and terminals against the frame, etc. As Alan said, its direction of travel will also depend quite a lot on the type of hit.
Greg McKaskle
That makes a lot more sense, so what we really need to check next is our mainbreaker crimps, and connections, and our converter for the d-link, it is also very possible for the battery to mess up connections because we had our battery board fall down on at an angle and there was no visual damage. We will also try a good shaking of the power wires to see if we can get the problem to re occur so we can pinpoint it. Thanks for all the feedback guys, and a special help to Alan for all of his hard work and helping us find the problems!
Did each of the SB50 terminals “snap click” into its proper place within the housing; i.e., toes over the end of the diving board? Sometimes if the terminal is bent slightly, the “snap click” does not happen and the terminal can back out a little, just enough to make intermittent contact. A good whack could open such a contact, and another good whack later could close it again. This can be hard to find until the terminal backs out far enough to cause the connection to open statically.
Guys,
There are two power supplies on the PD, one for the cRio and the other for the radio. while similar in design these are separate supplies. They will stop operating when the battery voltage input to the PD falls below 4.5 volts. Since they are separate supplies they may stop at different times. Loose battery connections, poor crimps, loose hardware on the battery and a bad SB50 connector can all contribute to losses that will shut down these supplies. However, even these supplies cannot stay up with no power to the robot. So, as inspector and electrical guy, I work this list…
- Are the terminals on the battery tight and non-moving? Typical of carrying the battery by the wiring. Cannot be repaired, battery is likely candidate for recycling.
- Do the wire terminals rotate on the battery terminals? Typical for these battery terminal types. The simple addition of a star washer between the terminals will prevent this.
- Is the hardware tight on the PD? Tighten using metric nut driver.
- Is the hardware tight on the main circuit breaker? Tighten using english nut driver.
- Is the wire secure in the terminals used? Tighten and strip to correct length if needed. SLA connector in particular have an optimum strip length that is needed for secure electrical connection.
- Is everything correctly insulated? Electrical tape, not duct tape is your electrical friend.
So we have been running tests with a multimeter and it doesn’t seem to be a PD board problem, and I replaced all of the power connections to the CRIO and when I got ready to drive I connected, enabled, then drove forward slightly and rebooted. The CRIO didn’t reboot until enabled and drove? We also had the whole power go out, and I am not certain what caused that, it was most likely the battery because the Anderson was at a weird angle on it. Aside from that, could the problem be in the Ethernet cord from the D-link to the CRIO? Because we did tether when we got the robot back and it drove fine, so could it possibly be a bad Ethernet cord causing it? We went ahead and replaced it but could it have been the problem?
That could be caused by a bunch of things. Two seem likely enough to be worth investigating. Maybe your battery was essentially dead and couldn’t provide enough current to run your drive motors. Maybe you have a couple of frame shorts that bring the cRIO power return to a high enough voltage that 24v on the power supply isn’t sufficient to keep it running.
A bad Ethernet cable won’t cause a cRIO reboot.
One thing I’ve noticed is that this problem seems very particular to the 4 slot cRio and will occur with very secure power connections. I’ve seen this happen to about 11 robots over the past two years, where the robot will be hit rather hard and will reboot for no apparent reason.
I even troubleshot this with a rookie robot where we discovered that if we dropped the robot (above 1" drop height) while the cRio was relatively not constrained to the robot, the cRio would reboot. Driving the robot at fast speeds and stopping quickly or tugging the wiring in different ways (which would have exacerbated any loose wiring) did not cause the cRio to reboot. However, driving at high speeds into any solid surfaces or getting the robot hit (even with a swift kick to the bumpers…) would cause the cRio to reset based on the force used.
Does the 4 slot cRio have a force-sensitive sensor inside of it that tells it to reset if a threshold is met? This seems much too silly to be the cause, but it seemed too consistent to not be the problem. It was even a problem for Team 1410 at the 2014 Utah Regional, where our cRio reset after hard connects with other robots.
A main circuit breaker with a certain kind of fault can do a good job of detecting impacts in specific directions and briefly shutting off before returning to normal. A piece of metallic debris near certain components, or a stray bit of wire strand, can cause a loss of power when the robot is jerked in a specific way.
There really ought to be a better way of finding such intermittent situations than just banging and tugging on things at random, but when pressed for time anything more systematic might not be worth trying.
Do you use a smart dashboard…
Our robot would reboot after autonomous because of it.
Turns out only Java Users can use it. So my team decided to set it to false along with other teams and we are all driving “blind” basically.
At the Utah regional right? I was wondering if you ever fixed that issue, glad to hear you got it figured out.
The cRIO does not have a shock sensor. The data sheet showing the shock and vibe testing is here … http://sine.ni.com/ds/app/doc/p/id/ds-354/lang/en.
If you are in a relatively static-free environment, you can clean debris from the cRIO in about five minutes. I almost always find lots of glitter and wire bits. I have seen cases where the reboots ceased for the remainder of the matches. I have also seen cases where nothing improved. Procedure link is below.
http://digital.ni.com/public.nsf/allkb/FA1B856FC4EB6F9D86257673007935A1
Greg McKaskle
Our team is in Michigan – the issue I mentioned occurred at the Southfield District in Week 1. The intermittent chassis short was caused by some wiring that had previously provided power to a custom circuit (LED string), which had been removed after it was damaged during a practice match.
What was the issue seen at Utah?
Canon,
The issue you describe is usually attributed to one or more of the following problems in no particular order.
- The cRio is bumping into robot frame or is attached to robot frame electrically.
- Wire whisker is touching the opposite polarity on the power supply wiring at either the cRio end or the PD end.
- Power wiring is not inserted into the connector properly, a tug on the wiring will show this problem.
- Rare manufacturing defects have been observed on the main breaker. while the robot is thruned on and booted, trying lightly tapping the red reset button on the breaker. If the lights on the robot blink, replace the main breaker.
- Improperly terminated battery wiring. Check everything from the battery to the PD. If it moves, it is intermittent.
- An intermittent short exists in the branch wiring or you have a defective motor controller or motor that is showing a short in one direction. Try removing all breakers and reinserting one at a time to locate the offending branch.
- If you are using multi motor transmissions, check that the motors are wired properly so that they are not running in opposite directions.
- You have an intermittent power wiring to the DLink or you have not wired the power convertor to the dedicated +12 volt output on the PD.
When you have a shock intermittent, use a large screwdriver to tap the handle (or other insulated tool) on various parts of the robot until you find the sensitive area of the robot. You may find swarf in the cRio, PD or DSC or you may find improper wiring on the outputs of one of the cRio modules. If the condition only occurs when you are driving, the problem is in the drive train or the cRio to robot frame short.
There has been a lot of mention of a frame short, could we test for this by testing the frame with a multimeter? we did that and it wasn’t conducting anything, we checked the battery and all the connections to the pd board with it and got a consistent 12.5 volts. After the testing when the robot drove forward and shut off, we tried turning it back on, it took a few times but after it did turn back on we had no connection whatsoever, we tried rebooting, reset the radio, re-deployed code, exited the driver station then came back in. Under the diagnostic page on the driver station there was no light for the bridge, but there was a red light beside “robot”. I’ve been trying to get it to shut off by shaking and dropping, and it still isn’t rebooting like it is when I drive it. Is there anything else in the driver station we can use to diagnose the problem? Thank you guys so much, the team really appreciate’s your help and GP!
Check your event log from the match and see what happened:
http://wpilib.screenstepslive.com/s/3120/m/8559/l/97119-driver-station-log-file-viewer
We had a similar problem once, it was due to binding in the drive train and when the driver moved the joystick forward and reverse rapidly enough, the current draw on drive motors caused a drop in voltage significant enough to cause the crio to reboot. The event log will show if voltage drops are causing this.