Communication lost

Every time we drive our robot, it suddenly stops moving and the communication sign on the computer (the one that tells if there is communication between the robot and the computer) turns red. It only happens when we are activating the drive train - “moving” (4 CIM motors). When we shoot, collect balls and any other thing except for drive - it doesn’t happen!
I hope you understand the problem, we realy need your help (off-sesason event in 3 weeks!)

Thanks!!

Hey-

Are you using a 2CAN? We were, and we traced the problem to our 2CAN causing us very similar problems. Try using CAN over the serial port on the crio (bypassing the 2CAN).

This happened another time and was solved by this

Skehtb,
You don’t elaborate on what other issues you are seeing so I will make some general statements.
If the robots stops functioning for something like 20-50 seconds, then you are drawing the battery voltage down to the point that the Crio is rebooting.
If the robot stops for ~50 seconds then the power supply to the radio is causing it to reboot.
If the robot just stops for about 3-4 seconds and you are using Jaguars, then you are faulting the Jags (probably to high current). If that is the case, the LED on the Jags will flash red during the 3-4 seconds.
As to what could be the cause, you could have batteries that are not fully charging or if they are old, may not have the capacity to drive the robot any longer. If you rule out the battery, then loose connections to the battery, the PD or the main breaker can also cause these issues. If the robot just stops but you can start driving almost immediately, the Crio has an inhibit function based on the battery connection you make to the analog module in slot 1. If that voltage falls to 5.5 volts, the Crio inhibits all output. Since you are in Israel, can we assume you made it to an event outside of Israel? If so, your drive train may have been damaged during transit and now has high friction due to the damage. Check the drivetrain by lifting the robot and moving the wheels by hand. If you are using a transmission that you can slip into neutral, the wheels should move very easy. Depending on your design, the friction could be caused by misaligned drive shafts, tight chain, bent wheel shafts or contact between drive components and robot chassis. In some case, stretched chain could be causing the drive train to bind or hang up on other parts.

Great advice on this thread.

We also had a similar problem a couple of year back and traced it to the cRio not being isolated from the frame, a screw under it had worked loose. When we removed this screw the problem went away.

Since this event we have always made a large effort to make sure the cRio stays isolated.

We’re having the exact same problem. Could it be the wireless bridge? We’ve already broken one this year.

Brandon,
It is really difficult to diagnose problems without more data. In your case, it is possible but watch the lights on the Dlink. The robot is talking to you, you must listen to find out what might be wrong.

The Crio is isolated, but i’ll check it again.

The problem appears with different batteries. We found out that the robot doesn’t stop functioning when we drive it forward. It happens only when try to turn it (and I’m pretty sure it happens only when it truns to the left), drive it backwards, or stop it after driving forward. I’m not sure about how long the robot stops functioning, but its definetly more than 20 seconds. I’ll check it.
I don’t understand how high friction or bent wheel shafts can cause the problem. And if it changes anything - our drivetrain is 8wd and this season we did not participated in an event outside of Israel.

High friction or bent wheel will cause the motor(s) to stall. Stall draws tremendous current which could draw down the battery voltage or cause the Jaguars to cut off due to over current.

Mike is correct. However when the condition exists only on turns, I am going to guess you have sticky tires and are using a tank turning for steering. When this type of drive mechanism is used with high friction tires, the motors do tend to stall or get very near. A CIM motor stalls at 131 amps so with all four motors in stall, you are drawing so much current from the battery that the terminal voltage falls below the point that the power supplies in the PD can keep running. If both the Crio and the Dlink reboot, the Dlink takes longer to come back. Also, and I have to say this, if the Dlink is not powered from the dedicated +12 volt radio output on the PD, then small fluctuations in battery voltage will cause the Dlink to reboot.

Look for a shorted wire between one of the left side drive motors and its speed controller. You might be getting voltage on the frame when a motor drives backwards, and another shorted frame connection somewhere (likely a power return or “ground” wire) could be causing your D-Link or cRIO to reset. Or you might have cross-wired motor leads, with one motor connecting to two different speed controllers. That results in very weird symptoms that can sometimes match what you’re describing.

You can start troubleshooting electrical issues by removing the circuit breaker from the power branch to one speed controller at a time and seeing if the problem can be narrowed down to one motor’s circuitry.

An easy way to test what Al is suggesting is to put some blocks under your robot so that none of the wheels touch the ground then “drive” the same manouvers that cause the problem. You now have no traction so none of the motors should stall due to the choice of wheel tread. You may also find certain wheels that don’t turn, or do not turn smoothly, due to high friction in gearboxes or some other issue.

Good luck.

Phil

When we tried to “drive” the robot while the wheels were “in the air” (not touching the ground), nothing happened - it worked smoothly. We lost communication only when we drove the robot on the ground.

Watch the lights: the big orange Robot Signal Light, the ones on the D-Link router, the ones on the cRIO. They’ll give you good clues as to what is going on with the robot when you lose control.

Your symptoms are starting to point more clearly to a power issue. One of the common problems teams have is connecting the robot radio incorrectly. Can you describe exactly how the D-Link is being powered? Specifically, where is it connected to the Power Distribution board, and what else is connected to its power input?

This is pointing more and more to drivetrain friction. Either misalignment or turning frictions are high.

Al and I are not really disagreeing here. Excessive drivetrain friction will drag the battery voltage down, but that alone will only cause a momentary loss of control. It shouldn’t cause a complete loss of communication. If things are properly wired, the control system is designed to shut down the motors before the voltage gets too low to maintain its operation. A robot that reboots when you try to turn on the ground, but works okay when up on blocks, has more than one thing wrong with it.

Alan,
Agreed, but I have seen times when the current spikes are so bad, the PD power supplies do reboot and therefore so follows the Crio, Dlink and camera reboots.

How long does it take for the problem to start. Does it start immediately when starting to drive or is it after an amount of time driving.

What is the battery voltage when the problem occurs?

If the battery voltage is adequate when the problem occurs, make sure that the battery voltage remains good to and thru the PD board and to the Crio. If you see a substantial split in voltage between the battery and the components look for a loose connection or possibly a bad 120 amp power switch.

It it the radio or the crio rebooting?

For example, when the problem occurs, do you loose connection with the router (like the Icon in the Windows 7 tray would indicate) or is it just the Crio that reboots and you stay connected to the robots router?

Open up netconsole on your driver station and replicate the problem. Watch the feed of data and see if the crio reboots if you are staying connected to its network.

As a side note, if you are not, you should always be using the 12v to 5v converter for the D-link and NOT the 5v out on the power distribution board. The 5v out is not regulated, and therefore varies with the voltage of the battery. On a high power draw (like turning with high friction wheels, causing the motors to work more), the battery voltage will actually drop significantly, and with an unregulated 5v out to the d-link, could easily cause the d-link to reboot.

**Quantitative measurement of friction using Robot Battery Current:
**
Friction is directly proportional to current in Amps drawn from robot battery

(test control: fresh fully charged battery OR better: Lab Power Supply set at 12.6v
[variable current limit feature permits safer testing than with battery]
10A continuous is adequate for non loaded, (+ ~200A for ~.1 sec surge)
50A minimum for Loaded (200A would be close to optimum) + ~500A ~.3sec

(test control: Vbatt=12.6 fully charged, record this, great to have a Lab PS for this)

** measure quiescent current robot draw power on, all motors off (~1-2A)**

quiescent subtracted from ea test to isolate Friction caused Current relevant to ea motor test

Baseline Friction measure & record: (every season, every motor!)

measure no-load, full speed currents at 12.6v, forward & reverse, ea motor alone (= friction of ea motors brushes, bushings)
repeat after adding gears, (no chain, etc)
repeat with chain (drive elevated)
repeat for each range of gears (more gears = expect more friction)
repeat for other drive side

note: Left & Right side Currents (friction) should be close, higher one has issues if more than a few .1’s amps different

For early detection “Prediction” of drive train problems (anything with motor)

Use this baseline during life of robot and after ea match when possible,
repeat off floor low gear full speed, fwd & reverse; compare to baseline.

Upward creeping current value indicates potential problem(s)
1A above baseline = needs immediate serious attention
- something is loose, misaligned, worn, broken, shorted (turns in motor)

(option: correct for battery voltage testing below 12.6v: lower batt voltage produces lower currents NOT representative of lower friction!

(test voltage to be closely controlled as practical or compensate for it (add proportional difference) be creative: Dedicate a Battery for friction testing i.e. always on charge, never used for match at this regional, etc.)

**For full friction loaded competition scenario:
**H/W Use 12 bit A/D input across a .001 ohm “shunt” resistor
(poor mans 1milliohm shunt = portion of #6 AWG with carefully positioned measurement wire attachment ~1’ apart TBD: use ammeter to calibrate, a Hall Effect clamp-on or shunt type, Scale: 1mV measured = 1 amp robot current draw

S/W: write a program to sample A/D each ~.1s, store in an array to be retrieved post run (sample rate depends on how much memory is free, shorter times increases detail of current usage at cost more memory 4 or 5 significant digits stored after calibration (if not close to 1 milliohm) is sufficient.
Import to Excel and graph it! The result is very instructive (Robot CAT scan!!)
can be used for early problem detection / avoidance including under charged, weak, & defective batteries i.e. average voltage drop when loaded increased below expected (from increasingly high internal resistance when fully charged)

(Calib is a luxury; ballpark & consistent is most important = relative friction)

At competitions serving as Robot Inspector I use a cheapo Harbor Freight DMM on ±199.9 mV scale with a 1 milliohm shunt = ±199.9 Amps
with a .1A resolution. I carry a ring terminal piece of 6AWG wire previously calibrated connections, but it requires adding it to the robot =time consuming, time being precious I usually use meter across 120A main breaker as ~.001 ohm shunt! When a good 120A breaker is closed it reliably & repeatably measures a bit under .001 ohm (~.096 ohm) Think about it… if it varied, so would trip current! so mfr’s commit to make it repeatable.

This test can also be used to detect a degrading or defective bound 120A breaker-- i.e. if voltage drop across 120A breaker suddenly increases (say doubles) it’s bad, & beginning to limit robot performance so needs replaced, otherwise it provides a quick easy convenient ballpark relative measure of robot current = friction gauge of potential motive drive problems

measure A/D in ~20msec intervals to capture surge currents

note 131A ea CIM stall current: 4 CIMs =521A surge for ~.1 sec unloaded, typically ~.3s for robot elevated, increases rapidly against defending robot or field element hits, until breakers begin cycling.

BTW our 40A breakers hold forever at 40A (that’s how they are spec’d)
begin cycling at ~50A (~25% over rated) holding 50A for several seconds before cycling, time shortens as over current increases (see mfr curves)

At 131A each 40A breaker holds of a few tenths of a second or more!
so Battery does see surges of hundreds of amps up to ~.3sec without breakers cycling!! causing Vbatt to Sag!! = decreased voltage to entire robot lessening top motive power & top speed due to current limit of less voltage

Note:
nominal fully charged battery and wiring:
voltage at battery drops 1v for each 50A Robot current drawn!
Vbatt at 50A = 12.6-1v (per50A) =11.6v
100A = 10.6v 150A =9.6v 200A=8.6v 250A = 7.6v 300A = 6.6v

See why we have “resets” under “loaded” drive train scenarios?

or binding bearings, chains, gears, misalignments? each of which lengthen the
surge current period & amplitude, collectively may induce one or more module reset(s) “randomly” as a result of “dynamic action”: turning, opposing defense, full fwd to full Reverse, binding, etc.

These Vbatt drops are typical motor startup current “surge” Vdrops.
built in design remedy provide some level of protection:
Voltage hold-up Capacitors used in ea electronics module determine when one may reset due to “brown out” (Capacitors store energy to help supply current to Radio, CRIO devices during surges to keep internal voltages higher then the batt terminal voltage drops sags “glitches” for brief periods, but has practical limit)

There were a few years when CPU + aux electronics were supplied by a separate 6 cell battery-- avoiding motor start-up surge scenario that causes Radio, CRIO, etc. resets.

Perhaps time to re-consider? random resets are hard to troubleshoot in heat of competition - this year is BIG case in point.

Electronics have become very complex so also troubleshooting them

(re: surge current: 4WD in turns is worst (highest) in amplitude and time,
6WD with center offset downward vastly improves performance (less surge),
8WD depends if all wheels in plane but likely near 4WD surge demand in turns)

When we drive, our battery voltage drops from about 12.7 to about 9 or 10. Would that be normal, or is that enough to re-boot the cRIO/D-Link? Another thing, with ours, if you leave it for a bit, comm will come back. We’ve tried different batteries and it happens with all of them.