Communication lost

[Al Skierkiewicz]

Alan,
Agreed, but I have seen times when the current spikes are so bad, the PD power supplies do reboot and therefore so follows the Crio, Dlink and camera reboots.

[Mark 3138]

How long does it take for the problem to start. Does it start immediately when starting to drive or is it after an amount of time driving.

What is the battery voltage when the problem occurs?

If the battery voltage is adequate when the problem occurs, make sure that the battery voltage remains good to and thru the PD board and to the Crio. If you see a substantial split in voltage between the battery and the components look for a loose connection or possibly a bad 120 amp power switch.

[dellagd]

It it the radio or the crio rebooting?

For example, when the problem occurs, do you loose connection with the router (like the Icon in the Windows 7 tray would indicate) or is it just the Crio that reboots and you stay connected to the robots router?

Open up netconsole on your driver station and replicate the problem. Watch the feed of data and see if the crio reboots if you are staying connected to its network.

As a side note, if you are not, you should always be using the 12v to 5v converter for the D-link and NOT the 5v out on the power distribution board. The 5v out is not regulated, and therefore varies with the voltage of the battery. On a high power draw (like turning with high friction wheels, causing the motors to work more), the battery voltage will actually drop significantly, and with an unregulated 5v out to the d-link, could easily cause the d-link to reboot.

[Dale(294engr]]

Quantitative measurement of friction using Robot Battery Current:

Friction is directly proportional to current in Amps drawn from robot battery

(test control: fresh fully charged battery OR better: Lab Power Supply set at 12.6v
[variable current limit feature permits safer testing than with battery]
10A continuous is adequate for non loaded, (+ ~200A for ~.1 sec surge)
50A minimum for Loaded (200A would be close to optimum) + ~500A ~.3sec

(test control: Vbatt=12.6 fully charged, record this, great to have a Lab PS for this)

measure quiescent current robot draw power on, all motors off (~1-2A)

quiescent subtracted from ea test to isolate Friction caused Current relevant to ea motor test

Baseline Friction measure & record: (every season, every motor!)

measure no-load, full speed currents at 12.6v, forward & reverse, ea motor alone (= friction of ea motors brushes, bushings)
repeat after adding gears, (no chain, etc)
repeat with chain (drive elevated)
repeat for each range of gears (more gears = expect more friction)
repeat for other drive side

note: Left & Right side Currents (friction) should be close, higher one has issues if more than a few .1's amps different

For early detection "Prediction" of drive train problems (anything with motor)

Use this baseline during life of robot and after ea match when possible,
repeat off floor low gear full speed, fwd & reverse; compare to baseline.

Upward creeping current value indicates potential problem(s)
1A above baseline = needs immediate serious attention
- something is loose, misaligned, worn, broken, shorted (turns in motor)

(option: correct for battery voltage testing below 12.6v: lower batt voltage produces lower currents NOT representative of lower friction!

(test voltage to be closely controlled as practical or compensate for it (add proportional difference) be creative: Dedicate a Battery for friction testing i.e. always on charge, never used for match at this regional, etc.)


For full friction loaded competition scenario:
H/W Use 12 bit A/D input across a .001 ohm "shunt" resistor
(poor mans 1milliohm shunt = portion of #6 AWG with carefully positioned measurement wire attachment ~1' apart TBD: use ammeter to calibrate, a Hall Effect clamp-on or shunt type, Scale: 1mV measured = 1 amp robot current draw

S/W: write a program to sample A/D each ~.1s, store in an array to be retrieved post run (sample rate depends on how much memory is free, shorter times increases detail of current usage at cost more memory 4 or 5 significant digits stored after calibration (if not close to 1 milliohm) is sufficient.
Import to Excel and graph it! The result is very instructive (Robot CAT scan!!)
can be used for early problem detection / avoidance including under charged, weak, & defective batteries i.e. average voltage drop when loaded increased below expected (from increasingly high internal resistance when fully charged)

(Calib is a luxury; ballpark & consistent is most important = relative friction)

At competitions serving as Robot Inspector I use a cheapo Harbor Freight DMM on +-199.9 mV scale with a 1 milliohm shunt = +-199.9 Amps
with a .1A resolution. I carry a ring terminal piece of 6AWG wire previously calibrated connections, but it requires adding it to the robot =time consuming, time being precious I usually use meter across 120A main breaker as ~.001 ohm shunt! When a good 120A breaker is closed it reliably & repeatably measures a bit under .001 ohm (~.096 ohm) Think about it.. if it varied, so would trip current! so mfr's commit to make it repeatable.

This test can also be used to detect a degrading or defective bound 120A breaker-- i.e. if voltage drop across 120A breaker suddenly increases (say doubles) it's bad, & beginning to limit robot performance so needs replaced, otherwise it provides a quick easy convenient ballpark relative measure of robot current = friction gauge of potential motive drive problems


measure A/D in ~20msec intervals to capture surge currents

note 131A ea CIM stall current: 4 CIMs =521A surge for ~.1 sec unloaded, typically ~.3s for robot elevated, increases rapidly against defending robot or field element hits, until breakers begin cycling.

BTW our 40A breakers hold forever at 40A (that's how they are spec'd)
begin cycling at ~50A (~25% over rated) holding 50A for several seconds before cycling, time shortens as over current increases (see mfr curves)

At 131A each 40A breaker holds of a few tenths of a second or more!
so Battery does see surges of hundreds of amps up to ~.3sec without breakers cycling!! causing Vbatt to Sag!! = decreased voltage to entire robot lessening top motive power & top speed due to current limit of less voltage

Note:
nominal fully charged battery and wiring:
voltage at battery drops 1v for each 50A Robot current drawn!
Vbatt at 50A = 12.6-1v (per50A) =11.6v
100A = 10.6v 150A =9.6v 200A=8.6v 250A = 7.6v 300A = 6.6v

See why we have "resets" under "loaded" drive train scenarios?

or binding bearings, chains, gears, misalignments? each of which lengthen the
surge current period & amplitude, collectively may induce one or more module reset(s) "randomly" as a result of "dynamic action": turning, opposing defense, full fwd to full Reverse, binding, etc.

These Vbatt drops are typical motor startup current "surge" Vdrops.
built in design remedy provide some level of protection:
Voltage hold-up Capacitors used in ea electronics module determine when one may reset due to "brown out" (Capacitors store energy to help supply current to Radio, CRIO devices during surges to keep internal voltages higher then the batt terminal voltage drops sags "glitches" for brief periods, but has practical limit)

There were a few years when CPU + aux electronics were supplied by a separate 6 cell battery-- avoiding motor start-up surge scenario that causes Radio, CRIO, etc. resets.

Perhaps time to re-consider? random resets are hard to troubleshoot in heat of competition - this year is BIG case in point.

Electronics have become very complex so also troubleshooting them

(re: surge current: 4WD in turns is worst (highest) in amplitude and time,
6WD with center offset downward vastly improves performance (less surge),
8WD depends if all wheels in plane but likely near 4WD surge demand in turns)

[Brandon Zalinsky]

When we drive, our battery voltage drops from about 12.7 to about 9 or 10. Would that be normal, or is that enough to re-boot the cRIO/D-Link? Another thing, with ours, if you leave it for a bit, comm will come back. We've tried different batteries and it happens with all of them.

[dellagd]

Its better practice to start your own thread for your own problem, but whatever-

If you have the D-link connected through the 12v to 5v converter module, then you should be fine. Not sure though about what would happen if you did not

[Alan Anderson]

Originally Posted by Flak-Bait

When we drive, our battery voltage drops from about 12.7 to about 9 or 10. Would that be normal, or is that enough to re-boot the cRIO/D-Link?

It's normal for the battery voltage to drop that low when you're pulling a lot of current with the drive motors. It's not normal for anything to reboot until you've managed to get the voltage down below 5 volts. If you're getting reboots at 9 volts, you're probably not connecting to the right output of the Power Distribution Board. Check to make sure you're getting 12 volts on the dedicated boosted radio supply going into the 12-to-5 converter, and make sure you're getting 24 volts on the dedicated boosted cRIO supply.

[Al Skierkiewicz]

Originally Posted by Flak-Bait

When we drive, our battery voltage drops from about 12.7 to about 9 or 10. Would that be normal, or is that enough to re-boot the cRIO/D-Link? Another thing, with ours, if you leave it for a bit, comm will come back. We've tried different batteries and it happens with all of them.

Brandon,
As Alan said, this is normal. However, short term dips in the battery voltage don't always show up on the dashboard because they are so short. They sometime occur between samples. That being said, the 5 volt regulator that feeds the Dlink will drop out around 7 volts. That is why it must be connected to the boosted +12 volt radio output of the PD. This power supply continues to put out +12 volts until the battery falls to 4.5 volts or less. However, when the Dlink reboots due to low voltage, the reboot time is 50 seconds. If you know exactly how long your outage lasted, we can determine more accurately the exact cause.

[Skehtb]

I went to check some things for you guys, and to found out what the problem. Apparently, some other team members tried to fix the problem by themselves (they thought the problem is a short circuit, so they isolated the router), and now we can't get any communication at all - few seconds after we find the router's network on the computer, the communication is lost and a red light flashes on the router.

nevermind, problem solved. now i'll check what happens when we drive our robot, and other details you asked for.

[Al Skierkiewicz]

Originally Posted by Skehtb

Apparently, some other team members tried to fix the problem by themselves (they thought the problem is a short circuit, so they isolated the router), and now we can't get any communication at all

What do you mean by "isolated"?