serious problem found - robot controller resets when jarred!

Technical Electrical
1

EDIT: DID SOME MORE TESTING IN OUR LAB - THE PROBLEM WE HAD IS NOT DUE TO THE BREAKERS RESETTING THE ROBOT CONTROLLER -

SEE MY POST ON PAGE TWO THAT BEGINS WITH THE LINE:

“OK I did some more testing in my lab.”

the subject line is a little misleading. The 20A breakers open when jarred causing the robot cotroller to lose power for an instant, and reset!

At the buckeye regional team 578 observed, in two matches, our robot restarted automode halfway through the 15 second period - both times after the robot was smacked.

We discovered the 20A and 30A circuit breakers open for an instant when jarred. You can try this for yourself - put a continiuty meter across a 20A breaker, then flick it with your finger, or tap it on the table - the circuit will open briefly.

These breakers are not defective, its a characteristic of the way they work - Bimetal strips that act as springs, and bend (pop) the other way when too much current heats them up.

This is what causes it to open on impact - it is esentailly a switch held closed with a small spring. When the impact force overcome the pressure of the ‘spring’ the breaker opens. The breakers look like a fat pizza box. We took one apart - the switch opens and closes in the same direction as if you were opening a pizza box.

What this means is, the robot controller power can be interrupted if the robot is hit, hits a wall, bounces hard on the ramp… when the power is interruped like this the controller resets, and if it happens in the 1st fifteen seconds, your auton mode code starts over from the beginning (we tested it - the controller does reset).

We emailed FIRST about this, and recommended all teams be allowed to use a 20A fuse (like the ones in the spikes) in the fusebox for the robot controller power. Its ok if your motors and other devices glitch for an instant when hit, so only the robot controller needs a fuse instead of a breaker.

if your bot is smacked at any point during the 2 minutes, and it resets, it will cost you several seconds of dead time, as the controller restarts and renegotiates the communication link - its more serious during auton mode because the automode sequence of events you programmed your bot to follow will start over from the beginning.

if FIRST wont let us use fuses instead of the breakers for the robot controller circuit, the one thing you can do is make sure your fuse panel is oriented, screwed to a vertical surface in the bot, so the ‘pizza box’ shape is flat in the robot. This will lessen the susceptability to impacts in the X and Y planes, and put it on the Z plane - so if your bot is dropped (gets lots of air) then the breaker will be most likely to open, and less likely to open from a front/back or side impact.

another solution is to shock mount your breaker panel.

The fuses that are in the spikes are 20Amp (and they are supplied by FIRST). The fuse is simply a strip of metal that melts when too much currrent flow (it gets hot). The fuse cannot reset. There is no logical reason why you should not be able to use a fuse for the robot controller. There is no condition in which the controller should be drawing more than 20 amps (unless you have loose metal bounceing around inside your robot.

I have not heard anything back from FIRST yet on the Q&A section in the robot rules section - they have not even posted the question yet. If you dont see a respose from FIRST on the Q&A forum then raise the issue yourself with the judges/inspectors at your next event - advocate a fuse be allowed in that circuit (for all teams) - or at least orient your fuze panel so the breakers are flat.

If you are trying to achieve king of the hill in the last seconds of your final match for the regional trophy, and you are smacked at the top of the ramp, breaker opens and your controller resets for several seconds

it will be all over.

-Ken Wittlief sparkie engineer team 578 ‘Blue Lightning’
NASA/Gleason/Fairport - see you in Toronto!

2

edited to fix subject line

3

i also heard that a team at the buckeye regional found that if you jarr the modem enough you can actualy cause a basic init error and cause your robot to reset.

So it seems that maybe our problems of robots dying last year from the 120 amp breaker getting jarred could have been an even bigger problem then previously expected.

4

Ken,
Are you sure the resets are due to the circuit breaker opening? I’m not surprised if the breakers open momentarily, but have you been able to run your robot into a wall or something and see it happen? There is a really monsterous capacitor inside the robot controller and I’d be a little surprised if it couldn’t overcome a very momentary loss of power like that.

Our team has had some really solid hits to the walls both from autonomous and driver controlled and we’ve never seen resets due to shock like you describe. By any chance to these resets only happen on the ramp? If so it could be the static discharge that’s getting you. There’s another thread that Al from our team started about this problem. We also discovered that the reset pin on the RC is exposed through the programming port and is highly susceptible to noise. On our robot controller, merely having a serial cable plugged into that port and having it unconnected on the other side will put the RC into a constant reset state.

5

The two matches where we saw our bot restart auton mode from the beginning, one the L/R switch we have on our bot was suppose to be on L, and the student set it to ‘KILL’ - :c)

the bot backed up like its suppose to, turned right 45° instead of left, then slammed full speed into the railing right in front of the judges table, then the bot reset, backed up again, turned 45° right again and slammed into the opponets operator station, leaving a 2 foot scratch in the plexiglass (Buckeye match 57)

we never made it to the ramp.

The other match was similar - we were pushed backwards down the ramp then hit by another bot at the bottom, and our auton sequence restarted.

We have nothing connected to our program or tether connectors during a match. In fact, i would not be surprized if this is the real cause of bots resetting at the top of the ramp (they get jarred when they land hard or hit the stack, lip, other bot…)

Other teams have reported seeing bots get hit, then go dead (light goes off) for a few seconds, then restarting.

And yes, I did test this with a robot controller on the lab bench with a 20A breaker in series with the 12V supply - you flick the breaker with your finger, or tap it on the bench (not that hard either) and the robot controller resets, and renegotiates the radio link (takes a few seconds).

This is a definate problem with using the breakers to protect the robot controller (instead of a fuse).

BTW - I have taken one of this years robot controllers apart (we sheared a fuse - now using it for a spare on old bots) I dont recall seeing any really big caps in there.

6

also - if you have your breakers oriented flat, or sideways, they are less suseptable to a front impact. We have our breakers vertical, narrow edge front to back - so they would be most suspectable to a side impact - but our bot still reset when it hit the railing head on.

7

There is (or at least should be) a capacitor in the RC that smooths out the power supply just for this reason. Momentary drops in power are pretty normal, and shouldn’t cause a full reset.

It sounds like either this RC doesn’t have one or it just isn’t working right.

Try this: Attach a capacitor (+12 volt and a couple hundred micro farads) across the power terminals of the RC. Power up and test it again. The capacitor should supply power for a second or two if the power flicks out and the RC will never know the difference.

Do you have another RC you can test?

-Andy A.

8

I like this idea, but isn’t this illegal for competion. I see this for troubleshooting purposes only.

9

The RC I tested on the bench is not the one that was in our bot when it reset, so I have bench tested one, and ‘field tested’ another.

I dont know why there would be a large electrolytic cap in the robot controller - Its powered by a battery, which would have very little ripple - but I will open our spare one up again and look for large caps.

The RC takes quite a bit of power - it powers the radio modem, any sensors you have on the same breaker, any servos connected to the PWM outputs - so some robot configurations might drop out faster than others.

But even still, caps are intended to smooth out noise and ripple on the power supply, not to hold it up during power interuption.

If anyone has the ability, please verify my results in your lab. I believe this is potentially a serious problem for all teams - having your auton mode restart, or loosing the link for several seconds during a match can result in your bot being damaged in a collision, or loosing the match in the last few seconds.

10

adding a cap would be against the rules - I have formally asked FIRST to allow us to use a fuse instead of the breaker in this one circuit (fits in the fuse box the same as the breaker)

no response from them yet.

11

There has to be a cap in the controller to filter out noise from the motors. The secondary effect would be to smooth out momentary power interruptions. There has to be some kind of regulator in there to step the supply voltage down to 5V, I bet theres at least a diode and a filter cap on the supply-side.

I know this is a dumb question, but did you check the leads on the battery? We had a loose screw on a battery terminal that was occasionally causing it to reset. Or maybe a connector on the breaker panel is loose? We saw that come up as well. If tapping the breaker is causing the reset then maybe one of the leads is loose or not seated in the connector all the way.

12

you can take a 20 or 30 A breaker, put alligator or push on test leads on it, and connect it to an OHM meter (that beeps for continuity)

and tap it on a hard surface, the breaker will open. Maybe I will take a JPEG of the one we cut open so you can see whats inside the breaker

there is a big contact slug on a fairly thin sprung piece of metal.

The breakers would make an excellent impact sensor :c)

13

*Originally posted by seanwitte *
There has to be a cap in the controller to filter out noise from the motors.

Yup, we have a controller apart here (it’s not this year’s, but I doubt they’ve changed much) and it has two good size caps near the power connector: a 100uF and a 1000uF.

Originally posted by KenWittlief
I dont know why there would be a large electrolytic cap in the robot controller - Its powered by a battery, which would have very little ripple

I recommend you watch the voltage readout on your OI sometime when you take your robot from a standing still position to full speed, or worse, full reverse to full forward. The voltage will drop by quite a bit. On our 4WD drive robot last year we would frequently see short dropouts to down near 8v when the match began, since our drivers were holding the sticks full forward. Those caps are in there to supply power when the battery voltage drops too low like that. Various parts of the RC use an 8v regulator which requires some voltage higher than 8 to maintain a steady output. Look at the PDF I’m attaching - the first graph is a plot of our battery voltage over 1 match at the Great Lakes Regional last year. The graphs following are current measurements from each of our motors. If it weren’t for the above mentioned caps the RC would reset pretty much all the time.

But even still, caps are intended to smooth out noise and ripple on the power supply, not to hold it up during power interuption.

In my mind, a very short power interruption from a jolt is the same thing as noise. Noise causes the voltage to quickly drop and come back, just like bumping the breaker. What’s the difference?

seeding5-stands.pdf (29.6 KB)


seeding5-stands.pdf (29.6 KB)

14

this is not good. However, i think that your auto mode is tost if you hit anything anyways, right??

15

I am not surprised that tapping a breaker on the table would show a momentary open circuit. But that is not a fair or scientific test. If your breaker was mounted near a hard surface and a robot maneuver would cause the breaker to crash into that surface, then I think that would be a valid test. As Dave described above, there are large capacitors in the controller to at least smooth out the voltage fluctuations supplied to the RC. We suspect that Innovation First has made an improvement on that in the new RC for this year as it appears to be less susceptible than even last year’s. However there are limitations on how sensitive this can be based on your electrical design. Frequently, high current connections pass through high resistance contacts that move when subjected to shock. This movement results in noise and power interruptions to an already low power circuit. I would more suspect a bad crimp, loose connector or voltage sag caused by high current loads than a breaker.
As to the large capacitors filtering out noise from the motors… large capacitors of the type we are talking about, usually have to high an ESR to be effective at high frequencies. A more reliable filter can be made from small ceramic capacitors, RF chokes, etc. Unfortunately, FIRST has always considered the addition of capacitors to anything outside of the custom circuit board to be illegal. I have heard a report that one regional allowed this addition but have not heard anything from FIRST yet. Expect to be asked to remove it if you are inspected with it in.

16

*Originally posted by magnasmific *
**this is not good. However, i think that your auto mode is tost if you hit anything anyways, right?? **

Not necessarily. Once we smooth out a few wrinkles with our INS, we should be able to sustain a hit of several G’s accompanied by a rotation of up to 300°/sec without any ill effects to our autonomous mode.

17

OK I did some more testing in my lab.

When I did the previous tests I had the robot controller connected to the operator interface with the tether cable. In this configuration the link cycles when you tap the breaker that is powering the RC (and the OI).

I looked at the signal on the scope. The 12V drops to 5V for about 1mS.

Then I tried it again with the radio link instead of the tether cable. The Robot controller will NOT reset when the breaker is tapped or smacked.

This is good news. I opened the RC up and saw the same thing noted in someone elses post above - 1000uF cap on the 12V input.

So the RC is not what was resetting - it was the OI, getting its power from the tether cable.

So thats good for most teams - I have to correct my first post - further testing shows the breaker does NOT cause the Robot Controller to reset when impacted (in the normal configuration used on the playfield).

I believe another possible cause of our teams robot controller reseting in those two matchs is this: we have 4WD, and when we slammed the railing all four motors most likely stalled at the same time, probabally popping all four 40A breakers together. This might have been enough current surge to drop the voltage at the fuse box below the dropout voltage of the RC - causing it to reset when the 4 motor breakers opened and the voltage returned to normal.

Im not sure why our robot reset during automode the other time though - when we were pushed to the bottom of the ramp and hit by another robot. I guess it could be the same cause - the drive motors were stalled and the battery was putting out a large amount of current.

I dont think we have any loose crimps on our bot - we always made the pirate noise when we crimped terminals on the cables (RRRRRRrrrrggggggggghhhhhh!) - maybe we have a cracked fuse box?

I think Ill have all the students bring all the pillows from the hotel and we’ll tiewrap them all around the bot for bumpers at our next event (Toronto) :c)

EDIT: I had said ‘relay’ when I meant ‘breaker’ -Ken

18

I dont think we have any loose crimps on our bot - we always made the pirate noise when we crimped terminals on the cables

Never say “never” but good work on the testing. You mentioned relay several times in that last post, but I am assuming you were talking circuit breaker. I am still concerned that you were able to reset on tether but not with the radio. That still sounds like a high resistance connection or a lot of current on the OI side or something else. We have had a problem over the years with the little fuse holders on the side of the controllers. They are prone to break where the pin leaves the printed circuit board. (This is a problem when you have a lot of wiring run close by that side of the controller.) That break is very hard to find and diagnose, but the good news it is repairable. Innovation First will take care of it for you. Let us know if find anything else.

19

AL - thanks for the catch on ‘relay’ - I meant breaker - Ill will go back and edit last post.

When I did these tests in our lab I used a bench power supply (so I can see the current draw) and alligator clips on the breaker, RC power…

Its possible we have a squirrely connection still on our bot - but dont think there are any intermittant connections on the lab bench setup.

When we first started putting terminals on wires this year I made the pirate noise while I wrenched the crimper. As the students gave me a funny look (happens often) - so I explained if you dont make that noise, the wire will not stay in the terminal. Now all our sparkies make the pirate noise when using the crimper :c)

Also - we sheared the little round fuse on our RC this year when our battery decided it wanted to be 1" further forward than where we had it mounted. That RC is now on last years bot with a hand mod fuse attached.

We are using last years RC in the bot now. We asked the FIRST guru at Cleveland about this during the event, he said there would be no difference.

BTW - scuttlebutt is there will be a whole new control system next year (USB has finally caught up with FIRST?!)

looking forward to it.

-Ken

20

I know this is beside the point on your robot reset issue. However, in autonomous mode, you can and should write your current “state” to EEPROM and read it back on the next cycle. You have to make sure you clear it out when you transition out of autonomous mode.

We assumed that we would have a reset during autonomous operation and didn’t want to experience the problems you described.