PDA

View Full Version : serious problem found - robot controller resets when jarred!


KenWittlief
03-18-2003, 11:49 AM
EDIT: DID SOME MORE TESTING IN OUR LAB - THE PROBLEM WE HAD IS NOT DUE TO THE BREAKERS RESETTING THE ROBOT CONTROLLER -

SEE MY POST ON PAGE TWO THAT BEGINS WITH THE LINE:

"OK I did some more testing in my lab."




the subject line is a little misleading. The 20A breakers open when jarred causing the robot cotroller to lose power for an instant, and reset!

At the buckeye regional team 578 observed, in two matches, our robot restarted automode halfway through the 15 second period - both times after the robot was smacked.

We discovered the 20A and 30A circuit breakers open for an instant when jarred. You can try this for yourself - put a continiuty meter across a 20A breaker, then flick it with your finger, or tap it on the table - the circuit will open briefly.

These breakers are not defective, its a characteristic of the way they work - Bimetal strips that act as springs, and bend (pop) the other way when too much current heats them up.

This is what causes it to open on impact - it is esentailly a switch held closed with a small spring. When the impact force overcome the pressure of the 'spring' the breaker opens. The breakers look like a fat pizza box. We took one apart - the switch opens and closes in the same direction as if you were opening a pizza box.

What this means is, the robot controller power can be interrupted if the robot is hit, hits a wall, bounces hard on the ramp... when the power is interruped like this the controller resets, and if it happens in the 1st fifteen seconds, your auton mode code starts over from the beginning (we tested it - the controller does reset).

We emailed FIRST about this, and recommended all teams be allowed to use a 20A fuse (like the ones in the spikes) in the fusebox for the robot controller power. Its ok if your motors and other devices glitch for an instant when hit, so only the robot controller needs a fuse instead of a breaker.

if your bot is smacked at any point during the 2 minutes, and it resets, it will cost you several seconds of dead time, as the controller restarts and renegotiates the communication link - its more serious during auton mode because the automode sequence of events you programmed your bot to follow will start over from the beginning.

if FIRST wont let us use fuses instead of the breakers for the robot controller circuit, the one thing you can do is make sure your fuse panel is oriented, screwed to a vertical surface in the bot, so the 'pizza box' shape is flat in the robot. This will lessen the susceptability to impacts in the X and Y planes, and put it on the Z plane - so if your bot is dropped (gets lots of air) then the breaker will be most likely to open, and less likely to open from a front/back or side impact.

another solution is to shock mount your breaker panel.

The fuses that are in the spikes are 20Amp (and they are supplied by FIRST). The fuse is simply a strip of metal that melts when too much currrent flow (it gets hot). The fuse cannot reset. There is no logical reason why you should not be able to use a fuse for the robot controller. There is no condition in which the controller should be drawing more than 20 amps (unless you have loose metal bounceing around inside your robot.

I have not heard anything back from FIRST yet on the Q&A section in the robot rules section - they have not even posted the question yet. If you dont see a respose from FIRST on the Q&A forum then raise the issue yourself with the judges/inspectors at your next event - advocate a fuse be allowed in that circuit (for all teams) - or at least orient your fuze panel so the breakers are flat.

If you are trying to achieve king of the hill in the last seconds of your final match for the regional trophy, and you are smacked at the top of the ramp, breaker opens and your controller resets for several seconds

it will be all over.


-Ken Wittlief sparkie engineer team 578 'Blue Lightning'
NASA/Gleason/Fairport - see you in Toronto!

KenWittlief
03-18-2003, 11:59 AM
edited to fix subject line

Josh Hambright
03-18-2003, 12:16 PM
i also heard that a team at the buckeye regional found that if you jarr the modem enough you can actualy cause a basic init error and cause your robot to reset.

So it seems that maybe our problems of robots dying last year from the 120 amp breaker getting jarred could have been an even bigger problem then previously expected.

Dave Flowerday
03-18-2003, 01:30 PM
Ken,
Are you sure the resets are due to the circuit breaker opening? I'm not surprised if the breakers open momentarily, but have you been able to run your robot into a wall or something and see it happen? There is a really monsterous capacitor inside the robot controller and I'd be a little surprised if it couldn't overcome a very momentary loss of power like that.

Our team has had some really solid hits to the walls both from autonomous and driver controlled and we've never seen resets due to shock like you describe. By any chance to these resets only happen on the ramp? If so it could be the static discharge that's getting you. There's another thread that Al from our team started about this problem. We also discovered that the reset pin on the RC is exposed through the programming port and is highly susceptible to noise. On our robot controller, merely having a serial cable plugged into that port and having it unconnected on the other side will put the RC into a constant reset state.

KenWittlief
03-18-2003, 01:56 PM
The two matches where we saw our bot restart auton mode from the beginning, one the L/R switch we have on our bot was suppose to be on L, and the student set it to 'KILL' - :c)

the bot backed up like its suppose to, turned right 45 instead of left, then slammed full speed into the railing right in front of the judges table, then the bot reset, backed up again, turned 45 right again and slammed into the opponets operator station, leaving a 2 foot scratch in the plexiglass (Buckeye match 57)

we never made it to the ramp.

The other match was similar - we were pushed backwards down the ramp then hit by another bot at the bottom, and our auton sequence restarted.

We have nothing connected to our program or tether connectors during a match. In fact, i would not be surprized if this is the real cause of bots resetting at the top of the ramp (they get jarred when they land hard or hit the stack, lip, other bot...)

Other teams have reported seeing bots get hit, then go dead (light goes off) for a few seconds, then restarting.

And yes, I did test this with a robot controller on the lab bench with a 20A breaker in series with the 12V supply - you flick the breaker with your finger, or tap it on the bench (not that hard either) and the robot controller resets, and renegotiates the radio link (takes a few seconds).

This is a definate problem with using the breakers to protect the robot controller (instead of a fuse).

BTW - I have taken one of this years robot controllers apart (we sheared a fuse - now using it for a spare on old bots) I dont recall seeing any really big caps in there.

KenWittlief
03-18-2003, 02:00 PM
also - if you have your breakers oriented flat, or sideways, they are less suseptable to a front impact. We have our breakers vertical, narrow edge front to back - so they would be most suspectable to a side impact - but our bot still reset when it hit the railing head on.

Andy A.
03-18-2003, 02:18 PM
There is (or at least should be) a capacitor in the RC that smooths out the power supply just for this reason. Momentary drops in power are pretty normal, and shouldn't cause a full reset.

It sounds like either this RC doesn't have one or it just isn't working right.

Try this: Attach a capacitor (+12 volt and a couple hundred micro farads) across the power terminals of the RC. Power up and test it again. The capacitor should supply power for a second or two if the power flicks out and the RC will never know the difference.

Do you have another RC you can test?

-Andy A.

Steven Carmain
03-18-2003, 02:27 PM
I like this idea, but isn't this illegal for competion. I see this for troubleshooting purposes only.

KenWittlief
03-18-2003, 02:32 PM
The RC I tested on the bench is not the one that was in our bot when it reset, so I have bench tested one, and 'field tested' another.

I dont know why there would be a large electrolytic cap in the robot controller - Its powered by a battery, which would have very little ripple - but I will open our spare one up again and look for large caps.

The RC takes quite a bit of power - it powers the radio modem, any sensors you have on the same breaker, any servos connected to the PWM outputs - so some robot configurations might drop out faster than others.

But even still, caps are intended to smooth out noise and ripple on the power supply, not to hold it up during power interuption.

If anyone has the ability, please verify my results in your lab. I believe this is potentially a serious problem for all teams - having your auton mode restart, or loosing the link for several seconds during a match can result in your bot being damaged in a collision, or loosing the match in the last few seconds.

KenWittlief
03-18-2003, 02:33 PM
adding a cap would be against the rules - I have formally asked FIRST to allow us to use a fuse instead of the breaker in this one circuit (fits in the fuse box the same as the breaker)

no response from them yet.

seanwitte
03-18-2003, 03:02 PM
There has to be a cap in the controller to filter out noise from the motors. The secondary effect would be to smooth out momentary power interruptions. There has to be some kind of regulator in there to step the supply voltage down to 5V, I bet theres at least a diode and a filter cap on the supply-side.

I know this is a dumb question, but did you check the leads on the battery? We had a loose screw on a battery terminal that was occasionally causing it to reset. Or maybe a connector on the breaker panel is loose? We saw that come up as well. If tapping the breaker is causing the reset then maybe one of the leads is loose or not seated in the connector all the way.

KenWittlief
03-18-2003, 03:07 PM
you can take a 20 or 30 A breaker, put alligator or push on test leads on it, and connect it to an OHM meter (that beeps for continuity)

and tap it on a hard surface, the breaker will open. Maybe I will take a JPEG of the one we cut open so you can see whats inside the breaker

there is a big contact slug on a fairly thin sprung piece of metal.

The breakers would make an excellent impact sensor :c)

Dave Flowerday
03-18-2003, 04:55 PM
Originally posted by seanwitte
There has to be a cap in the controller to filter out noise from the motors.
Yup, we have a controller apart here (it's not this year's, but I doubt they've changed much) and it has two good size caps near the power connector: a 100uF and a 1000uF.
Originally posted by KenWittlief
I dont know why there would be a large electrolytic cap in the robot controller - Its powered by a battery, which would have very little ripple
I recommend you watch the voltage readout on your OI sometime when you take your robot from a standing still position to full speed, or worse, full reverse to full forward. The voltage will drop by quite a bit. On our 4WD drive robot last year we would frequently see short dropouts to down near 8v when the match began, since our drivers were holding the sticks full forward. Those caps are in there to supply power when the battery voltage drops too low like that. Various parts of the RC use an 8v regulator which requires some voltage higher than 8 to maintain a steady output. Look at the PDF I'm attaching - the first graph is a plot of our battery voltage over 1 match at the Great Lakes Regional last year. The graphs following are current measurements from each of our motors. If it weren't for the above mentioned caps the RC would reset pretty much all the time.
But even still, caps are intended to smooth out noise and ripple on the power supply, not to hold it up during power interuption.
In my mind, a very short power interruption from a jolt is the same thing as noise. Noise causes the voltage to quickly drop and come back, just like bumping the breaker. What's the difference?

Alexander McGee
03-18-2003, 06:06 PM
this is not good. However, i think that your auto mode is tost if you hit anything anyways, right??

Al Skierkiewicz
03-18-2003, 06:09 PM
I am not surprised that tapping a breaker on the table would show a momentary open circuit. But that is not a fair or scientific test. If your breaker was mounted near a hard surface and a robot maneuver would cause the breaker to crash into that surface, then I think that would be a valid test. As Dave described above, there are large capacitors in the controller to at least smooth out the voltage fluctuations supplied to the RC. We suspect that Innovation First has made an improvement on that in the new RC for this year as it appears to be less susceptible than even last year's. However there are limitations on how sensitive this can be based on your electrical design. Frequently, high current connections pass through high resistance contacts that move when subjected to shock. This movement results in noise and power interruptions to an already low power circuit. I would more suspect a bad crimp, loose connector or voltage sag caused by high current loads than a breaker.
As to the large capacitors filtering out noise from the motors... large capacitors of the type we are talking about, usually have to high an ESR to be effective at high frequencies. A more reliable filter can be made from small ceramic capacitors, RF chokes, etc. Unfortunately, FIRST has always considered the addition of capacitors to anything outside of the custom circuit board to be illegal. I have heard a report that one regional allowed this addition but have not heard anything from FIRST yet. Expect to be asked to remove it if you are inspected with it in.

ahecht
03-18-2003, 07:58 PM
Originally posted by magnasmific
this is not good. However, i think that your auto mode is tost if you hit anything anyways, right??

Not necessarily. Once we smooth out a few wrinkles with our INS, we should be able to sustain a hit of several G's accompanied by a rotation of up to 300/sec without any ill effects to our autonomous mode.

KenWittlief
03-18-2003, 08:00 PM
OK I did some more testing in my lab.

When I did the previous tests I had the robot controller connected to the operator interface with the tether cable. In this configuration the link cycles when you tap the breaker that is powering the RC (and the OI).

I looked at the signal on the scope. The 12V drops to 5V for about 1mS.

Then I tried it again with the radio link instead of the tether cable. The Robot controller will NOT reset when the breaker is tapped or smacked.

This is good news. I opened the RC up and saw the same thing noted in someone elses post above - 1000uF cap on the 12V input.

So the RC is not what was resetting - it was the OI, getting its power from the tether cable.

So thats good for most teams - I have to correct my first post - further testing shows the breaker does NOT cause the Robot Controller to reset when impacted (in the normal configuration used on the playfield).

I believe another possible cause of our teams robot controller reseting in those two matchs is this: we have 4WD, and when we slammed the railing all four motors most likely stalled at the same time, probabally popping all four 40A breakers together. This might have been enough current surge to drop the voltage at the fuse box below the dropout voltage of the RC - causing it to reset when the 4 motor breakers opened and the voltage returned to normal.

Im not sure why our robot reset during automode the other time though - when we were pushed to the bottom of the ramp and hit by another robot. I guess it could be the same cause - the drive motors were stalled and the battery was putting out a large amount of current.

I dont think we have any loose crimps on our bot - we always made the pirate noise when we crimped terminals on the cables (RRRRRRrrrrggggggggghhhhhh!) - maybe we have a cracked fuse box?

I think Ill have all the students bring all the pillows from the hotel and we'll tiewrap them all around the bot for bumpers at our next event (Toronto) :c)

EDIT: I had said 'relay' when I meant 'breaker' -Ken

Al Skierkiewicz
03-19-2003, 08:10 AM
I dont think we have any loose crimps on our bot - we always made the pirate noise when we crimped terminals on the cables
Never say "never" but good work on the testing. You mentioned relay several times in that last post, but I am assuming you were talking circuit breaker. I am still concerned that you were able to reset on tether but not with the radio. That still sounds like a high resistance connection or a lot of current on the OI side or something else. We have had a problem over the years with the little fuse holders on the side of the controllers. They are prone to break where the pin leaves the printed circuit board. (This is a problem when you have a lot of wiring run close by that side of the controller.) That break is very hard to find and diagnose, but the good news it is repairable. Innovation First will take care of it for you. Let us know if find anything else.

KenWittlief
03-19-2003, 09:09 AM
AL - thanks for the catch on 'relay' - I meant breaker - Ill will go back and edit last post.

When I did these tests in our lab I used a bench power supply (so I can see the current draw) and alligator clips on the breaker, RC power...

Its possible we have a squirrely connection still on our bot - but dont think there are any intermittant connections on the lab bench setup.

When we first started putting terminals on wires this year I made the pirate noise while I wrenched the crimper. As the students gave me a funny look (happens often) - so I explained if you dont make that noise, the wire will not stay in the terminal. Now all our sparkies make the pirate noise when using the crimper :c)

Also - we sheared the little round fuse on our RC this year when our battery decided it wanted to be 1" further forward than where we had it mounted. That RC is now on last years bot with a hand mod fuse attached.

We are using last years RC in the bot now. We asked the FIRST guru at Cleveland about this during the event, he said there would be no difference.

BTW - scuttlebutt is there will be a whole new control system next year (USB has finally caught up with FIRST?!)

looking forward to it.

-Ken

Andrew
03-19-2003, 09:23 AM
I know this is beside the point on your robot reset issue. However, in autonomous mode, you can and should write your current "state" to EEPROM and read it back on the next cycle. You have to make sure you clear it out when you transition out of autonomous mode.

We assumed that we would have a reset during autonomous operation and didn't want to experience the problems you described.

KenWittlief
03-19-2003, 09:56 AM
EEPROMS have a limited number of write cycles, something like a million?

I dont think you want to be writing to the EEPROM on every SW loop. When something is running at hundreds or thousands of loops per second, it doesnt take long to get to a million.

in fact, how could you program this? if your bot did not finish auton mode the last time you ran it, or got hung up for some reason - how would you reinitialize it the next time you WANT to power the bot up?

wouldnt it try to go back and start from where it left off last time?

how does the RC know if the reset was a glitch or a deliberate power on / power off cycle?

Joe Johnson
03-19-2003, 11:54 AM
A million cycles is a probably on the low end of actual times you can program an EEPROM, but even so, we have found ways around this problem in the past.

Even writing to the same address every time you can still run for over 400 minutes without hitting a million writes, large but not large enough, perhaps.

We used a few tricks.
#1, read the contents first, only write to them if there is a changed state.

#2 move to another EEPROM address as you wear out the "real" EEPROM address. Basically keep track of how many addresses you've worn out -- increment the index after you find a write didn't take. To see if you have "worn out" an EEPROM address, check the contents after the write command.

#3 Devote a programming slot to essentially nothing but this purpose. I have never run out of code space entirely. So, I always have a slot to burn (pardon the pun). Basically use a RUN 7 command to run to your EEPROM burning code and then at the bottom of the EEPROM burning code do a RUN X to get back to your regular code. If you have to burn up EEPROM addresses, better that they all be from the same programming slot -- once you've worn out an EEPROM address, it cannot be used for program space either -- this could be a problem if you burned up all the EEPROM space in slot 0 for example -- this would leave you with no way to even download code to jump to another slot -- an extreme example, perhaps, but not impossible.

Something like this should work fine:

ADDR_INDEX data 0
ADDR_STATUS data 0 (256)
'reserves 256 locations for your burning needs
'omit the zeroes above if you do not wish to overwrite the prior
'stored values when downloading new code
temp1 var byte
temp2 var byte
status var byte
do
'start of loop code
read ADDR_STATUS, temp1
read ADDR_INDEX, temp2
if temp1 <> status then write ADDR_STATUS + temp2, status
read ADDR_STATUS, temp1
if temp1 <> status then write ADDR_INDEX, temp2+1
if temp1 <> status and temp2 = 255 then gosub _EEPROM_Error
'rest of loop code
loop

While there are a lot of reasons not to write every time through the loop (it takes too much time being the biggest one in my opinion) a scheme like the above can effectively make your EEPROMs last forever (in FIRST time, at least).

Joe J.

P.S. the above code is right from my brain, I did not get our old code out or even check syntax on it. I believe it is sound, but make no promises. JJ

KenWittlief
03-19-2003, 12:16 PM
Ive been thinking about this EEPROM idea over lunch - its a cool idea

but taking two steps back, if your robot was reset during auton mode, then something drastic has happened to it:

- it got lost and smacked a wall

- it got nailed by a superiour being (a faster bot)

- you have a loose wire somewhere...

so given that you dont know what is going to cause the robot to reset during auton mode, you cant possible know what to do when your RC resets, except maybe shutdown and do nothing?

Error detection is one of those weird problems in engineering. Its usually very easy to detect than some sort of error has happened (like an unexpected reset) - and its almost always impossible to decide what to do next - except log an error message and shut down.

If you could anticipate the error condition, then you can program / design the system to expect it, and handle it the way you want - thats not an error condition anymore, its another variable in your more robust system's normal operation :c)

Greg Ross
03-19-2003, 12:30 PM
FYI, the BASIC Stamp Manual gives 100,000 as the number of writes per location for a BS2SX. (BS2e and BS2p are also given as 100,000. BS1 and BS2 are 10,000,000.)