1726 Electric Problem
 

As anyone who watched the Arizona webcast (or was at the regional) knows team 1726 put on a great show, but would intermittently stop working. We believe this is an electrical problem (something we have never encountered before). When the robot stops working the victor lights turn off and the victor fans stop, but the pneumatics system and compressor continue to run and operate. Jostling the robot will result in the victors receiving power for an intermittent amount of time before the robot dies again. This originally led me to believe that the problem was a loose connection in the maxi fuse panel wiring (from power distribution to fuse panel to victors back to power distribution), but after re tightening all the connections the problem persists. There is no obvious evidence of a short such as melted wires and when examined the connecting wires appear to be in excellent condition.

Even more confusing when operating with the robot on blocks we are unable to recreate the problem.

So now I have several questions

Could this be a problem with our forty amp breakers, on Thursday we used nitrile tread and due to the high torque tripped our breakers continuously for several matches before recognizing and fixing the problem. Is it possible that some of the breakers are now damaged and trip very easily? It seems strange that all four would fail at once (as we observed however tripping one breaker could result in others tripping as we use two motors/gearbox) and that bumping the robot would reset them, does anyone think this is the problem?

If the above could not be the problem how would you recommend we trouble shoot. A multimeter simply isn't enough because it is highly possible that we have conductivity, but not enough current carrying capacity. Is there a proper way to trouble shoot this situation?

Re: 1726 Electric Problem
 

I'll throw my two cents in here, I'm sure other more knowledgeable people will pop in with some more suggestions later.

Regarding the breakers, it is possible that they are now easier to trip after being tripped repeatedly. The thing that makes me wonder if this is really the issue is the jostling the robot thing. Either that is a coincidence or your issue lies elsewhere.

It makes sense that you cannot recreate the issue on blocks as your motors will raw far less current when on blocks than it will on the carpet when pushing or turning.

If it is just the victors that are losing power, one thing I would make sure to check is the Rockwell Distribution block. If the wire to the maxi panel comes out of a different block than the wire coming in from the main breaker, check and recheck the jumper block. That yellow block can cause all sorts of issues if not seated well and tightened down properly (my team had this issue this year).

Hopefully some others can add to these suggestions.

Re: 1726 Electric Problem
 

Wow I can't believe we didn't check that... I'll add that as one of the first things we need to check once the robot comes out of the crate in LA. If the yellow jumper block is the problem how would be best fix that and ensure that it does not become a problem again?

Re: 1726 Electric Problem
 

On our robot, the problem was simply not tightening down all the screws on the jumper block, tightening them down solved it. If the jumper block looks "fried" in any manner (I would pull it out and test it for resistance if the screws are tight) then I would replace it (I believe there are replacements available at Pit Admin if you or other teams don't have one).

Re: Electric Problems
 

I don't know if any of this will help, but:

1717 had what I think is a similar problem - they would be driving and then all four of their drive motors would blink out simultaneously. I believe they eventually fixed the problem, but I don't remember the solution. They were also noticing a difference between on tether with the OI running off AC power, on tether with the OI running of the robot's battery power, and on radio with the OI running off the competition port power.

980 was also having electrical problems - we'd be driving along and then the RC would just reboot (sometimes several times in a second) or randomly turn off relays. (This reeked enough havoc to our auton mode, that the ref gave us a warning and we decided not to run autonomous until we'd fixed the problem.) We discovered that the red wire to the Anderson connector on the robot was very warm, so we think we were getting an intermittent connection there. We also discovered that the ground wire to the backup battery was loose within the plastic housing (although the "bad backup battery" light never came on). We replaced the Anderson connector on the robot, fixed the loose backup battery ground wire, re-downloaded the master code to the RC, added some printf() statements to the user code, and the problem seemed to go away.

Re: 1726 Electric Problem
 

We had some odd electrical problems in Oregon this year. Fortunately, most of them occurred on Thursday.

The first one was due to the RC main power spade connector ground being loose. This was due to an incident during build where the spade connector was pulled out of the RC. We thought we had it soldered back in well, but apparently we were wrong. We re-soldered it (just from the top this time) and it has held so far. We re-check it regularly now.

The second was due to a poor connection between the battery and the 120A circuit breaker. This resulted in our main breaker getting hot and tripping. The key that it was a poor connection (that we eventually clued in to) was that the stud on one side of the 120A breaker was hotter than the stud on the other side. The challenge in diagnosing this may have been compounded by loose connections in the Rockwell block, as mentioned in a post above, as we found a few of them needed an extra turn of the screwdriver or two.

We are fortunate to have a clamp on ammeter, as that allows us to easily monitor current flow in the motors/circuits. (If anyone is thinking of buying one... make sure you get one that does DC current as well as AC.) Ours has two settings 4A and 80A, so at maximum load we can only monitor the current to one CIM. Using the ammeter we were able to recreate medium to high current draw situations in the pit by having a couple of team members kneel in front of the robot and draw the bumper up tight to their legs. The driver would then gradually add "forward" to the robot until the motors got up to about 40A (in stall) and hold it there for about ten or 20 seconds. Another option would be to drive the robot up against a wall and push for a while. Again, it is nice if you can monitor current draw for at least one motor so you have an idea of how hard you are pushing the motors/electrical system.

We discovered (technically these probably aren't discoveries to the rest of the world, but they were enlightening to us) that with two CIMs hooked up to a toughbox that when the wheels run freely, one CIM would be drawing 5A and another would draw pretty close to zero. Differences in the motor? The Victor? We haven't figured that out yet, but the discrepancy seems disappear as the load is increased.

We also discovered that the eventual solution to our wiring problem, although obvious in retrospect, was the very last thing we checked... even going so far as to pull a gearbox apart to check if anything was wrong in there. (Okay, we just pulled a motor and peeked in... but we were most definitely looking in the wrong place.)

We also had a weird situation where the robot died during a match... but came back to life after hitting "robot reset". This is one of the advantages of experimenting with buggy code... the drive team gets to know the location of the "reset" button very well! If you conk out in the middle of a match, give it a push... you've got nothing to lose and it might just get things going again.

Good luck getting the wiring sorted out, intermittent wiring problems are a real pain and it is a shame they held you back. Mind you, it still looks like you shared a pretty good playoff run with a couple of rookie teams. Well done, and good luck,

Jason

Re: 1726 Electric Problem
 

I would definitely check the wire connections from the Distribution Block to the Circuit Panel, the wires tend to come loose after a lot of matches. A way to diagnose this would be to run the robot for a while (or at least try) and then feel the wires... or you could just try to tighten the wires up.

As for the breakers becoming weak; YES! it can happen. In fact, this happened to us last year during Nationals, and the simple fix was to replace all of them, but if all the Victors turn off with their fans, this points to a wiring problem. I believe all you need to do is tighten the screws that hold the wires in place. I would also recommend that you move all the heavier loads to the same block that the main power is fed, then all the smaller stuff run off of the jumped block, because I see a lot of power being fed through such a small piece of metal (80 AMPs with 2 CIMs at stall, 160 AMPs with 4 CIMs at stall; also those numbers are likely to be higher as the breakers aren't instantaneous)

Re: 1726 Electric Problem
 

Also, check your crimps on your battery connector. This is probably the most mechanically stressed connection in most people's machines as it is the only one which gets moved every time a battery is changed. Also, it is the highest electrical load connection in your robot. I suggest checking both your ground and positive terminals going into your distribuition and breaker blocks respectively.

Our positive terminal came off of our robot at the end of our first semifinal match in Chicago... thankfully we noticed it when it fell out of the robot while changing the battery! :yikes:

Then again, we also had the eh.. small problem... of one of our ultrasonic sensors catching fire... but that's another story entirely. :o

Hope you find your problem! If all else fails, tug on wires and see what pulls out/moves easily.

-q

Re: 1726 Electric Problem
 

Connectors are generally the weak link in electrical systems.

In the space business, people are trained and certified to mate and demate connectors. Before connectors are mated, the contact mating force is verified for each pin/socket, the connectors are visually inspected for bent pins/sockets, both sides of the connectors are cleaned to remove any debris (a little shard of metal can be disastrous) and the proper alignment of plug and socket is verified.

I'd check all connections (including the power distribution block) carefully before blaming the problem on other components in the electrical system.

Re: 1726 Electric Problem
 

Funny story: this year, our hurdle bot weighed only 87 lbs when we got to the competition. We decided to add some weight to get it closer to 120lbs using a couple of dumbbells. We figured that the easiest way to "secure" them would be to use zipties. We put two dumbbells in the front and one in the back near to our electronics board. During our final qualification match, we slammed into the alliance wall pretty fast and the dumbbell came loose and unplugged all of our motor connections! Ooops...

Re: 1726 Electric Problem
 

As a quick aside:

Have you tried running your robot without the extra weight? Since there isn't as much defense this year, you may gain critical extra performance from your robot's drive system.

-q

Re: 1726 Electric Problem
 

Troubleshooting: Find source of problem.

This means checking all the wires going into the Victor. Trace them back to wherever they originate. Test the breakers, the connections, everything. All wires includes the PWM cables, by the way, even though I'm fairly certain they aren't the problem.

If you don't find the problem before the breaker panel, figure out where everything on that panel goes. Do any of those have problems? If so, it's probably in the panel or "upstream" somewhere. If not, check the breakers.

Re: 1726 Electric Problem
 

Troubleshooting:

When you have a circuit that loses power, there are several methods of identifying the problem. One method, while not simple, is very fast and effective:

Determine how many different things there are in the circuit. For example:
Battery, anderson connector, main breaker, wiring block, yellow thingus, circuit breaker panel, victor, motot connectr(?) and motor. That's 9.

Get several 12 volt LEDs (use a 680 Ohm dropping resistor). Put all of the LEDs in a row and make them visible to an operator or observer.

Connect an LED using clip leads at each point in the circuit where there is a change in the path - at the input to the Anderson connector, the input of the main breaker, the input of the power block, etc.

Duplicate the condition - perhaps on the practice field, I can't say these LEDs are competition-legal.

Watch which LEDs are not lit when the condition occurs, the will lead you right to the problem. Works every time, this is a common technique in automotive repair when time is of the essence. Videotape the LEDs if necessary, to catch very fast transients.

Don

Re: 1726 Electric Problem
 

Wow, this thread went pretty far in just a few hours.
If the drive motors are all cutting out at once, this points to something in the 40 amp wiring, if you are using that. Remember that the black wiring is just as important here as the red wiring. That being said, the problem could be in either the 40 amp block or the Rockwell block. In the order of failures these problems are:
1. Improperly stripped and terminated wiring in the Rockwell Block. Strip 5/8" and do not tin the wires before insertion.
2: Improper termination of the input side of the 40 Amp Maxi Block. Better termination can be made by stripping the input wire about 1-1/4" and folding over the stripped wire before insertion in the Maxi-Block. If you use a #6 wire straight into the block, the strands tend to ride the threads of the clamp screw up the side of the hole and prevent them from becoming part of the connection. Folding over the wire, fills the hole and allows the best clamp when the screw is tightened.
3. Loose jumper strip mounting hardware.

If power is being lost to other robot systems or if your Anderson connector is getting hot, the likely cause is scratched surfaces on the Anderson connector. Often teams use the alligator clips on the charger to connect to the open Anderson connector on the batteries. This raises scratches that reduce the amount of current carrying metal to a fraction of the design surface. The resulting high resistance connection causes heat and a voltage drop across the connector. Spend the money to buy an extra Anderson connector kit. Remove the alligator clips and replace them with the Anderson connectors. You will have to strip the wires from the charger at least two inches and fold them twice to be able to solder into the connector if you con't have the proper crimper.
Breakers do go, bad but rarely. Hot breakers trip more easily than those at room temperature. These are temperature contolled devices after all. (that includes the main breaker.) Hot wiring conducts it's heat to the breaker it is connected to.
If the problem exists on the floor and not on blocks you should suspect loose wiring that is pulled or pushed by the robot frame when on the blocks. Or on rare occasions, wiring is contacting the frame and is shorting out that branch circuit. Suspect any wiring that passes through frame tubing. An errant drill may have put a hole in the tube and damaged the wiring running through it.
As to unbalanced current in a two motor transmission, suspect that the speed contollers are not calibrated. Although rare, a motor does go open and therefore no current flows. If the current draw does approach the same current at a higher speed or load, check the calibration. It is simple to move your joysticks while watching the LEDs on the Victors. Watch to see that both light the LED fully at the same point on the joystick.

Re: 1726 Electric Problem
 

Thanks for the suggestions, Al. I didn't really get a chance to get into trying to figure out what was going on during competition Saturday afternoon, but we do have some things to look over when we get to LA. We have Andersen connectors on our chargers, we have not used the clips at all.

Re: 1726 Electric Problem
 

I dont know if somebodys already posted this scenario, b/c i didnt bother to read all posts. your issue could in fact be that you are loading your drive motors too high- we at team 1647 from new jersey have had the same problem and we had the same assumptions (one of the questions i would like to ask is what kind of drive system are you using?) we used the omni drive setup where we have 2 sets of wheels on perpindicular axis to each other- one CIM to a transmission (4 trannys) and the gearing and the diameter of the wheels were factors in overloading on the amperage needed. we solved the problem by creating a whole new drive system that we are goin to swap out at the philadelphia regional. what i would suggest instead of such a darastic change that we did would be to change the gearing inside the tranny or change out to a different brand tranny if thats possible- or maybe find a more powerful matter of CIM. (that can still handle the 40 amp breakers- which may not be possible) other than that... i cant tell you anything else. basically we solved our problem by keeping the same number of motors but dropping trannys by 2 and shifting drive to one axis instead of 2 and shifting to a more tacky wheel for defense... aight thats all i ahve to say... oops i wrote alot sry!

Re: 1726 Electric Problem
 

Jim,
There is one other possibility but you are not going to like it. On rare occassions, the screws that hold the 40 amp Maxi block together come loose. They can only be accessed from the bottom side of the block. If they loosen up, the contact between the input and output terminals of the breakers do not make good contact. If you remove the cover and then pull all of the 40 amp breakers, try moving each of the contacts. If they move, the screws underneath have come loose. I am betting you are going to find a loose wire on the input to the block or on the Rockwell block. Let us know what you find.

Re: 1726 Electric Problem
 

Thank you all for the advice, I don't believe that the problem is a loose connection as we checked and double checked all of the connections frantically on Saturday and were unable to find anything loose. We also redid the majority of connections examining the wiring for signs of damage.
We did not however fold over the wiring in the maxi fuse block, of check to see if the screws were loose.

Re: 1726 Electric Problem
 

We had this same problem. The robot would flicker on and off as we drove. The main wire going into the distribution block was the culprit. It had come loose and would lose contact when bumped . A very quick fix by screwing it back in.

It was very sad when I saw you guys just sitting there. I honestly think you guys would have won Arizona. 1726 is one of my favorite teams this year. Viva las launchers:D

Re: 1726 Electric Problem
 

We ran our hurdling configuration all day on Thursday and Friday without the extra weight. (In hopes that our other configuration could be added on Saturday if we received permission to do so from the GDC.) We found that our robot was a little tippy in a situation we hadn't anticipated -- when carrying a trackball and then bumping up sideways against a trackball which was wedged against the side wall, the two trackballs would smoosh together, storing up energy which would then rebound and flip our robot over onto its side. Amazingly, this happened not just once, but twice in qualifying rounds while driving around the field with a trackball while the arm (and the trackball) where in the "stowed" position!)

Re: 1726 Electric Problem
 

I'm not going to be a big help here, but during semifinals last year, we had a similiar problem. The left motor of our drive system cut out during the first 15 secs. of the tele-operated period. We frantically tried to figure out what was wrong. We tethered and everything worked fine. During our second match the exact same thing happened with the same duration. The IFI guys ensured us it was not a field fault. We hadn't made any program changes for both regionals we went to and still have never been able to duplicate the fault outside the field.

Anyway, I would also take a look at the backup battery charge and connection, we have had issues with large current draws dropping the RC voltage just for a fraction of a second and resetting itself if the backup battery is low or not connected well. Sometimes the terminals in the molex connector can get loose. Just a thought

Re: 1726 Electric Problem
 

We found the problem pretty quickly this morning,the screw that holds the 6 gage feed wire into the Maxi fuse block was loose, so we put in a new wire (stripped a bit longer) and tightened all the connections on it and the Rockwell block, and it worked fine all day.

Thanks for all the help!

Re: 1726 Electric Problem
 

Jim,
I hate to say it but... This is a great photo of the damage done by loose wires and high current. Note the melted copper on one or two of the strands. This is why I recommend folding over the #6 before insertion (see my earlier post). It then fills the hole in the block and makes the screw a more effective clamp.
Jim, may I use this photo in my electrical presentation for the Championship?

Re: 1726 Electric Problem
 

Yes, you're certainly welcome to use the picture! We'll discuss putting in a longer wire with the end folded over, I just forgot to reread this thread last night...oops!

I would take a bigger close up picture of the wire, but it's pretty well mangled after being in my pocket all day and showing it off to people here :)

Also, the screw was discolored from the heat, we swapped it with an undamaged one.

Re: 1726 Electric Problem
 

I must agree with Al on this... it's a great photo of what can happen with a bad connection. Although I've never had an issue with it, I will tell our electrical team to fold over the wire upon pushing it into the maxi-fuse block. I've done this before with other connections, but I've always considered it trashy to do this, so I've never done this with our 6AWG wire. I've done it before with the small muffin fan wires to the little red ring connectors when they were just too big, then they kept on pulling out, so I soldered the wires one, and haven't had too much trouble since then. I'm not at all surprised that the copper melted with so much of a load at such a weak point. I'm sorry this happened to you, but take it as a learning experience that I'm sure you'll never forget, and will make sure that next year this will not happen.

For preventative maintenance, at the beginning of any regional/championship, and before the finals, I reinspect my electrical and make sure everything is all tight, nothing is chaffed, and that nothing in my control will go wrong.

Some preventative measures I have taken with our robot:
All PWM cables are held in with a small dab of hot glue. Hot glue works well for this, and easily comes off if I want it to. All the PWM wires that go to the controller are also hot glued in place.

Solder fan wires, and solder in the 40AMP ring connectors to the wire. (Unfortunately I didn't do this this year, but I still may, but it will have to be with lead free solder. I have never soldered with lead free solder before, so hopefully it won't be a bad experience)

Label EVERYTHING!!! It makes fixing something so much easier. Every victor, spike, circuit breaker, pwm cable, etc... is labeled.




Al, this is a question for you... I placed a label on top of our circuit breakers since the BUSS fuse block is hard to label. I did put a label on the long ways that says that all circuit breakers are 20AMP's... will an inspector ask me to verify that all breakers are 20AMP's? Should I put some temporary circuit breakers in for inspection? Is this fine? I don't remember anything in the rules about being able to read all the circuit breakers. <R50> states that "Circuit breakers must be accessible for inspection at each FIRST Robotics Competition
event." The breakers are accessible by opening our hood, and an inspector can easily see that these are the snap action breakers by the shape and the colors of the legs of the breakers. Will they take my word that they are all 20AMP, not that they all are required to be 20AMP, just the RC, and 3 spikes. Since you're an inspector, I wanted your input on this.

Re: 1726 Electric Problem
 

Think about the heat required to melt copper - those 'pearls' on the end of some strands were formed by heat.

That heat came not form contact resistance, but from arcing. An arc is electricity forming a plasma, reaching temperatures of several thousand degrees (often around 10k) in a split second. This is how many home fires start, because the plasma is not a dead short (which would trip a breaker) but a 'high resistance short' usually permitting around 15-18 Amps at 12 VDC. For homes, you can now buy an "arc-fault' circuit breaker, which detects the characteristic high-frequencies of an arc and cuts the power.

Glad the robot didn't catch fire.

Don

Re: 1726 Electric Problem
 

I'm not Al, and I don't play him on TV, but I'm going to chime in here and say that labeling the breakers doesn't seem very helpful to me. It's the position on the breaker panel that determines what device gets the power, and any given breaker is easily moved from one slot to another regardless of how it's labeled. If you pull two breakers, the labels don't help you put them back in the same place they came from, and you can thus lose track of which breaker slot is associated with what circuit.

I updated our electrical tracking spreadsheet this year to produce a worksheet full of wire and component labels that we printed on sticker paper and cut out. One part of the sheet is sized to fit in the recessed line between the rows of breakers, and it labels each of the twelve breaker slots. One slot's label is 0.6" wide and 0.25" high, enough for a short line of easily-read text or two lines of tiny but still readable letters.

Re: 1726 Electric Problem
 

I see your point... I think I will make a spreadsheet with all the breakers and label them by using the spreadsheet. I was anticipating that we would not need to take more than one breaker out at a time, but it could happen, and I do see where an inspector may ask about the breakers. Hopefully the electrical tape used to keep the labels on didn't break down and become gooey.

Re: 1726 Electric Problem
 

Ryan,
A good inspector needs to follow a certain size breaker to a certain size wire. 20 amps can be 18 AWG and so on for the larger breakers. So the answer is yes, they need to see the rating on the top. We mark everything with a 3M EIA (10 color smae as resistor color codes) marking tape. It comes in a convenient size dispenser with tape that is only 3/16 wide. It is a little pricey at $30 but worth every penny when looking for a problem.
The position on the breaker panel is marked with a color relating to a particular load. The breaker has the same color as does the wire(s) leading to the Victor or Spike and so also the wire leading from the Victor or Spike to the load. We also mark the Victor fan or the top of the Spike and the device it is controlling. Finally, we mark the RC output and the PWM cable with the same color. That way if the RED motor is not running we look to see if the RED fan on a Victor is not turning, then we look to see if the red breaker is in place at the RED position on the breaker panel. If The RED Victor fan is turning we then check to see if the RED PWM is inserted at both ends. All of this can be accomplished without referrring to documentation. Even the chassis is marked for our drive motors and steering motors. Likewise the pneumatic cylinders and the valves. We can even figure out problems on past robots without docs. Thanks to a former Motorola engineer who I shared electrical with many years ago, Steve Tine, we have been using his idea ever since.

BTW, we also produce a spreadsheet (in color) that lists all of this info for all devices including sensors, motors and valves so that software, hardware and electrical sub teams are all "on the same page".

Re: 1726 Electric Problem
 

Going a little off-topic, here....

At my insistence this year, 1533 copied Al's method of marking devices, wire ends, and PWM cable ends with color-coded tape. I cheaped-out and found a set of small rolls with five colors (plus we had black) at the local Home Depot, but that gave enough color combinations to give everything unique markings. This really saved us when we had to strip the robot down to the chassis and wheels on Thursday at Peachtree to rebuild the transmissions (which we'll have to do again at Champs, darn it). Everything went back together smoothly, quickly and properly.

Thanks, Al. Thank you, thank you, thank you.

One thing where we may have differed was combine colors for similar functions. We have two CIMs in AM Supershifters on each side. The right side motors are blue and white, the left side are red and yellow (single colors), and their Victors are each marked with a single color. The PWM cables and shaft encoders are then colored blue+white and red+yellow for the two sides, and the shifter servo cables are similar. When the shifter cables are connected together in a Y-cable, the single end becomes blue+white+yellow+red.

- Steve (thanks again, Al)

Re: 1726 Electric Problem
 

Anytime, glad it helped.