[FTC]: Tetrix motor controller weirdness with 4 motors

[aklego]

Greetings All,

My team and I just spent the day chasing a hardware bug that we have not seen before, and we are hopeful that someone can shed some light. We have a skid steer setup with two motors on each side, and all four motors are controlled by a single controller. Motors on each side are mechanically coupled -- a fairly standard set-up. The robot is lightweight, the motors are lightly loaded, and encoders are not used.

After a brief but variable bit of driving, the drive motor controller and every controller further down the daisy-chain would stop responding. The NXT and upstream controllers were unaffected. A power cycle of the Tetrix battery was required to bring everything back to life. We swapped controllers, cables, and changed positions in the dasiy-chain to no avail. We also noted that the fault would occur with a motor direction change, and we were able to get it to fault while the robot was elevated and just spinning its wheels. It was not a loose wire or the like, and it was not software.

Eventually, we unplugged a motor on each side and the problem vanished. We plugged them back in and removed plugs from the other two motors, and again the problem was gone. When all four motors were plugged back in, the problem would return. Our (hopefully temporary) solution was to add an additional controller and split the drive motors between the two controllers.

One thought (and I have no idea as to its validity): The electrical weak link in the old Tetrix motors was an internal inductor that provided some filtering. The new motors feature improved burnout protection, and they are what we are using. Is it possible that the inductor has been removed/changed and that there is a spike in noise when the motors change direction? If so, it seems the controllers are no longer able to handle the noise of two motors on the same channel. Seems pretty far-fetched, but we are at a loss for a definitive explanation.

Any thoughts, or has anyone encountered similar problems?

[DonRotolo]

Absolute speculation: Monitor the battery when this happens, low voltage can do weird things. If the battery is getting on in years, its capacity may be diminshed.

[Ian Curtis]

How did you verify it wasn't a software problem if a hardware swap didn't fix it?

[aklego]

How did you verify it wasn't a software problem if a hardware swap didn't fix it?
The software has no way of knowing or sensing the number of motors attached to a channel. If it was a software problem, the faults would still occur regardless of load.

Once it was faulted, the problem would remain regardless of program running (or not running) on the nxt.

[Al Skierkiewicz]

ak,
This sounds so much like an over current issue with four motors running. When you add up the motors, the reversing as a repeatable cause and reboot to clear the problem, it seems like everything is pointing to the motor loads dragging the battery voltage down. The inductor used in the previous motors was under rated for current such that repeated stall current would eventually burn open the inductor. This does not sound like RF interference as that would produce more problems and generally be not repeatable. RF is at it's highest while the motor is running wide open or under heavy load.

[aklego]

ak,
This sounds so much like an over current issue with four motors running. When you add up the motors, the reversing as a repeatable cause and reboot to clear the problem, it seems like everything is pointing to the motor loads dragging the battery voltage down. The inductor used in the previous motors was under rated for current such that repeated stall current would eventually burn open the inductor. This does not sound like RF interference as that would produce more problems and generally be not repeatable. RF is at it's highest while the motor is running wide open or under heavy load.

I would agree except that it would do it when the motors were unloaded, i.e the robot was up on blocks just spinning it's wheels. What is completely puzzling to me is that this is a pretty standard FTC configuration.

But I would also agree that my explanation, (RF issues) is unlikely.

Not knowing the root cause puts our "fix" in the realm of voodoo and that is particularly unsatisfying.

[Andrew Rudolph]

I would pull the motors out of their mounts and make sure they are all spinning the correct way. As Al said, it sounds like an over current issue. I have seen it where one of the motors on a side is shorting, putting into a break mode, so it causes the other motors to work harder to overcome the motor that's not spinning.

Otherwise I would rewire everything and make sure there are no shorts.

I would advise not daisy chaining the electronics on the robot, I have all my teams use Anderson Power Poles (http://www.andymark.com/product-p/am-2198.htm) or Wago Levernuts (http://www.andymark.com/product-p/am-2577.htm), so if there is a wiring issue, everything down line doesn't go down. Especially since the voltage sensitive Samantha module is always the end of the line in the daisy chain wiring scheme.

[Al Skierkiewicz]

ak,
I went back and read over your post again. Am I correct that you have one controller feeding all four motors? Is that one controller output or two motors on one output and two on the other output? The motor controller is intended to power only one Tetrix motor per output. The controller has over current sense and will shut down if the current is too high or the temperature climbs above a certain point. If the controller is shutting down for a fault, it may fail to pass I2C signals on to the following devices. The specification sheet lists the current output for each motor circuit at 4 amps which is below the stall current for the motors.

[orangemoore]

I do have a question,

What is the purpose of running four motors off one controller?
There is no limitations to how many DC motor controllers you may have, the limitation is being only allowed to have 8 motors.

[CougarRobot]

http://ftcforum.usfirst.org/showthread.php?1921-Robot-Electronics-and-Power-Answer-Thread&p=7575&viewfull=1#post7575

2 motors per port has always worked just fine and is allowed. However, we have not yet done it with the new motors.

Is anyone else using 2 motors per port with the new motors?

[Scott_4140]

4 motors on one controller worked fine for us with the new motors at our first Qualifier.

Don't confuse FRC and FTC. There is nothing in the FTC rules preventing 4 motors on one controller.

Also, from the LEGO Education site:

http://www.legoeducation.us/eng/product/hitechnic_dc_motor_controller_for_tetrix/1648

The HiTechnic DC Motor Controller for TETRIX® connects to an NXT Intelligent Brick sensor port and will enable you to control powerful DC gear motors for use with TETRIX robots. The controller has two H-bridge outputs to control the speed and direction of two DC gear motors and is designed to connect to the TETRIX hole pattern. A four-motor robot can also be built by paralleling two motors to one channel. A total of four DC Motor Controllers for TETRIX and/or Servo Controllers for TETRIX can be daisy-chained to provide additional motor and/or servo outputs on each port.

--Scott

[aklego]

4 motors on one controller worked fine for us with the new motors at our first Qualifier.

...


This is good to know although I'll admit that I was hoping other teams w/a similar setup could reproduce the problem.

We have our qualifier this weekend that we'll run with our current setup and then return to troubleshooting after the event.

Lots of suggestions of an over-current fault but that seems inconsistent with the applied load. When we return to troubleshooting I'll instrument the controller and see what our current actually is.

[Al Skierkiewicz]

ak,
I received some info that the I2C passes through the controller in the event of a fault. So if you are having issues down stream from a particular controller, that device could have an issue or you could have a defective cable. However, there is still the off chance that one of the drive motors has excessive friction causing some high motor currents. You need to isolate the motors and see if you can find that issue as well. You mentioned in your first post that the motors are mechanically coupled, so it is possible that one of the motor issues is being coupled into the other. If you remove one electrically the mechanical problem still exists. Good luck this weekend.

[PhilBot]

I would agree except that it would do it when the motors were unloaded, i.e the robot was up on blocks just spinning it's wheels. What is completely puzzling to me is that this is a pretty standard FTC configuration..

My experience is that rapidly reversing motors causes worse than Stall-like power consumption. Instead of the Back EMF apposing current flow (like in normal operation), when you rapidly reverse, it augment the applied voltage with no way for the motor to instantly reverse rotation, so for a moment you have twice the voltage applied to the pure resistance of the coils. So this action can invoke stall-like issues in motor controllers (I've seen it blow out a motor controller chip on another system, with no load on the motors).

So, continue down the overload diagnostic path.

I'd consider swapping out that motor controller. It's possible that it has a partial failure that prevents if from dealing with a full over-load.

Phil.

[wilsonmw04]

Are you making your two motors per channel connection in the controller itself? IF so, having eight wires in such a crammed space can be your problem. It does sound like excessive current. If the motors are not under load, and it's still happening, make sure you are not shorting the wires right at the controller. We have had that happen several times.

[F Elliott]

I believe I am seeing the same problem. I think the new v2 motors are the root cause.

CONFIGURATION:

#pragma config(Motor, mtr_S1_C1_1, LeftMotor, tmotorTetrix, openLoop, encoder)
#pragma config(Motor, mtr_S1_C1_2, RightMotor, tmotorTetrix, openLoop, reversed)
#pragma config(Motor, mtr_S2_C1_1, FlagMotor, tmotorTetrix, openLoop)
#pragma config(Motor, mtr_S2_C1_2, LiftMotor, tmotorTetrix, openLoop)


Robot is up on blocks. FlagMotor and LiftMotor are DISENGAGED from the gear drives; i.e., no load.

TEST #1:
LeftMotor: v2 Tetrix DC motor
RightMotor: v2 Tetrix DC motor
FlagMotor: ORIGINAL Tetrix DC motor
LiftMotor: v2 Tetrix DC motor

Robot drives just fine, low to max power, forward and reverse. FlagMotor is also fine. No surprises.

However, whenever you run LiftMotor (the third v3 motor in the set) for more than a few seconds, the NXT "loses connection", the downstream servo controllers go ape, and everything freezes, requiring a hard boot of the NXT and usually ROBOTC.

I've tried various changes in pragma (daisy chain, parallel, swapped the controllers...) to no effect. I have not tried swapping the order of the motors in their sequence.

TEST #2:
LeftMotor: v2 Tetrix DC motor
RightMotor: v2 Tetrix DC motor
FlagMotor: Original Tetrix DC motor
LiftMotor: ORIGINAL Tetrix DC motor

NO CRASH! Everything seems to run just fine with one of the original motors as the fourth motor. In fact, I can run all four motors simultaneously at max power, forward and reverse like a madman.

Now my problem is this: As a rookie team, we were shipped four of the original motors with the plastic gears in our KOP. LEGO Education recently sent us 4 v2 motors as replacements. We have not received our "gear replacement" kits. Our LiftMotor probably cannot be relied on as a plastic gear motor. Should I swap the metal gears from one of our v2 motors into our original plastic gear motors?

WHAT A MESS. We've also been unable to get an encoder to work so we're having to run open loop in autonomous. Advice on that problem is we got a bad encoder. Our first competition is next Saturday and we can't even get past our basic motor usage due to problems created by the exclusive vendor's failure to manage their supply chain. I imagine you guys have been living with this for years but as a rookie coach I find this especially shocking.

I am in a near panic at this point. The team has worked so hard and mechanically our robot is finished. However, we'll be lucky at this point to show up with a robot that does little more than drive.

[aklego]

I believe I am seeing the same problem. I think the new v2 motors are the root cause.
...

TEST #2:
LeftMotor: v2 Tetrix DC motor
RightMotor: v2 Tetrix DC motor
FlagMotor: Original Tetrix DC motor
LiftMotor: ORIGINAL Tetrix DC motor

NO CRASH! Everything seems to run just fine with one of the original motors as the fourth motor. In fact, I can run all four motors simultaneously at max power, forward and reverse like a madman.


This is extremely interesting. Anyone have any ideas? Our problem was remarkably similar. We "solved" it by adding a dedicated controller but still have no idea what the fundamental issue is.

[F Elliott]

I tried swapping order of the motors. Order doesn't seem to matter. Two total v2s on two motor controllers (combined or split) works fine. Add a third v2 to either controller and kaput.

We do not have a third motor controller or I would try it. We are going to use a plastic gear original motor and hope for the best. Sigh.

[Al Skierkiewicz]

F,
Everything you describe makes me think the fourth motor is drawing tons of current when used with the lift. If the old motor works and does the job, I think that is your fix. Of course you can swap the flag motor for a V2 motor and that should be ok if you are running low on motors. However, it is also possible your electrical layout is putting some electrical loss into the lines feeding from the battery out to the various other systems. My suggestion is to make the wires the largest gauge and the shortest possible run you can make. I am still confused as to why the NXT locks up during this event. I can only guess that the I2C data gets corrupted during the brownout.

[F Elliott]

During the test, all the motors are unloaded.They are free running. The motor leads are those supplied by Tetrix.

Our "old" motors were shipped to us with plastic gears. We wanted to use one v2 motor on the lift and two more as drive motors so we don't run the risk of stripping the plastic gears.

Any two v2 motors work in any combination or purpose. Add a third v2 anywhere and the controllers go wacko.

[Al Skierkiewicz]

F,
As I understood your statements, the lift motor is not unloaded but being used for the lift. Is this an incorrect assumption? A picture of the wiring would be a great help.

[mikets]

Our team has very similar problems. This actually started last year where the motor controllers would freeze when the robot ran into things hard and we would be dead in the water until we power cycled the main battery. We tried everything from swapping motor controllers to tracing through wiring, swapping and bypassing ports. Nothing worked then the season ended. Then this year with a new game and new robot design (but reusing old controllers), we started okay but then the problem started to appear again. Eventually, we swapped the NXT brick with my personal brick and the problem has disappeared so far. But not knowing why and the real root cause, it may be a time bomb. We took apart one of the Tetrix controllers and ohm check the two RJ12-like connectors and found that 5 out of 6 pins are tied from one port to the other. My understanding is that the port contains an analog pin. So the 1 pin that's not tied to the adjacent port must be the analog pin. But this means the I2C bus is a passive pass-through. If that's the case, the theory on one controller froze causing all the down stream controllers to malfunction is now questionable. Don't know if it's related but our robot suffered serious electro-static discharge issue this season. Every time there is a discharge, our brick would hang causing us to lose half of our matches. That was a terrible day apparently because the weather was too good and too dry. It might explain why when we swapped the brick to use my personal brick, our motor controller problem also went away (at least for now). My personal brick was never used in competition so it never suffered any electro-static discharge as the team bricks have suffered for so many years. I would really like to get to the bottom of this issue so I don't have to worry about the time bomb blowing up on us (especially during competition).

[F Elliott]

Thanks Al for your suggestions. I wasn't clear earlier. In the test the lift motor's gear was removed from the drive shaft so there was no load on he motor when it was ran.

We ran out of time and went with a plastic gear motor last Saturday. We didn't have a problem with that. whew! I plan to look into this whole issue more carefully when the season is over.

Static electricity history of the NXT... Hadn't thought of that as a possibility. We only have our one NXT so can't test that but maybe I can borrow one.

[Al Skierkiewicz]

I was wondering what battery you are using for the NXT and what version brick do you have? years ago we would fight loose battery terminals that would break open with a robot hit. Just that small interruption would halt instruction execution. A little tender loving care (stretching the battery springs) and some alcohol on a Q tip did wonders for the reliability.

[F Elliott]

It's a new NXT that came in this year's competition kit with the rechargeable battery pack. We put the battery pack in it the first day and it's never been removed.

We are going to relocate the NXT for regionals so we'll have it pulled out within the next week. I will look carefully at this as a possibility.

See my new post today concerning losing control during the match.
http://www.chiefdelphi.com/forums/showthread.php?t=124628

We've got a gremlin somewhere in our robot and it may indeed be unstable NXT power. It's one of the few things we haven't seriously looked at by now.

[Al Skierkiewicz]

OK,
The other really common issue is the Samantha module losing USB connectivity. You can usually tell if you can see the tally lights. Power is critical for the Samantha module and is often the problem with connectivity issues. The USB cable is also an issue if it vibrates during game play. It should be secure at both ends. It is not designed for vibration.

[F Elliott]

I just got through watching some videos of our matches trying to watch the lights. The all seem to be going crazy off and on but I can't really tell since there is so much camera movement and people jumping up and down. These parents really need to be more calm at these things LOL

[mikets]

In our case, we used Bluetooth and have verified at the time of the freeze Bluetooth was still connecting. We verified by turning on the "audible feedback" of the communication. So when it is still connected, the brick beeps. Also, we have two motor controller chains, one on sensor port 1 and one on sensor port 2. It was the chain in sensor port 1 that was frozen, the motors off the chain of sensor port 2 were still functional so we can raise the arm and slider etc. The chain off sensor port 1 is the drive train. That prompted us to replace the brick, thinking sensor port 1 somehow is damaged.

[aklego]

I came across another team's robot with the same problem today--4 V2 motors on single controller. They would randomly cut out while unloaded and the controller would need to be power-cycled to force a reset. There were no shorts. I do not have an explanation but the V2 motors seem to be the common denominator.

[VashonRobotics]

We ran into this very problem at our last competition. Two of the DC motors on the bot are the new style. Symptom: Sam module lights on, LEDS to motor controllers on. No control from game controllers EXCEPT the one NXT motor that the bot has, it still worked normally. All wiring was gone through - no tinned wire ends in motor controllers just bare copper, Anderson Power Pole connectors at battery, short USB cable with ferrite core. I think this tells us that the game controller, FCS, Sam module, and NXT were all working, the problem had to be in the Motor Controller(s). A power off and on again and it worked fine.

[aklego]

We ran into this very problem at our last competition. Two of the DC motors on the bot are the new style. Symptom: Sam module lights on, LEDS to motor controllers on. No control from game controllers EXCEPT the one NXT motor that the bot has, it still worked normally. All wiring was gone through - no tinned wire ends in motor controllers just bare copper, Anderson Power Pole connectors at battery, short USB cable with ferrite core. I think this tells us that the game controller, FCS, Sam module, and NXT were all working, the problem had to be in the Motor Controller(s). A power off and on again and it worked fine.

This is truly odd. It is still anecdotal but there are enough teams having issues with this configuration for me to believe the V2 motors are at least the trigger.

[pyroslev]

We ran into this very problem at our last competition. Two of the DC motors on the bot are the new style. Symptom: Sam module lights on, LEDS to motor controllers on. No control from game controllers EXCEPT the one NXT motor that the bot has, it still worked normally. All wiring was gone through - no tinned wire ends in motor controllers just bare copper, Anderson Power Pole connectors at battery, short USB cable with ferrite core. I think this tells us that the game controller, FCS, Sam module, and NXT were all working, the problem had to be in the Motor Controller(s). A power off and on again and it worked fine.

Had a team with almost the exact issue in a Semi-Final Match. They were using a loaner NXT from another Team that had some age on it (Screen was scarred). I'm not sure if it was their NXT or the V2 motors.

[F Elliott]

We are trying to get four v2 motors to work in our drive system. We got away with two v1 motors for our Qualifier but we were the slowest robot on the mat.

Four v2 motors on two controllers, one motor per port. Servos either upstream or downstream twitch out of control whenever the motors are running.

Here is a critical new piece of information:

1. The twitching is dependent on power setting to the motors.

So we started isolating the motors ONE AT TIME and as luck would have it the last motor running on his own still caused the twitching. So we isolated it and put back the other motors one at a time and everything is good. We pulled a brand new V2 motor out of the box and hooked him up to the the bad motors DC connector and all is good.

Conclusion:
We had a defective v2 motor which was affecting everything. The more power you command to it the worse it gets. I suspect this motor has some sort of intermittent short or weakness in it or something to cause temporary high current demand that pulls power below that needed for stable power to the servo controllers.

I believe now this was the same motor I tried for our lift motor in December. It was probably shipped as defective. Like an idiot I didn't label them and just put them back in the kit.

To trouble shoot, isolate and swap v2 motors one at a time. You might have one or more defective v2 motors in your robot.

[F Elliott]

We are now running four v2 motors in our drive system and one v2 as our lift motor with no problem.

The four drive motors are wired one per port, two front drives on one motor controller, two ack motors on another, and the lift on a third. They are wired parallel into the NXT. I suspect if we put them back onto one chain it would be fine.

The problem was the defective v2 motor was hosing the DC power in the robot.

Note to self: this now makes one encoder, one servo controller, and one DC motor we received as defective from PITSCO.

Off season task: Develop a full suite of rigorous acceptance tests for anything we get from PITSCO before it ever goes near the robot.