Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[DonRotolo]

Originally Posted by techhelpbb

If it were a breaker issue, I would expect it to trip and stay that way.

You'd be wrong. They trip for anywhere from a fraction of a second to a second or two. Self-resetting.

Originally Posted by techhelpbb

multimeter for resistance from the lugs that would go on the Jaguar back to the power board.

A multimeter was applied to the Jaguar input power while it was run off the ground no major issues were noted (but multimeters are fairly slow to measure).

Nyet. Measure voltage drop along the wire - from power board to input screw, and output screw to motor connector - while passing high current. Only way to spot a bad wire (in this case) is under load. An Ohmmeter will steer you wrong.

Originally Posted by techhelpbb

When the robot was misbehaving I did put an oscilloscope one the power supply once it was stationary and I saw no major issues that would account for the fact that the Jaguars at that time were basically locked up.

When they are 'locked up' there's no current draw, you won't see anything of interest - too late in the process.

I think 1676 will be visiting 11 soon, we'll look at the practice bot with you then.

Back to basics. Maybe build a test platform without external influences - just the bare basics - and see if you can reproduce it on the bench. Time-consuming, yes, but ultimately can be valuable. And it'll keep the students involved.

[techhelpbb]

Originally Posted by DonRotolo

You'd be wrong. They trip for anywhere from a fraction of a second to a second or two. Self-resetting.
Nyet. Measure voltage drop along the wire - from power board to input screw, and output screw to motor connector - while passing high current. Only way to spot a bad wire (in this case) is under load. An Ohmmeter will steer you wrong.
When they are 'locked up' there's no current draw, you won't see anything of interest - too late in the process.

I think 1676 will be visiting 11 soon, we'll look at the practice bot with you then.

Back to basics. Maybe build a test platform without external influences - just the bare basics - and see if you can reproduce it on the bench. Time-consuming, yes, but ultimately can be valuable. And it'll keep the students involved.

That's fine with me.
I just want nice...clean...functional solutions.

We ran this robot off the ground for 20 minutes at a clip under test and it was fine, though in fairness I was measuring from the battery negative to the Jaguar positive input terminal. Not actually measuring the difference across the wires.

However, again...if there is a problem here we've missed under test...then let's remove it.

I don't like all the effort we had to pump into this any more than anyone else.

[jhersh]

Originally Posted by PhilBot

The problem is that this "0.0 postion" error persisted for many reboots, debug download etc. and then just returned to normal and hour later.

Sounds like you had a short between the wiper and ground on your pot.

Originally Posted by PhilBot

The Jags are switched between "Voltage out" and "Position mode" each time a preset recall button is pressed (and held). Before the new mode parameters are loaded, the JAGs are disabled, and then re-enabled at the end. All of the JAG parameters (control mode, position source, pot turns, PI gains etc) are re-loaded, and the code checks for any errors and re-tries the command sequence 3 times if errors persist.

Note: We'd run into the occasional bad-command-write problem several days before shipping and had built in the retries. So it's pretty bullet proof.

Sounds like a nice implementation. We did something similar last year and had no problems like this. The main difference between our setup and yours is we were using an encoder instead of a pot.

Originally Posted by PhilBot

Do you think accidentally sending the JAG an invalid "position value" (eg: a negative value for target position) could lock up the controller????

I don't think it should matter, though I can't say I've explicitly tested for that. If you can reproduce it with a simple program, let me know.

-Joe

[nuttle]

FWIW, team 2641 also had CAN timeout issues at init time, using a 2CAN. We have 8 Jags on CAN, and pretty consistently saw errors before we ever started to drive, with the robot on a stand and so not much current being drawn from any of the motors (all 8 were part of a 4 drive/4 steer holonomic drive system). Also, this happened with a fresh battery. We saw some non-zero error counts when using the 2CAN web page (the 2CAN was only connected to a laptop in this configuration). We tried using the 2CAN web page to drop the bus speed for the CAN bus, but this didn't seem to help. We eventually switched to using the serial connection, and everything cleared up. Our 2CAN is from last season. We have short CAN cables (the Jags are all reasonable close to each other) and a good terminator. However, we do not use twisted pair wiring for the CAN bus, is this reccomended with the 2CAN? Just to note, the Jags are all powered by short 10-gauge connections to the 8 40-amp connections on the power board.

We'll try the new 2CAN firmware, but any other helpful advice would be appreciated. For other teams who might have trouble in the future, trying to lower the speed of the CAN bus to the lowest setting is probably worth a shot. Also, having code that handles timeout exceptions is a very good idea (we use Java and have a state machine that cycles through all of the steps we need to set up a Jag, this checks the power cycled condition after any error and if there has been a power cycle, resequences through the initialization states).

[Phalanx]

Originally Posted by jhersh

We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.
-Joe

As week 4 events will begin this coming week, I'm wondering if sufficient testing has been done to allow for a release of this new image?

[MikeE]

At the risk of being redundant and rehashing the problems that are described above and on other threads, we've identified two distinct problems with our Java / CAN system (we're using the BlackJag serial/CAN bridge).

1) Intermittent initialization failures. This always occurs as a failure of all Jags so we initially checked the cabling/termination extensively on the bridging Jaguar. Typically a power cycle of the robot fixed the issue, but sometimes a cRIO reboot worked and sometimes it persisted over multiple reboots. It seemed to be more prevalent on our practice bot than competition bot, but that may be due to a side effect of a higher frequency of power cycling on practice bot. We had one total failure to move in eliminations at Chesapeake which may have been caused by this problem. We didn't get the memo which provoked some teams to abandon CAN in favor of PWM. We will be implementing the initialization retry work-around in code on the practice bot once it's been de-cannibalized. I hope the potentially imminent NI patch solves the root cause.

2) Stuttering. This is not strictly a CAN problem because we see all outputs disabling briefly. Occurs both in autonomous and teleop, and looks quite dramatic when pneumatic solenoids are briefly disabled and our arm mechanism looks to be giving a round of applause. Originally appeared to be a code problem in teleop, but refactoring didn't solve the problem. I don't believe we've seen the problem in competition. We will check for Jags reporting a powercycle in case this is a voltage problem, although I don't think that's likely.

[martin417]

I don't think anyone has mentioned this before so I will bring it up. We're using the BlackJag serial/CAN bridge. We also have limit switches connected directly to one of the Jags. The only time we have ever had the scrolling CAN error is when the CrIo is booted with one of the limit switches depressed. Even with a limit switch depressed, it only happens about 5% of the time. We have never had the issue if no limit switches are depressed.

Part of our pre-match checklist was to make sure that the limit switch was not depressed. Naturally, the only time we forgot to check, we got the errors and didn't move at all that match. After that, we removed that limit switch and depended on the encoder to stop the mast at the bottom. We didn't have any more problems.

[PhilBot]

Originally Posted by jhersh

Sounds like you had a short between the wiper and ground on your pot.

-Joe

Unlikely. It hapenned on both arm joints at the same time. I also measured the return wiper voltage and both changed when the joints moved.

The only thing these joints had in common (other than system power) was the CAN bus and the cRIO/code.

Phil.

I tell you, there is weird S**t going on with this system.

[techhelpbb]

I am suspicious of something and I'll share it here merely as speculation until I can prove it.

If you look at the schematics of the Jaguars (both models) you'll note that in all cases virtually every single interface on the Jaguar reference either both their internal ground...and one of their internal power supplies, or just ground.

The exception to this rule is the PWM input...which does in fact have a logic level optoisolator.

Now...the people using PWM don't read errors out...but in using PWM they are isolated from the Jaguar's internal power supplies entirely.

Anything you put on the Jaguar other than that input has the ability to directly impact the internal power supplies...and unlike the input power to the Jaguar which is heavily filtered because of all those power supply circuits...you'd be impacting in some cases the very same power that runs the Jaguar's microcontroller...and provides signal conditioning for it's analog to digital circuits.

I think this is the sort of thing that could explain why...when people hang even modest limit switches off the Jaguar...odd things can happen...but don't always happen.

Further...CAN is a robust bus...but if the circuits within the Jaguar's suffer from a power supply issue...these issues might translate into things like CAN timeouts.

If you think about it...you are basically putting a long antenna on the Jaguar's internal power supply whenever you connect a sensor.

*I am merely speculating.* I am awaiting the opportunity to test this. I am also accepting the possibility from other posters in this topic that...at least in the case of our issues...it might be a wiring problem.

[kamocat]

This is a good point, but it doesn't seem to add up.
Most of the inputs have a resistor (1k or greater) inline with their power supplies. There are two exceptions to this: the encoder and the brake/coast jumper.
Furthermore, limit switches are active when they are left OPEN. Their default state is closed (when the jumper is installed). This means a limit switch must be "normally closed", and break the connection when pressed. When a limit is pressed, it is drawing LESS power than when it is.

It was a good suggestion, though. I'm curios how close the Jaguar is to drawing more than their power supplies can handle.

[techhelpbb]

Originally Posted by kamocat

This is a good point, but it doesn't seem to add up.
Most of the inputs have a resistor (1k or greater) inline with their power supplies. There are two exceptions to this: the encoder and the brake/coast jumper.
Furthermore, limit switches are active when they are left OPEN. Their default state is closed (when the jumper is installed). This means a limit switch must be "normally closed", and break the connection when pressed. When a limit is pressed, it is drawing LESS power than when it is.

It was a good suggestion, though. I'm curios how close the Jaguar is to drawing more than their power supplies can handle.

I'm not entirely thinking in terms of DC voltages and currents.

I'm thinking of it more like an antenna...merely a piece of wire.

In some cases they have it plugged directly into the microcontroller.

Take for example the limit switches...no AC decoupling capacitors.
Even the potentiometer input...no AC decoupling.

Even the grounds could be an issue if you extend the ground plane out the wrong way.

In the black Jaguar the 5V power supply is a: TPS54040
"3.5V to 42V Input, 0.5 A Step Down SWIFT™ Converter with Eco-Mode™ "

http://focus.ti.com/lit/ds/symlink/tps54040.pdf

The 3.3V regulator is:TPS73633
"Single Output LDO, 400-mA, Fixed (3.3 V), Cap-Free, Low-Noise, Reverse Current Protection"

http://focus.ti.com/docs/prod/folder.../tps73633.html

The microcontroller the: LM3S2616

http://www.luminarymicro.com/product...html#Datasheet

[kamocat]

The trouble with using AC decoupling capacitors is that they only work on an oscilating signal. Limit switches and potentiometers are often very constant.
I suppose you could put opto-isolators on the limit switch inputs and the coast/brake.
Potentiometers, however, I would expect to be noise-resistant. You could still put a op-amp inline if you want.

To answer my own question, the 3.3v power supply has about 15.5 mA of load on it, assuming the break/coast jumper and JTAG aren't shorted.
The 5v power supply has the 3.3v power supply, plus another 193mA of load.
(The actual loads may be less than what I've calculated; I used the maximum current draw. This assumes the encoder input is not shorted)

[techhelpbb]

Originally Posted by kamocat

The trouble with using AC decoupling capacitors is that they only work on an oscilating signal. Limit switches and potentiometers are often very constant.
I suppose you could put opto-isolators on the limit switch inputs and the coast/brake.
Potentiometers, however, I would expect to be noise-resistant. You could still put a op-amp inline if you want.

Use the capacitors as a part of a filter or as bypass for frequencies that should not be there. After all a potentiometer and limit switch should be pretty constant as you've said.

[MikeE]

Originally Posted by martin417

... The only time we have ever had the scrolling CAN error is when the CrIo is booted with one of the limit switches depressed...

I've read several comments about scrolling messages in the diagnostic tab when a CAN timeout occurs, but I don't think I've ever seen them in our system.
During development we write to System.err and in competition to the DS display with code like this:

try {
    driveMotor.setX(something);
} catch (CANTimeoutException e) {
    DSmessage.getInstance().println(Line.kUser4, 1, "CAN timeout on drive");
}
...
DSmessage.getInstance().updateLCD();

Since there seems to be a built in logging facility in the diagnostic tab I'd prefer that they appeared there. Also the FTA(A) is more likely to check that location if problems occur on the field.

Can anyone share a Java snippet that sends CAN timeouts to the diagnostic tab?

[jhersh]

Originally Posted by Phalanx

As week 4 events will begin this coming week, I'm wondering if sufficient testing has been done to allow for a release of this new image?

Indeed... You can find the update here: http://firstforge.wpi.edu/sf/go/proj...e_frc_2011_v29