Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Phalanx]

Originally Posted by jhersh

We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.
-Joe

As week 4 events will begin this coming week, I'm wondering if sufficient testing has been done to allow for a release of this new image?

[MikeE]

At the risk of being redundant and rehashing the problems that are described above and on other threads, we've identified two distinct problems with our Java / CAN system (we're using the BlackJag serial/CAN bridge).

1) Intermittent initialization failures. This always occurs as a failure of all Jags so we initially checked the cabling/termination extensively on the bridging Jaguar. Typically a power cycle of the robot fixed the issue, but sometimes a cRIO reboot worked and sometimes it persisted over multiple reboots. It seemed to be more prevalent on our practice bot than competition bot, but that may be due to a side effect of a higher frequency of power cycling on practice bot. We had one total failure to move in eliminations at Chesapeake which may have been caused by this problem. We didn't get the memo which provoked some teams to abandon CAN in favor of PWM. We will be implementing the initialization retry work-around in code on the practice bot once it's been de-cannibalized. I hope the potentially imminent NI patch solves the root cause.

2) Stuttering. This is not strictly a CAN problem because we see all outputs disabling briefly. Occurs both in autonomous and teleop, and looks quite dramatic when pneumatic solenoids are briefly disabled and our arm mechanism looks to be giving a round of applause. Originally appeared to be a code problem in teleop, but refactoring didn't solve the problem. I don't believe we've seen the problem in competition. We will check for Jags reporting a powercycle in case this is a voltage problem, although I don't think that's likely.

[martin417]

I don't think anyone has mentioned this before so I will bring it up. We're using the BlackJag serial/CAN bridge. We also have limit switches connected directly to one of the Jags. The only time we have ever had the scrolling CAN error is when the CrIo is booted with one of the limit switches depressed. Even with a limit switch depressed, it only happens about 5% of the time. We have never had the issue if no limit switches are depressed.

Part of our pre-match checklist was to make sure that the limit switch was not depressed. Naturally, the only time we forgot to check, we got the errors and didn't move at all that match. After that, we removed that limit switch and depended on the encoder to stop the mast at the bottom. We didn't have any more problems.

[MikeE]

Originally Posted by martin417

... The only time we have ever had the scrolling CAN error is when the CrIo is booted with one of the limit switches depressed...

I've read several comments about scrolling messages in the diagnostic tab when a CAN timeout occurs, but I don't think I've ever seen them in our system.
During development we write to System.err and in competition to the DS display with code like this:

try {
    driveMotor.setX(something);
} catch (CANTimeoutException e) {
    DSmessage.getInstance().println(Line.kUser4, 1, "CAN timeout on drive");
}
...
DSmessage.getInstance().updateLCD();

Since there seems to be a built in logging facility in the diagnostic tab I'd prefer that they appeared there. Also the FTA(A) is more likely to check that location if problems occur on the field.

Can anyone share a Java snippet that sends CAN timeouts to the diagnostic tab?

[jhersh]

Originally Posted by Phalanx

As week 4 events will begin this coming week, I'm wondering if sufficient testing has been done to allow for a release of this new image?

Indeed... You can find the update here: http://firstforge.wpi.edu/sf/go/proj...e_frc_2011_v29

[JasonStern]

Sadly to say, update 29 did not fix the can timeout issues for us at the dc regional. We have switched to pwm instead for the remainder of our matches.

Thanks to everyone who is trying to resolve these issues!

-Jason

[drakesword]

346 had issues at DC as well. During a couple of matches the speed of the robot would unexpectedly drop or the robot would begin shuddering (not a high speed shudder that a PID issue would show but a slow 1 second on 1 second off type issue). Finally in one of our last matches the second half of the can bus did nothing.

What is the upper limit of temperature before the jags begin to shutdown?

We did some test with our robot and we are still coming up inconclusive.

First we ran the robot and got no errors.
Then we unplugged a jag while running and still had no errors.
Then we unplugged a terminator without errors.
Unplugged any jag-jag wire and the entire bus stopped (not just beyond the lost connection)
Finally we unplugged encoders and still had no error.

We were running v29.

During the build season we were using the older image and ran the chassis for over 30 minutes without problems (except the one time we blew a grey jag by going full forward to full reverse too fast)

[Dad1279]

I'm sorry to hear that there were still issues with CAN at DC.

After our two matches at NY with 'NO COMM' issues, we thoroughly tested and could not come up with any ideas but to switch to PWM for DC.

We had no issues running PWM at DC.

[Phalanx]

We ran all of the DC Regional with CAN Jaguars in voltage control mode.
We had no encoders, pots, or limit switches in use on the JAGs.
We programmed in Labview, with Update 29.

We used encoders and limit switches attached to the digital side car.

We ran all of DC Regional, 9 Qualifiers(We missed one due to mechanical repairs), 2 Quarter Finals, and 2 Semi Finals matches. We had ZERO CAN issues.

I think it's time to investigate all of the following to attempt to find a commonality for failure.

1) Correct/Proper wiring both CAN and Power.
2) Ground isolation of components.
3) CAN Interface used, Serial or 2CAN
4) Type of JAGS used, Black, Grey, Both
5) The programming language used
6) The control mode(s) used.
7) encoders, pots, limit switches are being used.

So for us, I can say with 99.99999% certainty that for our machine...
1) CAN Wiring is correct with proper termination 100ohm.
2) All components are ground isolated from the frame and electrical wiring has no shorts or ground faults.
3) We used 2CAN on PORT 2 of the CRIO.
4) We used all BLACK JAGS.
5) We Programmed with Labview.
6) Used Voltage Mode only.
7) No Encoders, POTS, Limit Switches attached to the JAGS.

[heydowns]

v29 solved the problems we were seeing with our CAN-connected Jaguars. We no longer get random failures of the entire bus to work on robot startup as described in my earlier post.
We ran with v28 until Thursday lunch this past week and saw the boot-time failure twice during that morning.
After Thursday lunch when we put v29 on, the CAN chain (7 Jags) worked without issue for the remainder of the competition.

Originally Posted by drakesword

346 had issues at DC as well. During a couple of matches the speed of the robot would unexpectedly drop or the robot would begin shuddering (not a high speed shudder that a PID issue would show but a slow 1 second on 1 second off type issue). Finally in one of our last matches the second half of the can bus did nothing.

Sorry to hear you had problems, same for 1123 We were at DC too - wish I'd have know I would have come taken a look at your setup and try to offer some suggestions.


The CAN driver (at least for C++ maybe the others I have not looked) blocks while waiting for CAN replies to commands that are sent. If you have a Jag that is disconnected, failing, powered off, or has a poor bus connection the driver will block for a "long" time waiting for that response, thus stalling your overall code execution. This might or might not account for the slow on/off issues you describe. Of course that doesn't explain the root cause.

We've actually modified our copy of the CAN driver to shut down a device that's failing communications (repeatedly) rather than continuing to bog down the entire system in order to mitigate this.

One thing I've seen helping teams with CAN issues is that termination resistors connected in the manner described in the Jaguar documentation (that is, crimping them directly into the RJ connector on the appropriate pins) is a potential source of intermittent bus errors, particularly during match play when vibration and collisions are occurring a lot. Instead of making the cable in this fashion I recommend crimping in a short length of wire and soldering the resistor to that.
Again, not necessarily the problem your team is encountering but something worth trying/checking along with double-checking all your other bus cabling.

Also, on a very general note, as I read back on this thread it is really apparent that a number of different issues are being referenced here (symptoms may seem similar but they are caused by different things). For those just coming to this thread please keep that in mind and if you post your experiences please be as specific as possible.

[taichichuan]

Hi Gang!

Team 116 used CAN at the DC regionals as well with V29 and saw the exact same type of stuttering behavior as many of the other teams:

1) Correct/Proper wiring both CAN and Power.

This is our 2nd year with CAN. I check the cables with my automated cable tester and all checked good. Termination using the 2CAN on one end and 100 Ohm terminator at the other.

2) Ground isolation of components.

Our Jags are on standoffs and are isolated from the robot chassis ground.

3) CAN Interface used, Serial or 2CAN

We were using the 2CAN with firmware 2.5 (3/13/11). This worked well at the Bayou regional except for one time with the CAN connect problem from V28.

4) Type of JAGS used, Black, Grey, Both

Mixture of both black and grey jags. All jags at Firmware V92.

5) The programming language used

C/C++

6) The control mode(s) used.

Voltage mode (we were trying to use position mode for autonomous but ran out of time)

7) encoders, pots, limit switches are being used.

No encoders, pots or limit switches were being used.


I spoke to Brad Miller at DC and described the problem and we were actually able to see a little of the problem in the pits with him standing there. He suggested changing the motor safety timeout, which we did. However, in the heat of eliminations, I don't recall if we saw the stuttering post change.
__________________

HTH,

Mike

[drakesword]

1) Correct/Proper wiring both CAN and Power.
Going to rewire most of the can-related components at VCU. We were using 6 conductor wire. Going to switch to two. 120 ohm terminator soldered to wire not crimped.

2) Ground isolation of components.
Again as above. possible ground loop through our can wires

3) CAN Interface used, Serial or 2CAN
Serial

4) Type of JAGS used, Black, Grey, Both
Black (5)

5) The programming language used
Java.

6) The control mode(s) used.
Speed and Position

7) encoders, pots, limit switches are being used.
Us-Digital(I think) encoders 200 pulses per revolution. Don't know what switches we use but we have 2 on our gripper.

To address potential blocking we will make individual threads to set output to each motor.

[Phalanx]

Originally Posted by drakesword

1) Correct/Proper wiring both CAN and Power.
Going to rewire most of the can-related components at VCU. We were using 6 conductor wire. Going to switch to two. 120 ohm terminator soldered to wire not crimped.

We used 6 conductor wire, but we used 100ohm resistor as a terminator instead of 120ohm. I have heard second and third hand reports that using 120ohm terminators create issues, so I would try that before rewiring the entire CAN bus.

In the "Getting Started Guide" on Page 29 for the Black Jags the section "Jaguar Communication Cables" states 100ohm. Also see Table 8-1. CAN Wiring Parameters on page 26. http://www.luminarymicro.com/jaguar

[heydowns]

Originally Posted by drakesword

To address potential blocking we will make individual threads to set output to each motor.

Just a warning - the bus transactions are serialized via a monitor lock in the CANJaguar driver (at least on the C++ version, as well as in the last public source code I can easily get to of the Java version). Thus threading the transactions won't entirely alleviate any potential for blocking with respect to one another (that is, if one Jag thread were to block waiting for a bus response all other concurrently executing Jag threads will also block). Of course, it can offer the advantage of allowing the remainder of your code to execute while the Jag thread(s) are tied up.

[drakesword]

Originally Posted by heydowns

Of course, it can offer the advantage of allowing the remainder of your code to execute while the Jag thread(s) are tied up.

Thats the effect I was after

@Phalanx

According to some other threads the ground wire between jags can cause ground loop interference eg. the ground of one jag may be higher then the ground of another jag