Jaguars intermittently disconnecting

[Radical Pi]

Hi all,

I just updated our 2010 cRIO to the 2011 image and rebuilt the code against the 2011 libraries. After switching the jaguar code to the new WPILib version though, we are having some major issues. Jaguars have been updated to v92, and we are using a serial bridge.

When the code is enabled, at random intervals 1 or more motors stop for a few seconds, then continue moving again. On the jaguar when the motor stops the status LED is flashing yellow, when it is working it is the expected state. I've reduced the code down to just joystick input and output to the 4 jaguars with no change. I haven't seen it die though when using only two jaguars. Even when idle the status LEDs on the jaguars show the same symptoms. NetConsole gives no errors, MotorSafety has been disabled on all of the jaguars, and it works fine in BDC-COMM.

Any help would be appreciated

[Matt Krass]

Originally Posted by Radical Pi

Hi all,

I just updated our 2010 cRIO to the 2011 image and rebuilt the code against the 2011 libraries. After switching the jaguar code to the new WPILib version though, we are having some major issues. Jaguars have been updated to v92, and we are using a serial bridge.

When the code is enabled, at random intervals 1 or more motors stop for a few seconds, then continue moving again. On the jaguar when the motor stops the status LED is flashing yellow, when it is working it is the expected state. I've reduced the code down to just joystick input and output to the 4 jaguars with no change. I haven't seen it die though when using only two jaguars. Even when idle the status LEDs on the jaguars show the same symptoms. NetConsole gives no errors, MotorSafety has been disabled on all of the jaguars, and it works fine in BDC-COMM.

Any help would be appreciated

We had this problem, but it disappeared after a good reboot and code redeploy. That seemed suspiciously easy, so we'll keep an eye out for these symptoms. I suspect since you've made several alterations to your code I'm sure your cRio has been through a couple reboots, so clearly that isn't a solution.

Perhaps you have some CAN wiring in a bad place coupling in some interference from motors? That thought just occurred to me, and while I was doing my reboot, the wires had been rearranged (temporary test bed), so I may have incorrectly attributed the solution to the reboot.

Check and see where your wires are routed, if they're picking up EMI from the motors, it could be causing a communications fault.

Matt

[kamocat]

What happens when you decrease the rate you are sending the control messages?
Does this happen if you tell your Jaguars to have a certain output, and leave it at that, or does this only happen when you are continuously updating the outputs?

[Radical Pi]

I do have wires running a little close to a pair of motors, but the wires don't seem to be the problem since I still see the problem even if those motors aren't running. Also jaguars downstream of the fault don't always have the problem too and if I run the motors via BDC-COMM it seems to work fine.

This problem only appears to be happening under continuous messages

I haven't tried changing the rate of the main teleop loop, but I did see a difference when I lowered the code down to 2 jaguars instead of the 4 drive. Though that may have been because there were fewer active jaguars to see a problem on (I think the jags were still failing around the same rate)

One thing that may be of importance is that the heartbeat coming from the cRIO has never seemed to work on our network. When I run via BDC-COMM all of the jags turn solid, but when running through the cRIO the jags are only solid when they are receiving set commands from my code. (mainly a problem in autonomous, forced us to use a state machine last year)

[kamocat]

Could you check your termination resistor to make sure it has no chance of shorting?

It sounds like the rate you send messages DOES affect the problem, but I don't think it's the only issue.

[Matt Krass]

Originally Posted by kamocat

Could you check your termination resistor to make sure it has no chance of shorting?

It sounds like the rate you send messages DOES affect the problem, but I don't think it's the only issue.

More frequent messages means more datastreams to get interrupted by noise. I'm still leaning towards noise, we had the symptoms reappear today with some added CAN wiring on the system getting twisted up with the motor wiring. Shaking up the wires and pulling them apart seemed to resolve the problem.

[kamocat]

That's strange. CAN is a complementary signal, like USB and Ethernet. It shouldn't be susceptible to inductance like that.

Out of curiosity, what motors were you using? Did it matter whether they were stopped or at full power?

[Matt Krass]

Originally Posted by kamocat

That's strange. CAN is a complementary signal, like USB and Ethernet. It shouldn't be susceptible to inductance like that.

I would agree, and I'm sure it definitely helps, but it doesn't make it absolutely immune, just resistant. Perhaps seeing if you can get a scope on the pins and see if the signals are noisy?

[kamocat]

Originally Posted by Matt Krass

I would agree, and I'm sure it definitely helps, but it doesn't make it absolutely immune, just resistant. Perhaps seeing if you can get a scope on the pins and see if the signals are noisy?

To use the scope without removing your termination resistor, you can use a phone splitter, and make a pigtail to attach your leads to. Only the two center pins are important.

[jhersh]

Originally Posted by kamocat

To use the scope without removing your termination resistor, you can use a phone splitter, and make a pigtail to attach your leads to. Only the two center pins are important.

I would expect the serial connection to be more susceptible to noise than the CAN bus. Specifically susceptible to noise coupled from the CAN bus.

-Joe

[Matt Krass]

Originally Posted by jhersh

I would expect the serial connection to be more susceptible to noise than the CAN bus. Specifically susceptible to noise coupled from the CAN bus.

-Joe

This makes sense, especially since the wire we had tangled was coming from serial to the black Jag to be bridged. If the symptoms present themself again I will scope it and what I can find.

[kamocat]

Would you suggest shielding common potential, or shielding to the frame?

[Al Skierkiewicz]

This was a common problem last year and has more to do with one of the watchdogs. Joe, Andrew Watchom from NI identified it in Minnesota last year. It has something to do with restarting the code under certain conditions. Most often teams notice it as a compressor restart.

[kamocat]

Originally Posted by Al Skierkiewicz

This was a common problem last year and has more to do with one of the watchdogs. Joe, Andrew Watchom from NI identified it in Minnesota last year. It has something to do with restarting the code under certain conditions. Most often teams notice it as a compressor restart.

Al, could you clarify what the common problem was?
As far as I'm aware, CAN wasn't common enough last year to have any common problems.

Were teams running their compressor over CAN? I thought this was against the rules.

[Al Skierkiewicz]

Marshall,
This is not a problem strictly limited to CAN. As it is related to a code issue, CAN is not yet involved in the scheme of things. This is one of the problems with diagnosing the issue last year, in that both CAN on non CAN robots were affected, as were Labview and C+.