Jaguars intermittently disconnecting
 

Hi all,

I just updated our 2010 cRIO to the 2011 image and rebuilt the code against the 2011 libraries. After switching the jaguar code to the new WPILib version though, we are having some major issues. Jaguars have been updated to v92, and we are using a serial bridge.

When the code is enabled, at random intervals 1 or more motors stop for a few seconds, then continue moving again. On the jaguar when the motor stops the status LED is flashing yellow, when it is working it is the expected state. I've reduced the code down to just joystick input and output to the 4 jaguars with no change. I haven't seen it die though when using only two jaguars. Even when idle the status LEDs on the jaguars show the same symptoms. NetConsole gives no errors, MotorSafety has been disabled on all of the jaguars, and it works fine in BDC-COMM.

Any help would be appreciated

Re: Jaguars intermittently disconnecting
 

We had this problem, but it disappeared after a good reboot and code redeploy. That seemed suspiciously easy, so we'll keep an eye out for these symptoms. I suspect since you've made several alterations to your code I'm sure your cRio has been through a couple reboots, so clearly that isn't a solution.

Perhaps you have some CAN wiring in a bad place coupling in some interference from motors? That thought just occurred to me, and while I was doing my reboot, the wires had been rearranged (temporary test bed), so I may have incorrectly attributed the solution to the reboot.

Check and see where your wires are routed, if they're picking up EMI from the motors, it could be causing a communications fault.

Matt

Re: Jaguars intermittently disconnecting
 

What happens when you decrease the rate you are sending the control messages?
Does this happen if you tell your Jaguars to have a certain output, and leave it at that, or does this only happen when you are continuously updating the outputs?

Re: Jaguars intermittently disconnecting
 

I do have wires running a little close to a pair of motors, but the wires don't seem to be the problem since I still see the problem even if those motors aren't running. Also jaguars downstream of the fault don't always have the problem too and if I run the motors via BDC-COMM it seems to work fine.

This problem only appears to be happening under continuous messages

I haven't tried changing the rate of the main teleop loop, but I did see a difference when I lowered the code down to 2 jaguars instead of the 4 drive. Though that may have been because there were fewer active jaguars to see a problem on (I think the jags were still failing around the same rate)

One thing that may be of importance is that the heartbeat coming from the cRIO has never seemed to work on our network. When I run via BDC-COMM all of the jags turn solid, but when running through the cRIO the jags are only solid when they are receiving set commands from my code. (mainly a problem in autonomous, forced us to use a state machine last year)

Re: Jaguars intermittently disconnecting
 

Could you check your termination resistor to make sure it has no chance of shorting?

It sounds like the rate you send messages DOES affect the problem, but I don't think it's the only issue.

Re: Jaguars intermittently disconnecting
 

More frequent messages means more datastreams to get interrupted by noise. I'm still leaning towards noise, we had the symptoms reappear today with some added CAN wiring on the system getting twisted up with the motor wiring. Shaking up the wires and pulling them apart seemed to resolve the problem.

Re: Jaguars intermittently disconnecting
 

That's strange. CAN is a complementary signal, like USB and Ethernet. It shouldn't be susceptible to inductance like that.

Out of curiosity, what motors were you using? Did it matter whether they were stopped or at full power?

Re: Jaguars intermittently disconnecting
 

I would agree, and I'm sure it definitely helps, but it doesn't make it absolutely immune, just resistant. Perhaps seeing if you can get a scope on the pins and see if the signals are noisy?

Re: Jaguars intermittently disconnecting
 

To use the scope without removing your termination resistor, you can use a phone splitter, and make a pigtail to attach your leads to. Only the two center pins are important.

Re: Jaguars intermittently disconnecting
 

I would expect the serial connection to be more susceptible to noise than the CAN bus. Specifically susceptible to noise coupled from the CAN bus.

-Joe

Re: Jaguars intermittently disconnecting
 

This makes sense, especially since the wire we had tangled was coming from serial to the black Jag to be bridged. If the symptoms present themself again I will scope it and what I can find.

Re: Jaguars intermittently disconnecting
 

Would you suggest shielding common potential, or shielding to the frame?

Re: Jaguars intermittently disconnecting
 

This was a common problem last year and has more to do with one of the watchdogs. Joe, Andrew Watchom from NI identified it in Minnesota last year. It has something to do with restarting the code under certain conditions. Most often teams notice it as a compressor restart.

Re: Jaguars intermittently disconnecting
 

Al, could you clarify what the common problem was?
As far as I'm aware, CAN wasn't common enough last year to have any common problems.

Were teams running their compressor over CAN? I thought this was against the rules.

Re: Jaguars intermittently disconnecting
 

Marshall,
This is not a problem strictly limited to CAN. As it is related to a code issue, CAN is not yet involved in the scheme of things. This is one of the problems with diagnosing the issue last year, in that both CAN on non CAN robots were affected, as were Labview and C+.

Re: Jaguars intermittently disconnecting
 

His name is Andy Watchorn. A "watchdog error" as people are familiar with will make all motor controllers turn off at the same time. Not some of them on the bus. On the other hand, if noise is occasionally making the safety protocol fail to get through to a motor controller, then that controller will drop out until the next renegotiation. There is a 2 second renegotiation period.

-Joe

Re: Jaguars intermittently disconnecting
 

Well, I just confirmed with our student programmer that our watchdog has been disabled for all of these incidents. Also, we received no warnings about the watchdog not being fed, so I'm inclined to believe this is something else, unless there is a lower level watchdog potentially malfunctioning. We also experienced the same problem with BDC-COMM but an otherwise identical set-up, so I'm still leaning towards EMI.

Matt

Re: Jaguars intermittently disconnecting
 

That sounds like the symptoms I'm seeing. It doesn't explain though why this problem didn't appear until after I upgraded to v25 and v92.

Is there any code on the cRIO that could possibly be causing the set packets to be dropped? Also is the code supposed to send out a global heartbeat signal to the bus?

Re: Jaguars intermittently disconnecting
 

Joe,
Check with Andy, I never did fully understand the issue but when he explained it to a C+ programmer, he understood the problem and corrected it. (I am a hardware guy) That team had no further problems through the season and up to the Champ finals.
It manifested itself in a very similar manner to what has been described here. It did not show any watchdog error but it seemed to be less than a two second delay. If the robot was running, it would stop for a moment and then run normal. Since many teams used pneumatics last year, most manifestations occurred while observing the compressor.

Re: Jaguars intermittently disconnecting
 

It seems like a bug with communications, like I said before, we exhibited identical symptoms with the jaguar controlled by the desktop application with the cRio completely out of the loop. Unless the problem also exists in NIs application, that would seem to preclude it being a software issue.

My theory on EMI works if you think about the compressor kicking on causing a sudden surge of inrush current, or maybe IFI cheaped out and there's no snubbing diodes in those Spikes ;)

I don't know, I don't mean to seem stubborn, but I can't find any consistency with a software glitch.

Unfortunately I don't usually make it to the shop during the week (new job, don't want to start skipping out just yet!) so I won't be able to look in to this further until next weekend.

Re: Jaguars intermittently disconnecting
 

That's a key difference in our problems then. Driving all 4 drive motors at full speed via BDC-COMM makes no difference, so unless there is a interference problem with idle motors that only shows in the small gap between the cRIO and the black jag, there's something going on in software.

I'm fairly sure there is something weird that makes our setup different from NI's since I can find no evidence of a heartbeat signal being sent by the cRIO over the network even though Wind River shows FRC_JaguarHeartbeatTask (or whatever it's called) on the cRIO (though it's always shows as [delay])

and Al, this is a different problem. All the jaguars seem to fail independently of eachother, something a watchdog would not do (the new per-motor watchdogs have been disabled in my test program)

Re: Jaguars intermittently disconnecting
 

There's still a heartbeat (in fact, that's what keeps the motors enabled). However, I believe that's handled by the BlackJagPlugin.

Re: Jaguars intermittently disconnecting
 

are you sure the cables are good? i have had finicky cables before. usually a stripping error dmaging a wire or bad soldering on the termination.

also make sure they are firmly put in.

did you use 100 ohm resistors for termination? or 120? CAN standard is 120 but for robotics use you need to use 100.

Re: Jaguars intermittently disconnecting
 

This sounds very similar to a communication problem that I was having with my Jaguars. I built my serial cable/CAN terminator to the manual's specs which included putting the terminating resister in the shell of the DB9 connector on the cRIO side. This meant that the serial traffic and CAN traffic were running on parallel wire pairs for the length of the cable between the cRIO and the first Jaguar.

I learned from Joe H. that the BlackJag plugin forces the serial connection at 115200bps, which is pretty fast and is more susceptible to noise than slower bitrates. I suspect that CAN traffic was crosstalking with the serial traffic, since I'm using flat wire, and causing the serial connection to fail. My solution was to shorten the serial/CAN cable from 6ft to 2ft, and I haven't had any connection problems since then.

Joe also suggested moving the terminating resistor from the DB9 end of the cable over to the RJ-12 end. This would remove all CAN traffic from that cable and further decrease the potential for crosstalk or CAN-induced noise on the serial bus.

Re: Jaguars intermittently disconnecting
 

As far as I can tell that's not the case. Set commands are considered trusted and therefore can keep a jaguar alive. I've also never seen anything in the plugin to run the heartbeat.

In terms of my cables I haven't found anything wrong with them. Again it works fine in BDC-COMM all the way up to pulling the serial connector from the cRIO and plugging it into the laptop, leaving all the cables in the same configuration. I've checked that all connectors are seated properly, and there is no correlation between failure and physical location on the bus.

While the idea of interference occurring between the CAN and serial cables is reasonable, again BDC-COMM has shown no issues in communication in an identical setup (minus the cRIO), and under the (non-FRC version) source code runs under the same settings as the cRIO.

Re: Jaguars intermittently disconnecting
 

That is true. The Jaguar CAN Driver handles the heartbeats, but it also monitors the trusted commands. If you have sent a trusted command recently enough, then the heartbeat isn't sent. However, if you just tell the Jag to go to position X one time, the heartbeat will be sent every 75 - 90ms to keep the Jag running in the absence of trusted commands.

Are you running much other code on the robot at the time? Can you use the System Manager in LabVIEW under the Tools menu >> Real-Time Module >> System Manager to look at the processor utilization while the Jags are having this problem? I'm interested if this could be the cRIO being too busy with other things to keep up with all the Jags.

Thanks,
-Joe

Re: Jaguars intermittently disconnecting
 

Remember that the BDC-COMM does not put as much traffic on the CAN bus as the cRIO does, because the BDC-COMM usually sends messages to one Jaguar at a time, and only when you update a value. In contrast, your code is probably sending messages to four Jaguars every 20ms. (In my testing this took about 10-12ms for 4 Jaguars, but it may take longer if there are errors. I don't know what CAN does for error prevention, but I do know it has good error detection.)

Joe, must EACH Jaguar receive a trusted message to stay enabled, or will a trusted message to one Jaguar keep all of them alive?

Re: Jaguars intermittently disconnecting
 

Ah, that makes more sense. For some reason though that wasn't happening on last year's autonomous code. I'll check if there's a similar problem going on w/ v25. I assume the heartbeat is sent on a per-jaguar basis in this system?



I'll check when I get to the robot today, but this is happening under minimal code. The teleop loop is literally "leftA.Set(stickA.GetY());". I've also tried killing processes that seemed less important via Wind River (FTP task, RSL control, netconsole, etc) with no change.

Re: Jaguars intermittently disconnecting
 

Each Jaguar is independent. That's why one may drop out while others are fine.

-Joe

Re: Jaguars intermittently disconnecting
 

I was also able to successfully connect through BDC-COMM but not through the cRIO. I suspect that BDC-COMM connects with the Jaguars at a slower bitrate and is thus more tolerant of noise/crosstalk.

Here's the discussion thread where I outlined my problem and eventually found resolution via a shortened cable. Do my symptoms seem similar to your issue?

http://www.chiefdelphi.com/forums/sh...ad.php?t=89733

Re: Jaguars intermittently disconnecting
 

I got it working today by doubling the wait at the end of the teleop loop from 5ms to 10ms.

Joe, I tried to use the System Monitor in LabVIEW but it wanted a port number to connect to. I have no idea what that should be.

Re: Jaguars intermittently disconnecting
 

You need to enable the VI Server. Right click on the RT CompactRIO Target in your project and select Properties. Under VI Server: Configuration, turn on TCP/IP and leave it on port 3363. You should then be able to connect with the System Monitor.

-Joe