Reducing CAN network noise

[jwakeman]

Doing some testing on a prototype setup this weekend we encountered occasional comm faults in the CAN network that would cause the motors to shut off momentarily. We were running the motors continuously and every 30 seconds to 1 minute they would glitch. Each time this happened if we connected to the jaguars with BDC-COMM we could see a couple of comm faults had accumulated. The wiring was a bit of a mess on this setup (and we also discovered we had no isolation between the CAN network and the power electronics of the Jaguar as I described in this thread) so we suspected some noise on the network. We are using brand new jaguars, with a 2CAN and the can cables that came with the 2CAN. We had already checked the pins in the sockets on the Jaguar. So I am confident the noise/faults aren't coming from faulty cables or network termination or anything of that sort.

We later moved the cRIO, PD board, Jaguars and CIM motors to a table top (no metal frame, no other garbage to induce noise) and ran the same test where we were continously running the motors. This time no glitches, no comm faults, i.e. no noise. So we think if we do a quality job with wiring (as we normally do) on our final robot installation we can probably avoid noise issues. We plan to run signal wires separately from power wires, twist wires in pairs to cancel noise, we may even wrap the power leads in tin foil.

The question is have any other teams encountered noise on their CAN networks and found unique solutions (they would like to share) such as noise filtering circuits, cable routing techniques, etc? Replies with theoretical solutions are OK but I am really looking for things teams have done that worked.

[mikets]

I doubt it is noise in the CAN bus. CAN is designed for noisy environment. It is using differentiate pair so the noise on the wire can be canceled out. Are you daisy chaining the Jags? There are numerous posts talking about the RJ-11 connectors not making contact causing communication problem. When the robot is running on the ground, there is a lot of vibration that may cause RJ-11 contact issue. If you take the electronics out and run it on the table, the lack of vibration may explain why it didn't fail.

[jwakeman]

We have checked the ports on the jags and we have bought off the shelf cables to avoid some uncertainty there. I had the robot 'up on blocks' with the wheels spinning free in the air so no vibration.

When all the equipment was still on the test frame I connected to the jags with my PC and BDC-COMM and I could set a current reference and run the motors with no glitches. If I then just moved the cable from my PC to the cRIO and used my robot application on the cRIO to set a current reference the motors would occasionally glitch. This behavior was confirmed several times (worked with PC, glitched with cRIO). This is what gave me the idea to run it on the table top. I assumed the difference was that the PC and its power supply were well isolated from the CAN network, where the cRIO was not. The glitches also occurred with the cRIO whether we were connected via 2CAN or RS232. Once the equipment was moved off the metal frame and out the bird's nest of wires and run on the table top then no glitches.

If none of the teams that are successfully running CAN have done anything special to isolate the CAN network from the power network I am ok with that feedback. That just tells me if we carefully and cleanly install the equipment and wiring in our final robot then we should not have issues with comm faults on the CAN network. So instead of asking what have teams done to reduce noise maybe I should ask teams to voice if they have successfully run CAN with no comm faults without any special accomodations.

[mikets]

We have our test bot running on CAN. We don't have any issues with it (yet, fingers crossed) although we did not run it very hard either. The test bot is always in someone's house so there isn't enough space to run it hard. We did, however, have a brand new Jag died on us for unknown reason, no burn, just died. So we are RMA'ing it.

Disclosure: this is our first year switching to CAN Jaguars, so we cannot speak too much about our "experience". Since we have heard a lot of stories about the reliability of Jags, we took special care to address some of the "concerns" we have learned from the forums so far. For example, there were concerns on daisy chaining so if one segment of the cable had connectivity issue, all down stream jags would go. To address that, we use backbone configuration (see thread http://chiefdelphi.com/forums/showth...t=99554&page=3).

There were concerns about Jag's browning out, so we addressed that with software (http://chiefdelphi.com/forums/showthread.php?t=101871).

There was an issue with using the Jag's built-in PID to do speed control. So we addressed that with our own PID control in software.

So far, everything works fine. In the next few days, we will put all that to vigorous tests and see if they are still holding up. BTW, we also pre-wire all the Jags with PWM just in case CAN failed miserably or even swap to Victors if we have to. Hope we don't have to switch because we may not have enough time to test both configurations.

[mjcoss]

I'm not sure you can say it's noise given the tests that you've run. Running the CAN bus using BDC-COMM bypasses a lot of software/frimware on the cRIO and 2CAN. We have 2 platforms and both are seeing issues.

One interesting note is that on one of our setups where we are running in speed control mode, we see a glitch in the PID loop on a ~1 min interval. I initial thought that it might be just an overflow condition on the integrator term, so I set kI to 0. This appeared to make the glitch disappear but it came back last night. Then I read about the fact that if the Jag's brown out, they loose their configuration. I don't think our Jag's are browning out, but I'm adding code to check...of course it just adds to the bus traffic but I see no alternative. The configuration needs to be restored if the Jags do indeed brown out.

---Michael J Coss

[DonRotolo]

CAN is relatively noise-resistant, but wiring is important: Strive to use twisted pairs, particularly for longer runs (over 12" or so) and when near known sources of noise (such as brushed motors).

Automobile manufacturers have used CAN for years, and this is what they do.

[jwakeman]

Originally Posted by mjcoss

I'm not sure you can say it's noise given the tests that you've run. Running the CAN bus using BDC-COMM bypasses a lot of software/frimware on the cRIO and 2CAN.

Tests I ran:
1. cRIO w/ 2CAN on metal frame with mess of wires (test rig)- glitched
2. CRIO w/ serial cable on test rig - glitched
3. Laptop w/ BDC-Comm to jags/motors on test rig - NO glitches
4. cRIO w/ 2CAN on cafeteria table - NO glitches


I did suspect the cRIO software/firmware and also the 2CAN first. Testing with the serial cable eliminated the 2CAN (still glitched). Testing with the cRIO on the table top and seeing no glitches made me suspect noise in the test rig we had been using.

[mjcoss]

I agree that it certainly seems like a noise issue. We use very short, 4 - 6 inches, cables, just enough to hop from Jag to Jag. And we keep them away from power sources. There is also the whole issue of bad rj connectors on the Jaguars that people have been reporting, which makes me think it might also just be vibration within the robot that is potentially triggering a comm fault. Or a combination of any number of factors.

What I do know is it's a mess, and makes me wonder if using Victors or just PWM controlled Jaguars might be a better solution.

[jwakeman]

Originally Posted by mjcoss

What I do know is it's a mess, and makes me wonder if using Victors or just PWM controlled Jaguars might be a better solution.

I have spent the last couple of months trying to answer this question. I've gone back and forth several times at each success or failure along the way. After the first three tests I listed above I had made the "final" decision that it was going to be Victors w/ PWM this year. Test 4 gave me hope that a reliable CAN Jaguar solution was possible. At this point I'm all in w/ CAN Jaguar. Our robot this year will have 8/10 motors being controlled with CAN Jaguars...fingers crossed!

[Mike Copioli]

Originally Posted by jwakeman

Tests I ran:
1. cRIO w/ 2CAN on metal frame with mess of wires (test rig)- glitched
2. CRIO w/ serial cable on test rig - glitched
3. Laptop w/ BDC-Comm to jags/motors on test rig - NO glitches
4. cRIO w/ 2CAN on cafeteria table - NO glitches


I did suspect the cRIO software/firmware and also the 2CAN first. Testing with the serial cable eliminated the 2CAN (still glitched). Testing with the cRIO on the table top and seeing no glitches made me suspect noise in the test rig we had been using.

When using the 2CAN are you observing CAN errors in the web dash? Also how many jags are you running? Could you please describe what you mean by "glitches" all jags or only one or two? Have you tested using only one or two jags on the bus? If so what was the result.

Your problem sounds more like poor connectivity/termination. Try reducing the number of Jags on the bus, then adding additional jags until the problem reproduces.

[jwakeman]

Originally Posted by Mike Copioli

When using the 2CAN are you observing CAN errors in the web dash?

It did not show any errors in the web dash. But if I connected to one of the Jaguars with BDC-Comm after the glitch occurred I could see "Comm errors" in the extended status thingy of the BDC-Comm utility. I did notice that the packet counts in the web dash that are normally counting up very quickly would slow down or stop during the glitch.

Originally Posted by Mike Copioli

Also how many jags are you running?

Started with four. During my troubleshooting I went to using two. So during the 4 tests I listed I was using two Jaguars.

Originally Posted by Mike Copioli

Could you please describe what you mean by "glitches" all jags or only one or two? Have you tested using only one or two jags on the bus? If so what was the result.

I think all the jags would glitch. At least two of the four when i was running 4 and both when I was running two.

Glitch: With the 4 jag setup I was running a PID loop that was setting a current reference on the jags every 20 ms. When the glitch occured they would momentarily stop providing power to the motors. I could see the LED's on the jag also momentarily change color during the glitch. So if two were steady red they would go to amber I think (hard to tell since it was such a short time) momentarily. Also the definition of a glitch includes the behavior with the packet count in web dash and the Comm faults showing up in BDC-Comm as desribed above.

When I went to troubleshooting mode (the 4 tests with 2 Jags on the bus) I would set the current reference once in my code and then while(1){Wait()}. When the glitch occurred the motors would stop completely. So in the setup with the PID loop the glitch was only momentary because the PID loop would immediately set the current reference again. But in the troubleshooting tests I only set the current reference once so after the glitch occurred the motors stopped until the test was restarted.

I also ran some tests where I was calling GetFaults() and GetPowerCycled() periodically. They never showed any faults or power cycles when the glitches occurred. I did not call these functions during my 4 tests because I didn't want to put any extra traffic on the network and they had shown nothing in prior tests anyway.

Originally Posted by Mike Copioli

Your problem sounds more like poor connectivity/termination. Try reducing the number of Jags on the bus, then adding additional jags until the problem reproduces.

We are building our 'production' version of the robot now. We are going to do our best to be diligent in wiring/terminating/etc. I'm hoping I won't be able to reproduce the problem on this robot!!