CAN troubles with Black Jaguars

[Levansic]

Our team had always used the Jaguar controllers in PWM mode, but this year we wanted to jump overto CANbus control. We thought we had done enough prep work, downloading firmware and getting BDC-COMM onto an old laptop with a serial port.

Our black jaguars flashed to the new firmware just fine, and all were given unique ID’s. When we tried to update our beige jaguars, we hit a major snag. None of them were visible when hooked up to a single bridging black jaguar. We hooked up two black jaguars, and could not see any beyond the first, directly connected to the serial port.

We did web searches and scoured this forum. The most thrown-out solution to the same problem, as encountered by others were faulty terminators and faulty cables. After much wasted time trying to find non-existant faults in these components, we finally found the source of our problems.

In our case, all six of our black jaguars had faulty RJ-11 jacks (CAN - only), and two also had faulty RJ-12 jacks (Serial + CAN). Oddly, not a single one of our seven beige jaguars had a single faulty port. The problem is a minor manufacturing defect, where the sprung contactor wires in the jack are just not sprung enough. This causes contact to be lost with the slightest jostle of the cable.

We discovered this by noting that some connections could enable CAN communication, but lost communication If the cable was slightly disturbed. We had a lot of movement in the jacks. When we looked into the port, we found one of the center pair of contacts almost flat against the outer wall of the jack housing.

I carefully stretched these wires out with some needle pointed tweezers, and tested every port until I could maintain CAN communications, no matter how much I wiggled the cables. Now, all of our jaguars are maintaining communication over CAN

[jwakeman]

I think we may have experienced this problem as well. I bought off the shelf cables because I suspected my home made cables. We still had the problem with the new cables until I reached down and wiggled one and all of a sudden all were online! The problem was intermittent after that (some days I had all online, some days only the first in the chain). One of the benefits of CAN (opposed to PWM) is supposed to be the locking connectors. What can we do to get the manufacturer to take a serious look at this?

[dyanoshak]

@jwakeman

What can we do to get the manufacturer to take a serious look at this?

Thanks to both of you for pointing out this issue. We’ll investigate to see what happened and if this is a wide spread issue or isolated to a batch of bad sockets.

-David

[dyanoshak]

If either of you happen to still have a Jaguar with the bent pins, could you please post a picture of them? This will help us determine the root cause.

[Levansic]

Sorry. I fixed all of our working black jaguars before thinking about taking any pictures. We had one that was fried last year that I didn’t fix, but I think it was pitched over the weekend.

We only had two new 2012 KOP jaguars. Both had bad RJ-11 ports, and one had a bad RJ-12 port. I mean that port worked for the serial connection, but was faulty for CAN. These two units were untouched by student hands before this Saturday, so nothing got improperly jammed into the ports.

We were not aware of the connectivity issues of our jaguars before now, because we hadn’t attempted using CAN. Our issues were specific to the black jaguars, from all years, and not the older beige ones.

[Levansic]

OK, I got a picture of our remaining black jaguar, which I did not repair. This unfortunate unit was damaged in a finals match, in an earlier year. The RJ-11 port on this unit was not as bad as the other black jaguars that I had to repair.

If you look carefully at the two ports, you can see that the wires in the RJ-12 (6p6c) port on the left are pretty uniform, and are sprung down about half-way through their wire guides on the bottom. This is a “good” port.

Now, compare that to the wires in the RJ-11 (6p4c) port on the right. Those wires are not uniform, and the last three, especially the fourth one, are close to the back wall of the port. The first wire is sprung almost the same amount as the wires on the RJ-12 port.

On most of our black jaguars, the distance of these wires from the back wall alternated. For example wire 2 and 4 would be almost flat, while 1 and 3 would be sprung out more. I used a pair of needle tweezers to pull these wires out. They often would jump tracks to a neighboring wire guide. I would then move them back to the correct track, and test the port by inserting a plug.

This weekend, we reassembled the base of our robot from last year, but wired it with CAN. After tweaking the ports on our black jags, I was unable to get any comm errors, no matter how much we wiggled each and every cable.

We are so ready to swear off PWM cables forever!
– Len

[mjcoss]

I never even thought of looking at the connector. smacks forehead We used CAN bus last year, and had many issues with CAN bus timeouts. It was very intermittent, and as you say everyone suggests that it’s either termination or cables. I’ve replace these over and over with no noticeable improvement. I’ll take a look at the connectors tonight. We are using the 2CAN, and for a while I believed that it was some sort of electrical issue with brown out, and dropped messages. A faulty connector could be the root cause that I’ve been looking for. I’ll let you know what I find.

—Michael J Coss FIRST Team 303 - Mentor

[jwakeman]

I checked two jags we recieved this year and two from last year. One from last year had what we deemed a bad ‘net’ port as described in the original post. The others looked fine. I tried taking pictures but they didn’t turn out all that great.

[techhelpbb]

This was something we looked at when we first encountered issues. We had made cables ourselves and there was ample room for this to happen.

I can confirm that none of the black Jaguars we removed exhibit this behavior.

Also, I did literally remove one of these connectors and put a cable on there at one point (just in case…so much for the warranty).

I believe I currently have access to 9 of these black Jaguars and a few of the older model.

If there’s interest I’m willing to photograph these connectors on our units.

Mind you we could make our units talk on CAN, we could even pull on and otherwise malign the cables in the ports, but when the robot was on the floor driving around we’d start to accumulate errors.

We did develop a bad cable in our backup robot at one point (March 2011 I think), and when we did we had to replace it. The system would not cooperate with that. We did get some intermittent functionality but nothing even close to expectations.

[mjcoss]

Checked last night and no such luck. All look Jaguar connectors look fine to me. We also tested that the end to end termination was reading the correct resistance. So back to the same issue we had last year with CAN: Intermittent timeout of messages, with no clear cut root cause or solution.

One experiment that I will be running tonight is the use of sync groups. My code makes extensive use of sync groups for updating any motors that are linked together, I doubt that it’s involved with my problem but I’ll grasp at any straw at this point.

Unfortunately, looking at the forums CAN issues seem to be just as rampant this year You’d think we’d learn our lesson.

[jhersh]

@mjcoss

Checked last night and no such luck. All look Jaguar connectors look fine to me. We also tested that the end to end termination was reading the correct resistance. So back to the same issue we had last year with CAN: Intermittent timeout of messages, with no clear cut root cause or solution.

One experiment that I will be running tonight is the use of sync groups. My code makes extensive use of sync groups for updating any motors that are linked together, I doubt that it’s involved with my problem but I’ll grasp at any straw at this point.

Unfortunately, looking at the forums CAN issues seem to be just as rampant this year You’d think we’d learn our lesson.

Sorry to hear you’re still having trouble. Do you have LabVIEW installed? Have you read Tutorial 9 on the getting started window in LabVIEW? It describes best practices and may give you some ideas to consider.

What language are you programming in? The C++ implementation saw some improvements this year, to come up to the level that LabVIEW was last year. LabVIEW then surpassed it again (wrt supporting the new Jag firmware features). C++ may support the new features a little later this season. The Jag firmware was released too late to get all the APIs updated to support them in time for kick-off release.

-Joe

[techhelpbb]

@mjcoss

Checked last night and no such luck. All look Jaguar connectors look fine to me. We also tested that the end to end termination was reading the correct resistance. So back to the same issue we had last year with CAN: Intermittent timeout of messages, with no clear cut root cause or solution.

One experiment that I will be running tonight is the use of sync groups. My code makes extensive use of sync groups for updating any motors that are linked together, I doubt that it’s involved with my problem but I’ll grasp at any straw at this point.

Unfortunately, looking at the forums CAN issues seem to be just as rampant this year You’d think we’d learn our lesson.

Seeing as you’re in Bridgewater, NJ (which I often drive thru)…and I’m in Mount Olive, NJ…if you’d like to work together on this at some point please feel free to contact me.

That being said, the timeout errors are a little different in nature between the 2CAN and RS232 bridge.

Can you share which you are using?

[otherguy]

@mjcoss

Checked last night and no such luck. All look Jaguar connectors look fine to me. We also tested that the end to end termination was reading the correct resistance. So back to the same issue we had last year with CAN: Intermittent timeout of messages, with no clear cut root cause or solution.

The problem is one of two places, the wiring or the code.

You can quickly rule out the wiring by connecting your first jag to a computer and communicating with the bus using BDC-COMM (this pulls your software out of the loop). If that works then it is your code.

I would suggest getting ONE jaguar working over CAN. Verify that works, then extend your known working code to include communicating with more Jaguars.
One problem we ran into early on was sending too many messages onto the CAN bus at once. We were driving three groups of jaguars (master/slave configuration) using synch groups. We were not limiting the frequency at which the messages were being communicated, and things didn’t run reliably. I don’t know what language you’re using, but you’ll want to add a time delay or wait between transmissions to ensure you don’t flood the bus.
This is an issue we saw over the RS232/CAN bridge. I’d expect this problem would be less pronounced using the 2CAN.

[techhelpbb]

@otherguy

The problem is one of two places, the wiring or the code.

I completely agree that between these 2 issues there are a massive number of possible sources of issues.

However, I should like to point out that in the past errors have been found in the Jaguars and I have a very intricate test environment for them and I’ve seen a few things that despite entirely faithful efforts might make it difficult to pin down the exact cause.

I have not seen his particular environment however.

Also, it’s entirely possible that you can instigate issues if you overload the Jaguars (and the circuit protection behind them) without noticing it. The Victors in this regard seem a bit less sensitive from incidental examples I have seen. In a way the CAN bus can become a single point of failure that with PWM on the Victors wouldn’t otherwise exist.

[otherguy]

So here’s an interesting development.

We had one jag that came in the KOP that we couldn’t communicate with over BDC-COMM. We had 5 others flashed and working, so we didn’t look into the root cause until today.

When power is supplied the LED flashes amber, but no comms could be established… I looked at the connectors to see if we had the bent pin issue described above and to my surprise I find this!

Hopefully this is clear enough

RS232 rx/tx operates over pin 1 and 6 on the left port on the Jags. It’s a little difficult to communicate if those pins aren’t even present in the connector!

TI seriously needs to focus on their quality control. I wonder how many other teams are going to run into the same issue and don’t know enough to think to look for 6 conductors on a connector. This isn’t a fun one to troubleshoot if you’re still questioning if your cables/serial adapter haven’t been verified working yet.

[otherguy]

FYI, this jag can still be used, just not as the first one in the chain.

Also kinda weird… it came with a board ID of 3. They are supposed to come with a board ID of 1.

[ParthLetap]

When i had the issue of not getting the chain to work, i tried switching the CAN cable ports, as in the wire coming off the previous jaguar went into the port on left, i switched it to right, etc. That seemed to fix my problem. My question is, as long as it works does it matter where the cables are going?

[Levansic]

CAN cables don’t care which port they connect to, as long as the chain is terminated. The CAN bus is a party line, where all modules are listening at all times.

There is no directionality to the bus.

If a cable and port appear problematic, then you should treat that connection as suspect. Check the cables and ports thoroughly, as something is wrong in that link. If you move the cables to other ports, and the problem goes away, you may have just delayed the problem, and hid it somewhere else.

[Mr_Lim]

We have more probably 20 Jaguars in our shop, and used CAN extensively this year and last.

We recently discovered that two of our Jaguars had bad CAN ports (RJ-12 plugs), with either bent or missing pins.

I don’t know the cause, because I don’t think we “abused” them. But we do crimp our own CAN cables. It’s possible a person may not have crimped the pins in the connector all the way down.

Inserting a connector with pins not completely crimped down is likely a quick way to wreck a CAN port.

[mjcoss]

Sorry that I haven’t replied sooner just been busy. So we are using the 2CAN and we’re programming in C++. So for us, the points on the chain that can be an issue are the 2CAN, the Jaguars, and the port on the cRIO, and all the software in between. Most of which is not mine.

I’ve two platform that I’m testing on. The first is a drive base, and the second is a prototype shooter. The shooter is running the Jaguars in speed control mode, the drive base is running in the default mode, %V I think. Both are seeing timeout, although the shooter is having more issues than the drive base. Both are using CIM motors. Although I suspect that the load on the drive base is higher than on the shooter.

The code is pretty clean in that I’m not doing a whole lot. Tank drive on the base, and taking analog input on the driver station to drive the shooter. Updates to the motors are in TeleopPeriodic, and there just aren’t that many knobs that need to be tweaked to make that work.

What we’ve seen is that on the shooter, sometimes during the boot up, the 2CAN get’s very unhappy. Flashing red/yellow. Sometimes it boots just fine. If we make it past the boot problems, it tends to stay clean. Then while shooting, it sometimes completely drops out CAN bus communications is completely gone, and the console log if full of timeout messages. The only recovery is reboot.

The failure mode on the drive base is periodic timeout messages. Usually just a few off and on. I have seen it have the cascade failure but only once.

We do have the version 2 of the 2CAN, and the latest firmware on it, and the Jaguars.

I’ve already connected to each of the Jaguars using BD Comm, and flashed, set ids and so I’m kinda at a loss to understand what is going wrong.

The sync groups change didn’t seem to have any affect. I guess I can try the LabVIEW code to see if it behaves differently

@techhelpbb - more than willing to work together on this. I’d like to get a better understanding of the failure modes and figure out some way to solve the problems.