Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[drakesword]

346 had issues at DC as well. During a couple of matches the speed of the robot would unexpectedly drop or the robot would begin shuddering (not a high speed shudder that a PID issue would show but a slow 1 second on 1 second off type issue). Finally in one of our last matches the second half of the can bus did nothing.

What is the upper limit of temperature before the jags begin to shutdown?

We did some test with our robot and we are still coming up inconclusive.

First we ran the robot and got no errors.
Then we unplugged a jag while running and still had no errors.
Then we unplugged a terminator without errors.
Unplugged any jag-jag wire and the entire bus stopped (not just beyond the lost connection)
Finally we unplugged encoders and still had no error.

We were running v29.

During the build season we were using the older image and ran the chassis for over 30 minutes without problems (except the one time we blew a grey jag by going full forward to full reverse too fast)

[Dad1279]

I'm sorry to hear that there were still issues with CAN at DC.

After our two matches at NY with 'NO COMM' issues, we thoroughly tested and could not come up with any ideas but to switch to PWM for DC.

We had no issues running PWM at DC.

[Phalanx]

We ran all of the DC Regional with CAN Jaguars in voltage control mode.
We had no encoders, pots, or limit switches in use on the JAGs.
We programmed in Labview, with Update 29.

We used encoders and limit switches attached to the digital side car.

We ran all of DC Regional, 9 Qualifiers(We missed one due to mechanical repairs), 2 Quarter Finals, and 2 Semi Finals matches. We had ZERO CAN issues.

I think it's time to investigate all of the following to attempt to find a commonality for failure.

1) Correct/Proper wiring both CAN and Power.
2) Ground isolation of components.
3) CAN Interface used, Serial or 2CAN
4) Type of JAGS used, Black, Grey, Both
5) The programming language used
6) The control mode(s) used.
7) encoders, pots, limit switches are being used.

So for us, I can say with 99.99999% certainty that for our machine...
1) CAN Wiring is correct with proper termination 100ohm.
2) All components are ground isolated from the frame and electrical wiring has no shorts or ground faults.
3) We used 2CAN on PORT 2 of the CRIO.
4) We used all BLACK JAGS.
5) We Programmed with Labview.
6) Used Voltage Mode only.
7) No Encoders, POTS, Limit Switches attached to the JAGS.

[heydowns]

v29 solved the problems we were seeing with our CAN-connected Jaguars. We no longer get random failures of the entire bus to work on robot startup as described in my earlier post.
We ran with v28 until Thursday lunch this past week and saw the boot-time failure twice during that morning.
After Thursday lunch when we put v29 on, the CAN chain (7 Jags) worked without issue for the remainder of the competition.

Originally Posted by drakesword

346 had issues at DC as well. During a couple of matches the speed of the robot would unexpectedly drop or the robot would begin shuddering (not a high speed shudder that a PID issue would show but a slow 1 second on 1 second off type issue). Finally in one of our last matches the second half of the can bus did nothing.

Sorry to hear you had problems, same for 1123 We were at DC too - wish I'd have know I would have come taken a look at your setup and try to offer some suggestions.


The CAN driver (at least for C++ maybe the others I have not looked) blocks while waiting for CAN replies to commands that are sent. If you have a Jag that is disconnected, failing, powered off, or has a poor bus connection the driver will block for a "long" time waiting for that response, thus stalling your overall code execution. This might or might not account for the slow on/off issues you describe. Of course that doesn't explain the root cause.

We've actually modified our copy of the CAN driver to shut down a device that's failing communications (repeatedly) rather than continuing to bog down the entire system in order to mitigate this.

One thing I've seen helping teams with CAN issues is that termination resistors connected in the manner described in the Jaguar documentation (that is, crimping them directly into the RJ connector on the appropriate pins) is a potential source of intermittent bus errors, particularly during match play when vibration and collisions are occurring a lot. Instead of making the cable in this fashion I recommend crimping in a short length of wire and soldering the resistor to that.
Again, not necessarily the problem your team is encountering but something worth trying/checking along with double-checking all your other bus cabling.

Also, on a very general note, as I read back on this thread it is really apparent that a number of different issues are being referenced here (symptoms may seem similar but they are caused by different things). For those just coming to this thread please keep that in mind and if you post your experiences please be as specific as possible.

[taichichuan]

Hi Gang!

Team 116 used CAN at the DC regionals as well with V29 and saw the exact same type of stuttering behavior as many of the other teams:

1) Correct/Proper wiring both CAN and Power.

This is our 2nd year with CAN. I check the cables with my automated cable tester and all checked good. Termination using the 2CAN on one end and 100 Ohm terminator at the other.

2) Ground isolation of components.

Our Jags are on standoffs and are isolated from the robot chassis ground.

3) CAN Interface used, Serial or 2CAN

We were using the 2CAN with firmware 2.5 (3/13/11). This worked well at the Bayou regional except for one time with the CAN connect problem from V28.

4) Type of JAGS used, Black, Grey, Both

Mixture of both black and grey jags. All jags at Firmware V92.

5) The programming language used

C/C++

6) The control mode(s) used.

Voltage mode (we were trying to use position mode for autonomous but ran out of time)

7) encoders, pots, limit switches are being used.

No encoders, pots or limit switches were being used.


I spoke to Brad Miller at DC and described the problem and we were actually able to see a little of the problem in the pits with him standing there. He suggested changing the motor safety timeout, which we did. However, in the heat of eliminations, I don't recall if we saw the stuttering post change.
__________________

HTH,

Mike

[drakesword]

1) Correct/Proper wiring both CAN and Power.
Going to rewire most of the can-related components at VCU. We were using 6 conductor wire. Going to switch to two. 120 ohm terminator soldered to wire not crimped.

2) Ground isolation of components.
Again as above. possible ground loop through our can wires

3) CAN Interface used, Serial or 2CAN
Serial

4) Type of JAGS used, Black, Grey, Both
Black (5)

5) The programming language used
Java.

6) The control mode(s) used.
Speed and Position

7) encoders, pots, limit switches are being used.
Us-Digital(I think) encoders 200 pulses per revolution. Don't know what switches we use but we have 2 on our gripper.

To address potential blocking we will make individual threads to set output to each motor.

[Phalanx]

Originally Posted by drakesword

1) Correct/Proper wiring both CAN and Power.
Going to rewire most of the can-related components at VCU. We were using 6 conductor wire. Going to switch to two. 120 ohm terminator soldered to wire not crimped.

We used 6 conductor wire, but we used 100ohm resistor as a terminator instead of 120ohm. I have heard second and third hand reports that using 120ohm terminators create issues, so I would try that before rewiring the entire CAN bus.

In the "Getting Started Guide" on Page 29 for the Black Jags the section "Jaguar Communication Cables" states 100ohm. Also see Table 8-1. CAN Wiring Parameters on page 26. http://www.luminarymicro.com/jaguar

[heydowns]

Originally Posted by drakesword

To address potential blocking we will make individual threads to set output to each motor.

Just a warning - the bus transactions are serialized via a monitor lock in the CANJaguar driver (at least on the C++ version, as well as in the last public source code I can easily get to of the Java version). Thus threading the transactions won't entirely alleviate any potential for blocking with respect to one another (that is, if one Jag thread were to block waiting for a bus response all other concurrently executing Jag threads will also block). Of course, it can offer the advantage of allowing the remainder of your code to execute while the Jag thread(s) are tied up.

[drakesword]

Originally Posted by heydowns

Of course, it can offer the advantage of allowing the remainder of your code to execute while the Jag thread(s) are tied up.

Thats the effect I was after

@Phalanx

According to some other threads the ground wire between jags can cause ground loop interference eg. the ground of one jag may be higher then the ground of another jag

[techhelpbb]

Just to clear up my post above...

Even with our Jaguars using %VBUS with CAN and only the potentiometers connected to the Jaguars as sensors...

We still get time out errors in addition to the occasional lock up.

The more wear and tear we put on our batteries the worse the problem gets.

Using the PID loops in the Jaguars seems to make the problems worse for us (to the point of dysfunction in some cases).

Due to the competition schedule, and now our additional appearance in St. Louis, we'll probably not want to tinker too much with the practice or production robot until it can't impact on performance at competition.

As soon as we can, I'd like to revisit this issue...even if we have to slap together another frame.
Our team has a spare new cRIO and I have a new Jaguar and some CIM motors.

[kamocat]

I wonder if we could be seeing the effects of process starving on the Jaguar?

CAN and RS232 communications are low priority tasks in the processor.

[taichichuan]

Originally Posted by kamocat

I wonder if we could be seeing the effects of process starving on the Jaguar?

CAN and RS232 communications are low priority tasks in the processor.

That might explain the serial CAN problems, but we saw them with the 2CAN via Ethernet as well. I think serial CAN tops out at something like 2K pps but the 2CAN jumps that to more like 8K pps from what I understand. At least it's comforting to know that it's not just a 2CAN problem.

Mike

[JasonStern]

1) Correct/Proper wiring both CAN and Power.
We verified all wiring. In addition, the termination resistor was soldered onto short wires and not crimped directly to the plug.

2) Ground isolation of components.
All components were mounted to Plexiglas and the frame was verified as electrically isolated by the DC inspection team.

3) CAN Interface used, Serial or 2CAN
2CAN*

4) Type of JAGS used, Black, Grey, Both
Both

5) The programming language used
Java

6) The control mode(s) used.
Speed, %VBUS

7) encoders, pots, limit switches are being used.
US Digital E3 quadrature encoders with 1024 CPR and 1/2" bore for the black jaguars; nothing for the grey.

*We can cause the jaguars to freeze in both %VBUS and Speed mode using the BD-COMM utility and quickly changing values. This is true even if we remove all other jaguars from the bus.

-Jason

[nuttle]

Not to pile on, but in case this may help either another team or anyone working on these problems:


- If you try to use an indexed encoder with the closed-loop PID control feature of the Jags, I'm pretty sure this won't work (based on our experiences). The issue here is probably that the index mark causes the encoder count to be reset once per revolution, and the PID logic doesn't expect this behavior, since it is not a continuous count but a count that essentially wraps around at a certain point. You can get around this by disconnecting the index pin -- certainly worth a try if this applies to you. You'll have to rely on the encoder being at zero when things start up, or doing something else to reference the count.


- If you are using PID, stuttering could certainly be caused by not having the P/I/D coefficients set correctly.

- I second the recommendation not to crimp an RJ connector directly to the solid leads coming from a resistor; this is not as reliable as making a short pigtail using cable that is designed for use with these connectors and soldering the terminator resistor to this.
- If you use a 2CAN, the 2CAN webpage is helpful for seeing how things are behaving. In particular, it provides error counters that can help validate wiring and basic communications connectivity.

[John Heden]

Greetings All,

We started this thread after experiencing a number of catastrophic CAN startup problems at the GSR regional and while we were hopeful that the V29 update would provide some relief on this issue, we continued to see this problem occasionally at the Hartford regional this past weekend. While our Drive team was instructed to monitor the Driver Station diagnostic tab for streaming CAN errors, they became complacent on Friday after never encountering the issue on Thursday. A Driverstation side CRIO Reboot did NOT recover the situation after the match began and we sat idle during the entire match (our Alliance won without our participation).

We experienced this problem again on Saturday while setting up before opening ceremonies (we had the first match) and again the system did NOT recover with a warm Driver station reboot on the first attempt and required either a third reboot attempt or possibly an actual robot power cycle to recover. This early morning triple failure scared us a bit but we NEVER experienced this catastrophic CAN failure during our later matches. The drive team did occasionally see a few startup CAN errors that concerned (panicked) them but did not see the catastrophic scrolling CAN error behavior.

We saw this failure occur during GSR at frequency of about 1 out of 6 matches ONLY while we were actually on the playing field and never while tethered to the robot with approximately the same statistic at Hartford. There was an observation that was seen once on the practice field making us curious as to whether the use of the radio was some how a catalyzing factor. Our radio was physically touching the 2CAN so we decided to try to give them some space (inverse square law). We couldn't try this with the radio in the pits but we proceeded to do some repetitive tethered power on/off tests trying out different power up sequences (relative to laptop) to try to reproduce this. We must have done this 20 or 30 times and NEVER saw a single CAN transaction failure and certainly not the continuous scrolling catastrophic CAN failure signature. We thought this was a radio ONLY failure but we did eventually experience this once while we were tethered in the pits. A quick power cycle and the problem went away! I wish we had tried a soft reboot as an experiment but our robot was being queued and our goal at that point was recovery rather than experimentation.

I believe we have some type of CAN/2CAN/CRIO/WPILib startup race condition that occasionally prevents some type of low level initialization causing the complete loss of the CAN bus. The manifestation we see is as if we simply pulled the CAN cable out of the 2CAN preventing any successful transaction to any CAN device. I believe use of the radio somehow amplifies this window of opportunity for failure given our ratio of match failures to pit failures and given we power up much more often while tethered in the pits than during actual matches. We had little working radio based experience prior to arriving at GSR due to the late availability of the physical robot for software testing. This radio testing and its influence on CAN failures will be a priority when we get our robot back. My apologies that some of this data is so soft but we were unable to find any hard correlation or anything definitive other than an occasional complete startup failure that always recovers on a power cycle mostly ruling out cabling issues. This failure occurs BEFORE being enabled essentially ruling out any real voltage drop or current/noise problems. If we startup successfully, we do run successfully. In fact, we have performed number of tests where we pull the breakers out of the Jags and even the 2CAN. This causes CAN errors to be reported but the system nicely recovers within a couple of seconds after we plug the breakers back in. We use the default voltage mode so others who have a more complex initialization or control scheme may not recover so easily.

There was also some anecdotal data coming from others at Hartford (other teams and even some of the Harford technical field folks) that believe the serial CAN interface is more robust than the 2CAN and recommended we switch away from the 2CAN. While this startup problem is catastrophic, it feels like some type of simple initialization glitch that is solvable. The CAN & 2CAN approach is a nice technology with perhaps this single gremlin to be exorcised. We'll try to diagnose this further when our bot comes home but unless we can convince our team that this is behind us, we may be forced to return to the simpler ways of PWMs....

Cheers and thanks,

John

1) CAN Wiring is correct with proper termination.
2) All components are ground isolated from the frame and electrical wiring has no shorts or ground faults.
3) We run the CRIO connection directly to the radio and connect the radio directly to the 2CAN rather than passing all CRIO traffic through the 2CAN.
4) We used all Tan JAGS.
5) We Programmed with C++.
6) Used Voltage Mode only.
7) 3 Jags with optical encoders, 1 of these has a single limit switch as well
2 Jags each with 2 limit switches, no encoder

8) I should also add that our software launches a separate dashboard thread at the end of the constructor AFTER the Jags and other robot objects are created with this data (encoder values, currents, voltages, etc) being read for display and capture by our custom dashboard. This explains why we see a continuous never ending data stream while others, I suspect, may see a number of errors during startup that stop but resume once they are enabled and autonomous & control Jaguar transactions begin.