Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[jhersh]

Originally Posted by WizenedEE

Source Please? I'm looking here:

<R59> If CAN-bus communications are used, the CAN-bus must be connected to the cRIO-FRC through either the Ethernet network connected to Port 1, Port 2, or the DB-9 RS-232 port connection.

And

<R50> Connections to the cRIO-FRC Ethernet ports must be compliant with the following parameters:
A. The DAP-1522 radio is connected to the cRIO-FRC Ethernet port 1 (either directly or via a CAT5 Ethernet pigtail).
B. Ethernet-connected COTS devices or custom circuits may connect to either cRIO-FRC Ethernet port; however, these devices may not transmit or receive UDP packets using ports 1100-1200 except for ports 1130 and 1140.

That seems to be two conflicting rules, but the 2CAN would probably count as a "pigtail"

I don't see these as conflicting. A pigtail is simply a wire, not an active component like the 2CAN. The 2CAN does not count as a pigtail. It still doesn't matter, however, since you can plug the 2CAN into one of the other ports on the DAP-1522 switch, not in between the radio and the cRIO.

Originally Posted by WizenedEE

Hooray for rules quotations. I am confused about in rule <R59> they say the radio may be connected directly OR with a pigtail. Them female to female connections, eh?

(Yeah, yeah I know, a pigtail is a male-female...)

The idea of the pigtail for this application is obsolete and probably should be removed from the rules. Instead of ever unplugging your cRIO from your radio, you should instead have a pigtail from one of the other ports on the DAP-1522 that you use to tether to your DS in the pits. This means you will never forget to plug the radio back in and miss a match (or delay it if your FTA is especially generous).

-Joe

[Mark McLeod]

Originally Posted by jhersh

The idea of the pigtail for this application is obsolete and probably should be removed from the rules. Instead of ever unplugging your cRIO from your radio, you should instead have a pigtail from one of the other ports on the DAP-1522 that you use to tether to your DS in the pits. This means you will never forget to plug the radio back in and miss a match (or delay it if your FTA is especially generous).

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

[jhersh]

Originally Posted by Mark McLeod

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

I guess that's true... however if a team wanted to avoid that, they could simply have a spare power connection for the practice radio and connect it to the tether pigtail.

[John Heden]

I would like to thank everybody for their thoughts on our CAN issue and all the suggestions you have offered. Our team crafted a custom dashboard that saves all of our data to disk including any UDP NetConsole data that it happens to catch and I’ve started to work way through this information looking for possible clues. Our freeze in our first quarter final match did yield some interesting results that I’m still trying to fully understand but certainly looks like a complete loss of CAN integrity through some mechanism. The following is part of the recorded error sequence (it goes on quite a bit longer) that shows a total of 16 InitCANJaguar() calls. The only call I can find to InitCANJaguar, however, is in the constructor for CANJaguar() so I a bit perplexed at this point given we have only 5 CANJaguars. After this initialization like sequence there is a litany of getTransaction() errors before we eventually do a reset and regain control.

At this point I’m partial to the startup race condition theory of some type. I would also add that we do launch a status monitoring thread that does read information from CANJaguar at the end of our robot constructor well before the autonomous loop is initiated. This seems to work but I wonder if the 2CAN occasionally needs a bit more time to settle down before it is called upon for status and CAN transactions…

Thanks again,

John

Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47Etc. etc. etc...

[jhersh]

Originally Posted by John Heden

At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

[techhelpbb]

Originally Posted by jhersh

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

[jhersh]

Originally Posted by techhelpbb

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

No... this is a start-up issue only. Can you describe as much about your setup and the behavior you see?

Thanks,
-Joe

[techhelpbb]

Originally Posted by jhersh

No... this is a start-up issue only. Can you describe as much about your setup and the behavior you see?

Thanks,
-Joe

We usually don't have connectivity issues at startup. We'll be driving and all of a sudden we'll loose one or more Jaguars off the bus with timeout or communications errors (haven't looked at the debug information myself, relying on the student feedback for the specific details).

The odd part is that in many cases these Jaguars are performing similar tasks to other Jaguars in the system, so it's not something specific to their actuators.

The Jaguar will drop off the bus, it'll come back and won't respond to further adjustments until we soft boot it. Hence we detect when they fail like this and automatically force a soft boot at this point.

Our most specific issue with this has been on the drive system. We have 2 Jaguars per side connected to CIM motors in the CIMiple gear boxes. We split the encoders and they are 100% isolated from each other and we want to run PID to target a velocity setpoint (it works fine when we don't get timeouts), but even with potentiometers we've seen this (but those Jaguars are high ratio shallow pitch worm drives). It seems to happen less when we use CAN for Vbus and hence loose the external reference, but it still does happen.

Given that we'll be fine for protracted periods of time, then suddenly experience a timeout under hard driving conditions this is what leads me to believe we have some sort of noise issue at work. Obviously if there was a spike that reached logic level on the CAN bus it would cause issues as the CAN bus is basically unmodulated single ended open collector digital. However, when using PWM, the worst you'd get is a shorter than expected pulsewidth at a frequency that is possibly wrong unless you have a periodic source of interference (unlikely in this case at the normal center frequency this system uses). So basically PWM would be more noise immune mostly because the Jaguars aren't fast enough or powerful enough electrically to instantly decompose it and overcome the load's inertia in response.

For the most part, the biggest problem we've had at startup with the cRIO using JAVA has been the bridge Jaguar just outright failing or the bus being improperly terminated. We had one Jaguar that just literally up and died once we turned it off. It was raining that day, we thought maybe water got into it somehow, but I inspected it and it was dry.

[Zme]

i can provide a little more information on something that seems to have similar behavior to this problem.
We are using c++ and have had similar issues with jaguars suddenly no longer responding to commands when using a closed loop control mode.
We noticed that it generally happened when there was a fault on the jaguar (current, voltage etc). when this happened the jag stopped working for whatever reason. it was thought that perhaps the fault caused the heartbeat to time out for some reason and therefore the jag would no longer respond to commands.
the fix we tried was running back through the initialization of the jaguar, (setting pid's and enabling control) whenever we detected a fault, this seemed to alleviate the problem but didn't catch everything. we then put a button on the joystick that would run through the re-init and while it didn't solve the problem it made things bearable.

[jhersh]

Originally Posted by Zme

i can provide a little more information on something that seems to have similar behavior to this problem.
We are using c++ and have had similar issues with jaguars suddenly no longer responding to commands when using a closed loop control mode.
We noticed that it generally happened when there was a fault on the jaguar (current, voltage etc). when this happened the jag stopped working for whatever reason. it was thought that perhaps the fault caused the heartbeat to time out for some reason and therefore the jag would no longer respond to commands.
the fix we tried was running back through the initialization of the jaguar, (setting pid's and enabling control) whenever we detected a fault, this seemed to alleviate the problem but didn't catch everything. we then put a button on the joystick that would run through the re-init and while it didn't solve the problem it made things bearable.

This also sounds like your Jags are rebooting due to a brown out of the input voltage. If you brown out the Jag or trip a breaker, the closed-loop settings are lost. You will have to re-initialize them (as you are) to recover the closed-loop control.

-Joe

[Zme]

i can tell you that it is not loosing power as there were outputs added to the code that would trigger when the GetPowerCycled() function returned true. a very early version of our code actually used both faults and power cycles as a trigger to re-init but it was taken out later, why it was taken out i don't know.

[John Heden]

Originally Posted by John Heden
At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

[jhersh]

Originally Posted by John Heden

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

It has nothing to do with the FMS. We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.

Are you aware of any teams having trouble at any regionals using v28 this weekend?

-Joe

[jhersh]

Originally Posted by techhelpbb

We usually don't have connectivity issues at startup. We'll be driving and all of a sudden we'll loose one or more Jaguars off the bus with timeout or communications errors (haven't looked at the debug information myself, relying on the student feedback for the specific details).

The odd part is that in many cases these Jaguars are performing similar tasks to other Jaguars in the system, so it's not something specific to their actuators.

The Jaguar will drop off the bus, it'll come back and won't respond to further adjustments until we soft boot it. Hence we detect when they fail like this and automatically force a soft boot at this point.

Our most specific issue with this has been on the drive system. We have 2 Jaguars per side connected to CIM motors in the CIMiple gear boxes. We split the encoders and they are 100% isolated from each other and we want to run PID to target a velocity setpoint (it works fine when we don't get timeouts), but even with potentiometers we've seen this (but those Jaguars are high ratio shallow pitch worm drives). It seems to happen less when we use CAN for Vbus and hence loose the external reference, but it still does happen.

Given that we'll be fine for protracted periods of time, then suddenly experience a timeout under hard driving conditions this is what leads me to believe we have some sort of noise issue at work. Obviously if there was a spike that reached logic level on the CAN bus it would cause issues as the CAN bus is basically unmodulated single ended open collector digital. However, when using PWM, the worst you'd get is a shorter than expected pulsewidth at a frequency that is possibly wrong unless you have a periodic source of interference (unlikely in this case at the normal center frequency this system uses). So basically PWM would be more noise immune mostly because the Jaguars aren't fast enough or powerful enough electrically to instantly decompose it and overcome the load's inertia in response.

For the most part, the biggest problem we've had at startup with the cRIO using JAVA has been the bridge Jaguar just outright failing or the bus being improperly terminated. We had one Jaguar that just literally up and died once we turned it off. It was raining that day, we thought maybe water got into it somehow, but I inspected it and it was dry.

Are you certain you aren't tripping a breaker or browning out the Jaguars under high load (due to poor wiring to the power input terminals of the Jaguar)? What you described are all symptoms of the Jag rebooting.

-Joe

[techhelpbb]

Originally Posted by jhersh

Are you certain you aren't tripping a breaker or browning out the Jaguars under high load (due to poor wiring to the power input terminals of the Jaguar)? What you described are all symptoms of the Jag rebooting.

-Joe

I can be certain we didn't blow any fuses or trip any breaker.

It's possible, I suppose that the load is too great and we are somehow browning out, we did consider that and try fresh batteries.

We don't seem to be consistently dragging down the batteries which would loosely indicate we aren't being overly aggressive in a general sense.

I can certainly try putting a datalogger on the power supply side of the Jaguar and see what I can like that. Not sure I have a current probe I can use in that configuration on 12VDC and a shunt is not a good idea given the load impedance.

The wires are clearly of sufficient gauge...I didn't wire that part of the system but it's better than 12AWG (so it's not 12AWG), and the problem doesn't seem to follow any particular wire so it's not likely the crimps. We've crimped plenty of other wires in the past with these tools and this make and brand of lug and they are firmly screwed down.

I could try putting a large 25V or 50V capacitor near the Jaguar on the input side to provide additional low internal resistance power if somehow the impedance of the wires from the power distribution board was suspect. It may not be legal on the competition floor, but for troubleshooting I think the battery will handle that.