Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Mark McLeod]

Originally Posted by jhersh

The idea of the pigtail for this application is obsolete and probably should be removed from the rules. Instead of ever unplugging your cRIO from your radio, you should instead have a pigtail from one of the other ports on the DAP-1522 that you use to tether to your DS in the pits. This means you will never forget to plug the radio back in and miss a match (or delay it if your FTA is especially generous).

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

[jhersh]

Originally Posted by Mark McLeod

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

I guess that's true... however if a team wanted to avoid that, they could simply have a spare power connection for the practice radio and connect it to the tether pigtail.

[John Heden]

I would like to thank everybody for their thoughts on our CAN issue and all the suggestions you have offered. Our team crafted a custom dashboard that saves all of our data to disk including any UDP NetConsole data that it happens to catch and I’ve started to work way through this information looking for possible clues. Our freeze in our first quarter final match did yield some interesting results that I’m still trying to fully understand but certainly looks like a complete loss of CAN integrity through some mechanism. The following is part of the recorded error sequence (it goes on quite a bit longer) that shows a total of 16 InitCANJaguar() calls. The only call I can find to InitCANJaguar, however, is in the constructor for CANJaguar() so I a bit perplexed at this point given we have only 5 CANJaguars. After this initialization like sequence there is a litany of getTransaction() errors before we eventually do a reset and regain control.

At this point I’m partial to the startup race condition theory of some type. I would also add that we do launch a status monitoring thread that does read information from CANJaguar at the end of our robot constructor well before the autonomous loop is initiated. This seems to work but I wonder if the 2CAN occasionally needs a bit more time to settle down before it is called upon for status and CAN transactions…

Thanks again,

John

Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47Etc. etc. etc...

[jhersh]

Originally Posted by John Heden

At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

[techhelpbb]

Originally Posted by jhersh

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

[jhersh]

Originally Posted by techhelpbb

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

No... this is a start-up issue only. Can you describe as much about your setup and the behavior you see?

Thanks,
-Joe

[techhelpbb]

Originally Posted by jhersh

No... this is a start-up issue only. Can you describe as much about your setup and the behavior you see?

Thanks,
-Joe

We usually don't have connectivity issues at startup. We'll be driving and all of a sudden we'll loose one or more Jaguars off the bus with timeout or communications errors (haven't looked at the debug information myself, relying on the student feedback for the specific details).

The odd part is that in many cases these Jaguars are performing similar tasks to other Jaguars in the system, so it's not something specific to their actuators.

The Jaguar will drop off the bus, it'll come back and won't respond to further adjustments until we soft boot it. Hence we detect when they fail like this and automatically force a soft boot at this point.

Our most specific issue with this has been on the drive system. We have 2 Jaguars per side connected to CIM motors in the CIMiple gear boxes. We split the encoders and they are 100% isolated from each other and we want to run PID to target a velocity setpoint (it works fine when we don't get timeouts), but even with potentiometers we've seen this (but those Jaguars are high ratio shallow pitch worm drives). It seems to happen less when we use CAN for Vbus and hence loose the external reference, but it still does happen.

Given that we'll be fine for protracted periods of time, then suddenly experience a timeout under hard driving conditions this is what leads me to believe we have some sort of noise issue at work. Obviously if there was a spike that reached logic level on the CAN bus it would cause issues as the CAN bus is basically unmodulated single ended open collector digital. However, when using PWM, the worst you'd get is a shorter than expected pulsewidth at a frequency that is possibly wrong unless you have a periodic source of interference (unlikely in this case at the normal center frequency this system uses). So basically PWM would be more noise immune mostly because the Jaguars aren't fast enough or powerful enough electrically to instantly decompose it and overcome the load's inertia in response.

For the most part, the biggest problem we've had at startup with the cRIO using JAVA has been the bridge Jaguar just outright failing or the bus being improperly terminated. We had one Jaguar that just literally up and died once we turned it off. It was raining that day, we thought maybe water got into it somehow, but I inspected it and it was dry.

[Zme]

i can provide a little more information on something that seems to have similar behavior to this problem.
We are using c++ and have had similar issues with jaguars suddenly no longer responding to commands when using a closed loop control mode.
We noticed that it generally happened when there was a fault on the jaguar (current, voltage etc). when this happened the jag stopped working for whatever reason. it was thought that perhaps the fault caused the heartbeat to time out for some reason and therefore the jag would no longer respond to commands.
the fix we tried was running back through the initialization of the jaguar, (setting pid's and enabling control) whenever we detected a fault, this seemed to alleviate the problem but didn't catch everything. we then put a button on the joystick that would run through the re-init and while it didn't solve the problem it made things bearable.

[jhersh]

Originally Posted by Zme

i can provide a little more information on something that seems to have similar behavior to this problem.
We are using c++ and have had similar issues with jaguars suddenly no longer responding to commands when using a closed loop control mode.
We noticed that it generally happened when there was a fault on the jaguar (current, voltage etc). when this happened the jag stopped working for whatever reason. it was thought that perhaps the fault caused the heartbeat to time out for some reason and therefore the jag would no longer respond to commands.
the fix we tried was running back through the initialization of the jaguar, (setting pid's and enabling control) whenever we detected a fault, this seemed to alleviate the problem but didn't catch everything. we then put a button on the joystick that would run through the re-init and while it didn't solve the problem it made things bearable.

This also sounds like your Jags are rebooting due to a brown out of the input voltage. If you brown out the Jag or trip a breaker, the closed-loop settings are lost. You will have to re-initialize them (as you are) to recover the closed-loop control.

-Joe

[Zme]

i can tell you that it is not loosing power as there were outputs added to the code that would trigger when the GetPowerCycled() function returned true. a very early version of our code actually used both faults and power cycles as a trigger to re-init but it was taken out later, why it was taken out i don't know.

[John Heden]

Originally Posted by John Heden
At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

[jhersh]

Originally Posted by John Heden

Greetings,

I was curious as to whether your testing was succesful and whether we have a fix at this point ? Would it be a fix on the FMS side or a new CRIO image ?

Thanks,

John

It has nothing to do with the FMS. We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.

Are you aware of any teams having trouble at any regionals using v28 this weekend?

-Joe

[John Heden]

Originally Posted by jhersh

It has nothing to do with the FMS. We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.

Are you aware of any teams having trouble at any regionals using v28 this weekend?

-Joe

Thanks Joe!

No, I'm not aware of any other teams reporting CAN startup problems with V28 from this weekend. Perhaps the Team Update #17 suggestion to monitor for CAN errors was effective (or perhaps scared a few more teams into a PWM conversion). Maybe someone at this weekend’s competition might add their observations on this issue…

[Phalanx]

Originally Posted by jhersh

We are testing a new image that should fix it. We have had no failures yet. However, FIRST wants more testing before making it public.
-Joe

As week 4 events will begin this coming week, I'm wondering if sufficient testing has been done to allow for a release of this new image?

[jhersh]

Originally Posted by techhelpbb

We usually don't have connectivity issues at startup. We'll be driving and all of a sudden we'll loose one or more Jaguars off the bus with timeout or communications errors (haven't looked at the debug information myself, relying on the student feedback for the specific details).

The odd part is that in many cases these Jaguars are performing similar tasks to other Jaguars in the system, so it's not something specific to their actuators.

The Jaguar will drop off the bus, it'll come back and won't respond to further adjustments until we soft boot it. Hence we detect when they fail like this and automatically force a soft boot at this point.

Our most specific issue with this has been on the drive system. We have 2 Jaguars per side connected to CIM motors in the CIMiple gear boxes. We split the encoders and they are 100% isolated from each other and we want to run PID to target a velocity setpoint (it works fine when we don't get timeouts), but even with potentiometers we've seen this (but those Jaguars are high ratio shallow pitch worm drives). It seems to happen less when we use CAN for Vbus and hence loose the external reference, but it still does happen.

Given that we'll be fine for protracted periods of time, then suddenly experience a timeout under hard driving conditions this is what leads me to believe we have some sort of noise issue at work. Obviously if there was a spike that reached logic level on the CAN bus it would cause issues as the CAN bus is basically unmodulated single ended open collector digital. However, when using PWM, the worst you'd get is a shorter than expected pulsewidth at a frequency that is possibly wrong unless you have a periodic source of interference (unlikely in this case at the normal center frequency this system uses). So basically PWM would be more noise immune mostly because the Jaguars aren't fast enough or powerful enough electrically to instantly decompose it and overcome the load's inertia in response.

For the most part, the biggest problem we've had at startup with the cRIO using JAVA has been the bridge Jaguar just outright failing or the bus being improperly terminated. We had one Jaguar that just literally up and died once we turned it off. It was raining that day, we thought maybe water got into it somehow, but I inspected it and it was dry.

Are you certain you aren't tripping a breaker or browning out the Jaguars under high load (due to poor wiring to the power input terminals of the Jaguar)? What you described are all symptoms of the Jag rebooting.

-Joe