Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Mike Copioli]

Originally Posted by jhersh

I'm not sure how this is handled in the 2CAN. Mike or Omar, can you please comment?

I will give you a preliminary answer until Omar has a chance to provide a more detailed one. In short the problem is caused by a lack of sychronicity between the cRIO CAN transactions and the 2CAN dashboard transactions. This is a simple explanation of the problem, it is actually a bit more involved as Omar has explained it. Omar has written some management code that is intended to deal with this problem, however the web dashes ability to interact with the CAN bus is second chair to the user code. If the user code is sending can throttle requests to frequently, for example, the time the web dash has to interact with the bus is limited. This is not an issue with the Cross-link Control System because the 2CAN performs all synchronization and has more of a 'master' role. But again this is Omars area of expertise.

[John Heden]

It sounds like the experts are convergent on the problem and we remain hopeful for a robust CAN solution. For anybody following this CAN problem thread, there was an interesting statement in the March 8 (yesterday) Team Update #17:

If a team is using a CAN network on the robot, they should check the messages in the “Diagnostic” tab of the Driver Station before a match starts to ensure that there aren’t any scrolling CAN timeouts. If there are such messages, give the MC a “thumbs down” to show you’re not ready and click on “Reboot Robot” to restart the cRIO and clear the errors. Teams will only see such timeout errors if it's properly handled in code, and they should take care to ensure that these exceptions are handled such that they can be seen on the field.

I’m going to try to convince our team that we should maintain our CAN implementation (not go back to PWM cables) but carefully monitor for this possible problem.

Thanks again,

john

[Zme]

our team wrote some code so that the robot would reboot itself automatically when/if it did not have communication with the can bus on boot, not the prettiest workaround but it works for what we needed.

[ozrien]

Originally Posted by Mike Copioli

I will give you a preliminary answer until Omar has a chance to provide a more detailed one. In short the problem is caused by a lack of sychronicity between the cRIO CAN transactions and the 2CAN dashboard transactions. This is a simple explanation of the problem, it is actually a bit more involved as Omar has explained it. Omar has written some management code that is intended to deal with this problem, however the web dashes ability to interact with the CAN bus is second chair to the user code. If the user code is sending can throttle requests to frequently, for example, the time the web dash has to interact with the bus is limited. This is not an issue with the Cross-link Control System because the 2CAN performs all synchronization and has more of a 'master' role. But again this is Omars area of expertise.

The 2CAN keeps track of outstanding transactions from the cRIO and from the Web dash. The 2can knows which CAN ids represent requests that will have ACK responses and keeps track of when the cRIO or WD is waiting on an ack.
So if WD wants to request say bus voltage on Jag1 it first checks if cRIO has performed any requests (resync tokenization, throttle set, etc..). If cRIO has then WD will hold off on transactions to Jag1 until Jag1 sends an ACK (intended to go to cRIO).

cRIO Set Throttle
<--------------------------User opens Web Dash, WD holds off...
cRIO ACK
WB Get Voltage <--------------- Jag sends an ACK, it is now available
WB ACK

Similarly if the WD is waiting on Jag1 for a response/ACK and cRIO request comes in, it will delay transmission until Jag1 responds, OR a one millisecond timeout occurs (to ensure cRIO gets minimal latency to bus ).

Without this management code there would be problems where...
cRIO Sets Throttle
WB Gets Voltage <--------------------------User opens Web Dash,
ACK
ACK
...which confuses the Jag and sometimes causes Jag to not respond at all.