Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[jhersh]

Originally Posted by John Heden

At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

[techhelpbb]

Originally Posted by jhersh

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

[jhersh]

Originally Posted by techhelpbb

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.

No... this is a start-up issue only. Can you describe as much about your setup and the behavior you see?

Thanks,
-Joe

[kamocat]

Originally Posted by Mark McLeod

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

Are you saying that the D-link will broadcast a network, even in bridge mode?

Joe:
Was that issue with starving the Ack worked out? I haven't retested it.

[jhersh]

Originally Posted by kamocat

Joe:
Was that issue with starving the Ack worked out? I haven't retested it.

Remind me what issue that was? Are you referring to the starved token resynchronization? If so, then yes, the v28 image (and several before that) include the work-around that restarts the token synchronous to the sendMessage call where the token stream is detected to be expired.

-Joe

[Ken Streeter]

Originally Posted by Mark McLeod

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

There is one other time that it is important for teams to unplug the cRIO from the DLink -- if you are in the finals and are tethering the robot to prepare it for the next match of the finals (for example, performing a system check or compressing air to have full tanks for the next match).

We ran into this problem at the Week Zero Scrimmage, so we were ready for the problem during the GSR finals. During the finals, the same teams are on the field in consecutive matches, so the field access point is still configured to communicate with the teams that were just on the field. Accordingly, as soon as the Driver Station is connected to the DLink, the DS enters the "FMS Connected" mode, forcing the robot into a disabled state and prohibiting "tethered" control.

If you find yourself in the final matches and need to tether the robot in between matches to add air to the tanks or perform any system checks, you'll want to connect the DS directly to the cRIO without going through the DLink, in order to avoid the FMS control.

[Mark McLeod]

Originally Posted by Ken Streeter

There is one other time that it is important for teams to unplug the cRIO from the DLink -- if you are in the finals and are tethering the robot to prepare it for the next match of the finals (for example, performing a system check or compressing air to have full tanks for the next match).

That makes sense.
The field staff must leave the previous match teams up in FMS until the scores and penalties have been debated and submitted. I sometimes borrow one of the unoccupied player stations to test laptop link-ups in those moments (setting the laptop to one of the absent team #s).

[jhersh]

Thanks for the detailed write-up!

Originally Posted by John Heden

1) How do folks recommend powering the actual 2CAN device ? There are references to ensuring the Dlink radio is powered by the boosted 12V (white Wago connector) to help avoid excessive voltage drops and possible problems. The 2CAN itself has a wide voltage range (6.5 to 28V) but should it be connected on the raw 12V fused power bus? Our autonomous start is probably the worst case power consumption where we simultaneously power up 3 big CIM motors to near full power.

Powered from a 20A fused PD output.

Originally Posted by John Heden

2) Simultaneously accessing the 2CAN webpage while our robot code is simultaneously executing does result in a number of our robot CAN transactions failing (~ 1/1000). This is more of an annoyance for us than a problem but is suggestive of some type of system vulnerability when the 2CAN is being accessed from multiple sources. The WPI libraries seem to have access protection but the 2CAN itself may have some issue when it is being accessed by both the CRIO robot code and a browser on the2CAN page.

I'm not sure how this is handled in the 2CAN. Mike or Omar, can you please comment?

Originally Posted by John Heden

3) There is some mention that the FRC system does send additional CAN related messages to CAN based robots, can anybody confirm this possibility ? Could there be some type of negative interaction with the FMS system?

This is inaccurate. The FMS has no knowledge of the CAN bus. I'm not aware of anything at the application layer that would cause any interaction between CAN and FMS.

Originally Posted by John Heden

4) I did speak to a few CAN Jaguar users who spoke of possible overloading the CAN bus and needing to drop their access intervals down. The current WPI library interfaces, however, look like simple blocking calls where all access is nicely serialized and completes before the next sequence is initiated. While all of these calls may eventually slow down our control loops, it seems unlikely that we could overload the bus.

This is generally accurate. There was an issue in beta where the Jaguar token negotiation could be starved out by continuous traffic from the user application, but that was worked around.

Originally Posted by John Heden

5) We do see some type of “too much error data!” message once in a while we were playing but could not locate this error message anywhere in our code or in the WPI libraries. Any idea as to the source of this unique message?

That message comes from the error reporting to the driver station. If too many errors are generated in a short time, they won't all fit in the fixed sized packets to the driver station, and will result in that message.

Originally Posted by John Heden

6) There is a rev 66 1/29/2011 for FRC plug-in update for FRC posted on the Cross the Road website. We bagged our robot before I could confirm what version of 2CAN plug-in is incorporated in the 2/11/2011 V28 CRIO update? Is there a need for manually installing this file or is the latest included with the V28 update ?

Revision 66 of the 2CAN plug-in is included in v28.

Please let me know if you come up with any more details that might lead to the issue.

Thanks,
-Joe

[Mike Copioli]

Originally Posted by jhersh

I'm not sure how this is handled in the 2CAN. Mike or Omar, can you please comment?

I will give you a preliminary answer until Omar has a chance to provide a more detailed one. In short the problem is caused by a lack of sychronicity between the cRIO CAN transactions and the 2CAN dashboard transactions. This is a simple explanation of the problem, it is actually a bit more involved as Omar has explained it. Omar has written some management code that is intended to deal with this problem, however the web dashes ability to interact with the CAN bus is second chair to the user code. If the user code is sending can throttle requests to frequently, for example, the time the web dash has to interact with the bus is limited. This is not an issue with the Cross-link Control System because the 2CAN performs all synchronization and has more of a 'master' role. But again this is Omars area of expertise.

[John Heden]

It sounds like the experts are convergent on the problem and we remain hopeful for a robust CAN solution. For anybody following this CAN problem thread, there was an interesting statement in the March 8 (yesterday) Team Update #17:

If a team is using a CAN network on the robot, they should check the messages in the “Diagnostic” tab of the Driver Station before a match starts to ensure that there aren’t any scrolling CAN timeouts. If there are such messages, give the MC a “thumbs down” to show you’re not ready and click on “Reboot Robot” to restart the cRIO and clear the errors. Teams will only see such timeout errors if it's properly handled in code, and they should take care to ensure that these exceptions are handled such that they can be seen on the field.

I’m going to try to convince our team that we should maintain our CAN implementation (not go back to PWM cables) but carefully monitor for this possible problem.

Thanks again,

john

[Zme]

our team wrote some code so that the robot would reboot itself automatically when/if it did not have communication with the can bus on boot, not the prettiest workaround but it works for what we needed.

[ozrien]

Originally Posted by Mike Copioli

I will give you a preliminary answer until Omar has a chance to provide a more detailed one. In short the problem is caused by a lack of sychronicity between the cRIO CAN transactions and the 2CAN dashboard transactions. This is a simple explanation of the problem, it is actually a bit more involved as Omar has explained it. Omar has written some management code that is intended to deal with this problem, however the web dashes ability to interact with the CAN bus is second chair to the user code. If the user code is sending can throttle requests to frequently, for example, the time the web dash has to interact with the bus is limited. This is not an issue with the Cross-link Control System because the 2CAN performs all synchronization and has more of a 'master' role. But again this is Omars area of expertise.

The 2CAN keeps track of outstanding transactions from the cRIO and from the Web dash. The 2can knows which CAN ids represent requests that will have ACK responses and keeps track of when the cRIO or WD is waiting on an ack.
So if WD wants to request say bus voltage on Jag1 it first checks if cRIO has performed any requests (resync tokenization, throttle set, etc..). If cRIO has then WD will hold off on transactions to Jag1 until Jag1 sends an ACK (intended to go to cRIO).

cRIO Set Throttle
<--------------------------User opens Web Dash, WD holds off...
cRIO ACK
WB Get Voltage <--------------- Jag sends an ACK, it is now available
WB ACK

Similarly if the WD is waiting on Jag1 for a response/ACK and cRIO request comes in, it will delay transmission until Jag1 responds, OR a one millisecond timeout occurs (to ensure cRIO gets minimal latency to bus ).

Without this management code there would be problems where...
cRIO Sets Throttle
WB Gets Voltage <--------------------------User opens Web Dash,
ACK
ACK
...which confuses the Jag and sometimes causes Jag to not respond at all.