Unexplained intermittent CAN / 2CAN Jaguar problems at GSR

[Phalanx]

Hi John,

We also used CAN, but with all BLACK JAGS and 2CAN. We didn't have any issues at all with it.

I can answer a few of your questions base on how we did things.

1) We power 2CAN from the raw 12V on the PD. We didn't experience any brown outs even when applying full power to 4 cims and 4 banebots simultaneously. I will say that low battery voltage, bad terminators, faulty wiring are the biggest contributors to CAN network problems.

2) We haven't experienced that issue, however, the 2CAN web page is only supposed to be accessible when attached to CRIO Ethernet Port 1, which is great for diagnostics, but is not allow in competition. 2CAN must be on port 2 in competition mode, it's a clearly defined rule. The only question that comes to mind is the 2CAN connect to the Wireless Bridge/AP. If it is, that could explain some the packet loss.

3)There is a "TRUSTED" mode that CAN uses for FIRST and it requires FIRST specific firmware on the JAGS. Level 92 I believe is the latest. It shouldn't have any negative impact on the FMS. We certainly didn't experience any through 9 qualifying rounds plus QF's, SF's and Finals.

4 & 5) I haven't seen or experienced either of those.

6) I took no chances and FTP'd the latest 2CAN driver to the CRIO after I installed V28.

[WizenedEE]

Originally Posted by Phalanx

2CAN must be on port 2 in competition mode, it's a clearly defined rule.

Source Please? I'm looking here:

<R59> If CAN-bus communications are used, the CAN-bus must be connected to the cRIO-FRC through either the Ethernet network connected to Port 1, Port 2, or the DB-9 RS-232 port connection.

And

<R50> Connections to the cRIO-FRC Ethernet ports must be compliant with the following parameters:
A. The DAP-1522 radio is connected to the cRIO-FRC Ethernet port 1 (either directly or via a CAT5 Ethernet pigtail).
B. Ethernet-connected COTS devices or custom circuits may connect to either cRIO-FRC Ethernet port; however, these devices may not transmit or receive UDP packets using ports 1100-1200 except for ports 1130 and 1140.

That seems to be two conflicting rules, but the 2CAN would probably count as a "pigtail"

Also, the 2CAN must be connected to the PD board by a dedicated 20 A breaker, ie not on the end part.

<R39> F. Custom circuits and sensors powered via the cRIO-FRC or the Digital Sidecar are protected by the breaker on the circuit(s) supplying those devices. Power feeds to all other custom circuits must be protected with a dedicated 20-amp circuit breaker on the PD Board.

Hooray for rules quotations. I am confused about in rule <R59> they say the radio may be connected directly OR with a pigtail. Them female to female connections, eh?

(Yeah, yeah I know, a pigtail is a male-female...)

[Andy A.]

As another data point-

95 also had CAN bus issues at GSR, as did several other teams. We used all black jags and the serial port. It was enough of an issue that FIRST personal polled CAN teams and advised them that they couldn't say what was going on, but that it seemed like CAN wasn't always working. We elected to switch to PWM control (which also brought a half dead sidecar to our attention).

The last I had heard was that a common factor in all this was that the team was using Java as their programming language. Beyond that, there didn't seem to be any commonality. I heard some accusations of it only occurring on one driver station, or side of the field, but I can't confirm if that was true or not.

[heydowns]

1511 also saw similar failures during our time at FLR this past weekend. Fortunately for us this never manifested out on the playing field, only when testing in our pit.

I was able to attach the Windriver debugger a few times when we caught the error and have some technical details to compile and share with the developers (on my list of stuff to do hopefully today).

It appears to be almost certainly a race condition during startup whereby when the failure is tripped a task initiated for CAN handling is terminating due to following a bad pointer.

For completeness, our setup is serial/black jag based, all black jags in chain. C++ with latest (2/16) update, cRIO with image v28.

[Phalanx]

Originally Posted by WizenedEE

Source Please?

http://www.usfirst.org/uploadedFiles...%20%281%29.pdf

I also remember reading it somewhere else, like FIRST Forums, or a Team Update, but I can't remember exactly where. If/when I find/remember I'll update this.

Originally Posted by WizenedEE

<R39> F. Custom circuits and sensors powered via the cRIO-FRC or the Digital Sidecar are protected by the breaker on the circuit(s) supplying those devices. Power feeds to all other custom circuits must be protected with a dedicated 20-amp circuit breaker on the PD Board.

I guess I'll need to move the power from direct to the PD on a 20amp.

Originally Posted by Andy A.

We used all black jags and the serial port.
The last I had heard was that a common factor in all this was that the team was using Java as their programming language. Beyond that, there didn't seem to be any commonality. I heard some accusations of it only occurring on one driver station, or side of the field, but I can't confirm if that was true or not.

We are using Labview

We intentionally chose to not use the serial interface for the following reasons:
128Kbps transfer rate over serial versus 1MB transfer rate using 2CAN. (CAN is typically a 1MB network)
I also noticed that under heavy processing load the serial interface on the CRIO would lag, or appear to garble some data, etc.. So we opted to avoid that potential pitfall.

[boomergeek]

Add Team 241 to the list of teams at GSR with CAN/2CAN problems.

But we were using C++. We did not ask around early enough and did not get the word that it was a general problem. We did switch our robot over to PWM before it was crated for Boston.

http://www.chiefdelphi.com/forums/sh...13&postcount=3

[techhelpbb]

Just call me curious...

Everyone that has the Jaguar problem on the CAN bus, please consider telling us:

Have you put capacitors on the backs of your motors to dampen the noise?

Have you considered putting capacitors across the encoder power wires?

It's possible that noise from your electrical sources and the brush motors attached to the motor side of the Jaguars is causing issues for the CAN bus.

We've noticed this issue as well, but we have 2 solutions. We detect the failure and force a reset. We are trying to get rid of all the noise that could cause things like this to happen.

[Mike Copioli]

We have also experienced the -44087 error along with the same host of symptoms, It happened once during a practice match and never during an actual match. We of course are using a 2CAN and have only seen the issue when it is used in FRC (cRIO, FRC JAG firmware 92). We have not seen the issue when the 2CAN is being used in the Crosslink Control System. It does seem to be a race condition based on the observed behavior and difficulty reproducing the failure. I have reported the problem to Omar and he has been unable to reproduce the issue. The fact that a soft reboot of the cRIO fixes the issue, and the problem has been seen in both the serial and Ethernet gateways, leads me to believe the problem is above the gateway level. We will continue to test using the 2CAN and advise if there are any bugs found on the side of the 2CAN.

[Hugh Meyer]

Last year we experienced several issues using CAN and lag was one of them.

We connected an oscilloscope to two digital outputs. One scope channel was controlled by software code that turned on at the start and off at the end of our periodic loop. The other channel was driven by a toggle function at the beginning of our code. This helped to define the state and made a good trigger signal.

We found that our code was taking longer than our periodic time. Not a good situation. The problem occurred because of the CAN communications. We were driving 8 Jaguars and polling for just about all the data we could, so we could log the data.

As a result we multiplexed the data out over 4 periods. This reduced our code run time down to about 55 milliseconds. We run our periodic loop at 100 milliseconds.

This arrangement has been institutionalized this year with a special cable that connects directly to the break out board and scope. We regularly verify that the code is running under the period.

A change we made from last year was ground loop isolation. Remove the ground wire in the CAN cable. This is a direct ground loop mess. If you are using one encoder to drive two Jaguars you will need optical isolation to remove the ground loop.

Another change is that we added filtering on the Jaguar at the encoder connector. We were seeing resets on the Jaguars and the filter seems to be the solution. Resets will cause all kinds of problems.

I have posts regarding these issues on other threads.

We are using a 2CAN and C++. The serial port baud rate of 115k was just not fast enough to transfer all the data we wanted to transfer.

We don’t have all of the issues fixed, but I wanted to share these in hopes that it will help others using CAN Jaguar.

Good luck!

-Hugh

[jhersh]

Originally Posted by WizenedEE

Source Please? I'm looking here:

<R59> If CAN-bus communications are used, the CAN-bus must be connected to the cRIO-FRC through either the Ethernet network connected to Port 1, Port 2, or the DB-9 RS-232 port connection.

And

<R50> Connections to the cRIO-FRC Ethernet ports must be compliant with the following parameters:
A. The DAP-1522 radio is connected to the cRIO-FRC Ethernet port 1 (either directly or via a CAT5 Ethernet pigtail).
B. Ethernet-connected COTS devices or custom circuits may connect to either cRIO-FRC Ethernet port; however, these devices may not transmit or receive UDP packets using ports 1100-1200 except for ports 1130 and 1140.

That seems to be two conflicting rules, but the 2CAN would probably count as a "pigtail"

I don't see these as conflicting. A pigtail is simply a wire, not an active component like the 2CAN. The 2CAN does not count as a pigtail. It still doesn't matter, however, since you can plug the 2CAN into one of the other ports on the DAP-1522 switch, not in between the radio and the cRIO.

Originally Posted by WizenedEE

Hooray for rules quotations. I am confused about in rule <R59> they say the radio may be connected directly OR with a pigtail. Them female to female connections, eh?

(Yeah, yeah I know, a pigtail is a male-female...)

The idea of the pigtail for this application is obsolete and probably should be removed from the rules. Instead of ever unplugging your cRIO from your radio, you should instead have a pigtail from one of the other ports on the DAP-1522 that you use to tether to your DS in the pits. This means you will never forget to plug the radio back in and miss a match (or delay it if your FTA is especially generous).

-Joe

[Mark McLeod]

Originally Posted by jhersh

The idea of the pigtail for this application is obsolete and probably should be removed from the rules. Instead of ever unplugging your cRIO from your radio, you should instead have a pigtail from one of the other ports on the DAP-1522 that you use to tether to your DS in the pits. This means you will never forget to plug the radio back in and miss a match (or delay it if your FTA is especially generous).

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

[jhersh]

Originally Posted by Mark McLeod

Not quite true, although in principle I agree.

On the practice field teams must unplug their DLink and replace it with the Practice field DLink.

I guess that's true... however if a team wanted to avoid that, they could simply have a spare power connection for the practice radio and connect it to the tether pigtail.

[John Heden]

I would like to thank everybody for their thoughts on our CAN issue and all the suggestions you have offered. Our team crafted a custom dashboard that saves all of our data to disk including any UDP NetConsole data that it happens to catch and I’ve started to work way through this information looking for possible clues. Our freeze in our first quarter final match did yield some interesting results that I’m still trying to fully understand but certainly looks like a complete loss of CAN integrity through some mechanism. The following is part of the recorded error sequence (it goes on quite a bit longer) that shows a total of 16 InitCANJaguar() calls. The only call I can find to InitCANJaguar, however, is in the constructor for CANJaguar() so I a bit perplexed at this point given we have only 5 CANJaguars. After this initialization like sequence there is a litany of getTransaction() errors before we eventually do a reset and regain control.

At this point I’m partial to the startup race condition theory of some type. I would also add that we do launch a status monitoring thread that does read information from CANJaguar at the end of our robot constructor well before the autonomous loop is initiated. This seems to work but I wonder if the 2CAN occasionally needs a bit more time to settle down before it is called upon for status and CAN transactions…

Thanks again,

John

Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in setTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 392

<Code>-44087 ERROR: status == -44087 (0xFFFF53C9) in getTransaction() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 425

<Code>-63194 ERROR: status == -63194 (0xFFFF0926) in InitCANJaguar() in C:/windriver/workspace/WPILib/CANJaguar.cpp at line 47Etc. etc. etc...

[jhersh]

Originally Posted by John Heden

At this point I’m partial to the startup race condition theory of some type.

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

[techhelpbb]

Originally Posted by jhersh

Me too. I believe I've found and fixed the issue. We will be testing on a real field this evening and working on a plan for distributing the fix.

Please stay tuned.

-Joe

Does this solution also address the problem when it doesn't happen at startup?

We get CAN issues even when the system manages to come fully online.