CAN reliability

[kamocat]

One of the things I like about CAN is that it gives us the ability to detect failures (loss of power, loss of communication, etc).
I made a program to do just that, and log it to a file. After running it for an hour, I got some interesting results. I've linked the code and the log file, so I'll explain the format it is logged in.

First off, the file is created in the top directory, using the name "CAN_status_"+timestamp (In the file I've provided, it is 5:33pm on July 6th.)

Anyways, for each entry (each line), there is the status, the list of devices which the status applies, and the timestamp.
The possible statuses are:

• lost (communication with this Jaguar has been lost)
• power up (power has been cycled since this was last checked)
• got comms (communication with Jaguar has resumed, but the interruption was not due to loss of power)
• brown out (voltage fault)
• over temp (temperature fault)
• over current (current fault)

I was surprised at the number of interruptions there were, seeing as the robot was undisturbed during this hour. It seems each of the black jaguars (devices 10, 11, and 12) had an interruption in communication about once a minute, and the tan jag (device 13) had no interruptions whatsoever.
The interruptions seem to be on the scale of 200ms.

Why might there be communication interruptions on an undisturbed robot? Is enumeration a flaky thing?

[Bot190]

My team used CAN this year on our robot and found some issues with connection. We found that when polling for information while sending output information to fast the jaguars would lose connection briefly. The bandwidth afforded by a serial connection is a lot less than that of a CAN connection. This creates a big chokepoint on sending and receiving information from the jaguars.

[Radical Pi]

I haven't seen the kind of information in your program, but I've found that CAN is VERY unreliable if there are any problems on your robot that affect the code (primarily its execution speed), the jags will become intermittent (watchdogs I think)

When you were running this, was the robot disabled? If so, is there any change when enabled and you can see if the status LEDs blink?

[Ether]

The bandwidth afforded by a serial connection is a lot less than that of a CAN connection.

RS232 and CAN are both serial connections. But their speeds may differ.

Does anyone know the max speed supported for RS232 by the various drivers provided (LabVIEW, Java, C++), and do these drivers offer interrupt-driven receive and transmit buffers?

If 115Kbps and interrupt-driven software buffers are supported by the RS232 drivers, it seems that should be adequate for both commands and data acquisition at 50Hz for 5 motors (depending on how efficiently the data is encoded).


~

~

[Bot190]

By serial, i was refering to the connection on the Crio, The CAN bus has a bandwidth of 1Mb/s much greater than that of the serial connection on the Crio.

[RyanCahoon]

Originally Posted by Bot190

The bandwidth afforded by a serial connection is a lot less than that of a CAN connection.

Marshal - You mention the black jags, so were you using the serial connection as well? (I should say I haven't actually looked at your code)

Does anyone have a 2CAN that they try could the code with? I've been thinking about switching to CAN this year, but if it's going to be flaky, I'm starting to wonder if that's the best idea.

--Ryan

[Ether]

Originally Posted by Bot190

The bandwidth afforded by a serial connection is a lot less than that of a CAN connection. This creates a big chokepoint on sending and receiving information from the jaguars.

cRIO RS232 hardware can do 115200 baud.

That's approximately 11,520 data bytes per second.

At 50Hz, that's 230 bytes every 20ms iteration (230 bytes each direction, xmit & recv).

Are you trying to send or receive more than 230 bytes each iteration?

~

[kamocat]

Originally Posted by Ether

cRIO RS232 hardware can do 115200 baud.

That's approximately 11,520 data bytes per second.

At 50Hz, that's 230 bytes every 20ms iteration (230 bytes each direction, xmit & recv).

Are you trying to send or receive more than 230 bytes each iteration?

~

Let's use the voltage mode "set vbus" command, and assume there message sent over RS232 is the very same message sent over CAN.
The sections are:
Start of Frame (1 bit)
Arbitration field (32 bits)
Control field (6 bits)
Data field (vbus is a 8.8 signed fixed-point number, so that's 16 bits)
Cyclic Redundancy Check field (15 bits)
Acknowledge field (2 bits)
End of Frame (7 bits)

That is 79 bits, or 10 bytes. 230 / 10 is 23. At least for the black Jaguar controlled network, the physical limit for devices is 16. (I don't know the physical limit for 2CAN)
Therefore, at least during normal use, the RS232 communication will have enough throughput. Now, during startup, it may take a little longer. I'll get theoretical stats for how long it should take to initialize a Jaguar for position control in just a bit. (Position control is something that takes a lot of initialization.)
EDIT: I was assuming the time it takes for the message to be sent on the 1Mb/s CAN bus is negligible, but I suppose it could be included. 100 bits / 1Mb/s is about 0.1 ms. (The reason for multiplying it by 10, not 8, is to approximately account for the bit stuffing.) Let's make that 0.2ms to allow for a reply message. 0.0002s * 115,200 bits/s means there's about 23 bits of wait time. I'll round that up to three bytes. So it's 230 / 13, which is a little under 18. If you're using speed mode, that would now be 15 bytes. 230 / 15 is a little over 15, and you're almost at the limit there of 16 devices per network.


Now, I've been assuming worst-case scenario: that the entire CAN message is sent over RS232. It's quite possible that only the arbitration and data fields are being sent. The arbitration field is 4 bytes. For "set vbus", the data field is 2 bytes. Add in the 3 bytes of wait time, and you have 5 bytes. 230 / 5 is 46.
It would then be a 7-byte message for set speed. 230 / 7 is about 33.


NOTE: I am following the Robert Bosch CAN standard. The poster I'm looking at is the vector "CAN Protocol Reference Chart" which you can order (free, I think) from can-solutions.com

[kamocat]

Okay, so on to initialization. Here's a list of commands that would be logical to use when configuring a Jaguar for position control. (I'm choosing position control because it has the most things to configure)

1. Firmware version (0 bytes data)
2. Position Mode Enable (0 bytes data)
3. Position reference (0 bytes data)
4. Proportional Constant (4 bytes data)
5. Integral Constant (4 bytes data)
6. Derivative Constant (4 bytes data)
7. Encoder lines or Potentiometer turns (4 bytes data)
8. Break/Coast (1 byte data)
9. Soft limit switches (1 byte data)
10. Forward soft limit (5 bytes data)
11. Reverse soft limit (5 bytes data)
12. Position Set (4 bytes data)

The sum of data bytes being sent here is 32 bytes, though it is being sent over 12 messages.
If the whole CAN message is sent through RS232, that should be (14*(8+3)+32)bytes / 11,520 bytes/s or 16ms to initialize a Jaguar for position mode.

If only the arbitration field and the data is sent, it should be (14*(4+3)+32)bytes / 11,520 bytes/s or 9ms to initialize a Jaguar for position mode.

So, now that I have some theoretical calculations to compare it to, I shall make some tests.

[Bot190]

Looking at the Can Jaguar library for C++, The maximum size of a sent message is 10 bytes, as was previously determined, and the maximum size of a received message is 14 bytes.
This assumes that a separate message is sent for every request to every jaguar. In this experiment 5 jaguars were used, this would limit the requests for information to 4, but limits the data being received to 3 messages.
Please correct me if I'm not thinking about this right, or if anything looks off.

[kamocat]

Originally Posted by Bot190

Looking at the Can Jaguar library for C++, The maximum size of a sent message is 10 bytes, as was previously determined, and the maximum size of a received message is 14 bytes.
This assumes that a separate message is sent for every request to every jaguar. In this experiment 5 jaguars were used, this would limit the requests for information to 4, but limits the data being received to 3 messages.
Please correct me if I'm not thinking about this right, or if anything looks off.

It seems I don't quite understand their thinking. The maximum length of the data field in CAN is 8 bytes, so the maximum length should be 12 bytes. (I suppose it's true the Jaguars don't have any messages with data fields longer than 5 bytes)
Perhaps there is some error handling in the message returned, which would be added on by the main Black Jaguar?

You are correct that a separate message is sent to each jaguar, in most situations. There are some messages (heartbeat and enumerate) that are sent to all devices. I'm pretty sure that the main Black Jaguar acts both as the master and a slave on the CAN bus.
I don't understand where your numbers (4 and 3) came from. Could you please elaborate?

[Bot190]

If every data request is 10 bytes, and your requesting it from 5 jaguars, 10 bytes * 5 jaguars = 50 bytes, 230 bytes / 50 bytes = 4.6, rounded down to 4 messages.

Given 14 bytes per response for 5 jaguars, 14 bytes * 5 jaguars = 70 bytes, 230 bytes / 70 bytes = 3.3 rounded down to 3 messages. The size of the Messages was based off the C++ Jaguar library, in the JaguarCANDriver.h file if you want to look.

[Radical Pi]

Just a quick confirmation for your assumption about the serial-to-can bridge. Assuming FIRST didn't change too much from the default Jaguar firmware, I'm looking at the default source code right now and it looks like it directly resends the body of the serial message over the CAN bus if the message is intended for another jaguar or if it's meant for all the jaguars. It looks like the CAN headers are added on in the process, so the serial message is only containing the device ID and the message itself. If the message is intended for the bridging jaguar, it never reaches the CAN bus and is just directly processed.

[kamocat]

Originally Posted by Radical Pi

Just a quick confirmation for your assumption about the serial-to-can bridge. Assuming FIRST didn't change too much from the default Jaguar firmware, I'm looking at the default source code right now and it looks like it directly resends the body of the serial message over the CAN bus if the message is intended for another jaguar or if it's meant for all the jaguars. It looks like the CAN headers are added on in the process, so the serial message is only containing the device ID and the message itself. If the message is intended for the bridging jaguar, it never reaches the CAN bus and is just directly processed.

Really? That's not what I expected, but I'm glad you looked it up.
I suppose this means there could be a small difference in communication speed between the bridging jaguar and the rest.

Bot190:
I understand your reasoning now, thank you.
From this new information, it appears that a 6-byte data field can be sent, but a 10 byte data field can be received. The maximum data field length in CAN is normally 8 bytes, so perhaps those last two bytes are used for something else.

[kamocat]

Okay, I tested how long it took to configure a Jaguar for Position mode.


Clearly, there is something here I'm not accounting for. My estimate was 16ms, and it's turning out to be 172ms.
The figure is very consistent, always 172 or 173 with this combination of actions.
There is no difference in time between the black or tan jags, nor between the bridging jag and the others.

(For some reason, the Device Query action was taking 500ms, and not returning anything, so took that out to make the "total time elapsed" a relevant figure.)

EDIT: Device Query now seems to be taking about 172ms, but still not returning anything.
Anyways, when I run this VI in a loop, with DS Comms running in parallel, it seems to take around 220ms disabled, 240ms enabled. (teleop vs autonomous doesn't make a difference) However, it has a LOT more jitter; the time varies by at least 10ms each way.