Intermittent connection on field only

[RyanN]

Originally Posted by Mark McLeod

Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.

You've got something else wrong going on with Teleop.

I'm speaking of the actual time it takes to finish the teleop vi, not how often it is called. It is still called every 50ms as it should be. The actual processing of tr teleop vi is what takes only 3-4ms, and it should be pretty low and consitent for all teams.

[linuxboy]

[quote=RyanN;1168414
We have also been testing CAN out a bit, and found some unsettling results.

If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time.
[/QUOTE]
I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.

[quote=RyanN;1168414
This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this.
[/QUOTE]
Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"
[quote=RyanN;1168414
We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again.

So that's another idea that I had as far as troubleshooting.

We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk.
[/QUOTE]
So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

[mjcoss]

I know that in the C++ case TeleopPeriodic is triggered, by default, on the arrival of a DS packets. This is bad if the network has a lot of jitter or latency issues. You can set the period, but then of course you need to handle the case where you have stale data.

[RyanN]

Originally Posted by linuxboy

I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

[Alan Anderson]

Originally Posted by RyanN

The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.

[Al Skierkiewicz]

Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

[Jon236]

Originally Posted by Alan Anderson

There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.

Alan,

Great analysis. What I'm trying to figure out is how we've advanced, if when we used C, we would be careful about interrupt priorities so that critical functions (i.e. packet transmission) were not delayed. When I use Labview, I run sensors and actuators in separate vi's, not in the main case structure. I know that if I'm not careful, a separate process can eat up resources as well, but doesn't Labview allow you to prioritize specific processes? Could we then not protect comms? Of course, careless programming might have the robot stop because of the causes you outlined. But the comms would stay up.

[Racer26]

While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

[RyanN]

Originally Posted by linuxboy

I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

Originally Posted by 1075guy

While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

[linuxboy]

Originally Posted by Al Skierkiewicz

Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

The terminator is required by the CAN spec, but it is there to prevent "signal reflection" which messes things up, however it will operate, without it, albeit with a smaller number of devices on the bus, however, my understanding is that no terminator at the end of the bus leads to issues with the Jaguars at the beginning of the bus, and lacking a terminating resistor at the beginning of the bus leads to issues at the end of it.

I have seen pictures of CAN splitters (there is a discussion about them on here somewhere), but they need terminators on each branch, and I believe the more branches you have the shorter each cable must be. In terms of CAN not being something that should shut everything down, when it comes to error handling and timeouts it can, even if it shouldn't.

Originally Posted by RyanN

Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

I wasn't saying that CAN could have caused the problem at Bayou for you guys, I was just responding to the testing you were doing with it. I really do hope that the root cause of what happened to you guys at Bayou is found, I'd love to hear it.

[Al Skierkiewicz]

Oliver,
The nature of the signaling on the CAN bus uses switching in the transceivers to cause a device to enter a 'dominant' state for transmit and a 'recessive' state for receive. It is my understanding this is accomplished by driving the bus in a way that causes current flow in the terminating resistors. The CAN terminating resistors also provide the transmission line termination to allow 1 Mb/s traffic on a buss 40 meters long. Microchip has a nice application note, AN228, describing one of their products and the CAN signaling.

[EricVanWyk]

Mikets did some interesting work on CAN backboning and posted about it here: http://www.chiefdelphi.com/forums/sh...d.php?t=102108

[Al Skierkiewicz]

Thanks,
I remember that discussion now.

[linuxboy]

Originally Posted by Al Skierkiewicz

Oliver,
The nature of the signaling on the CAN bus uses switching in the transceivers to cause a device to enter a 'dominant' state for transmit and a 'recessive' state for receive. It is my understanding this is accomplished by driving the bus in a way that causes current flow in the terminating resistors. The CAN terminating resistors also provide the transmission line termination to allow 1 Mb/s traffic on a buss 40 meters long. Microchip has a nice application note, AN228, describing one of their products and the CAN signaling.

I will talke a look at the info about that chip, thanks! My comments are based purely on observation of the behavior in certain fault conditions with the FRC control system. I don't know exactly how the internals of CAN work (yet, this is what the offseason is for!).

- Oliver

[Al Skierkiewicz]

Oliver,
Still learning myself. Considering the actual implementation was for cars, I would suspect that the protocol would allow for devices to go off line without killing the whole thing. So I am wondering what actually would cause the effect described above with a loss of a single device. Having seen the internal destruction on some Jaguars, it is always possible that an internal failure could have taken out the transceiver and shut down (shorted) the bus. In that condition, I would guess it is possible for the Crio to keep attempting to transmit data and do nothing else. I do not have any data to suggest that is what happened, just that it is possible. I bet Jim Zondag would know or at least could render an opinion.