Intermittent connection on field only

[Al Skierkiewicz]

Brian,
These students???...

Originally Posted by techhelpbb

Furthermore, some of those students you have on that field crew are students I mentored. So let me be clear. I am extremely unhappy when I see students in my chosen profession. A profession to which I have been an apprentice. To which I have clawed my way up in versus vast nonsense. I am extremely unhappy when I watch them get reduced to tears. I am tired of FIRST using them as reason why I should watch what I say.

[techhelpbb]

Originally Posted by Al Skierkiewicz

Brian,
These students???...

A fair number of your field crews are college students. They come back and help out.
No specific event was implied.
Further, I'm including all the personnel at an event in the crew.

Additionally (post 139):
http://www.chiefdelphi.com/forums/sh...106042&page=10
"This could have been prevented long before Einstein, but that is a discussion for other threads which we have all been following all season. I saw one FTA in tears afterwards, and I know everyone involved was doing all they could."

[Al Skierkiewicz]

Hugh,
The LED wall does give off a fair amount of noise. It is mostly in the form of buzz and wreaks havoc with dynamic microphones. That's why you don't see anyone speaking from behind the scorer's table on the big stage I would suspect. I have some experience with RF mics and wireless "IN EAR" monitors and have used a spectrum analyzer when checking for problems in several shows here. Most are in the UHF band now that the FCC has been involved in unlicensed wireless devices in unused TV channels. Almost everyone is using Wi-Fi to interconnect various audio interfaces and haven't had any issues. I have not seen anything in the bands I am looking at so that leads me to believe that harmonics beyond 700 MHz do not exist. Each little block connects to the one next to it to assemble a large screen. Then you tell the master-controller how many blocks wide and high and if you want it to portion the screen or simply display the entire video signal. There is a CRT monitor on the scorer's table but is an analog device as I remember. There are a variety of LCD monitors as well. Most of that stuff has a fair amount of shielding internal so I don't expect anything in 5 GHz range from any of that stuff.

[RyanN]

I've been messing around with things today and yesterday trying to break the robot.

Power and brownouts should be a non-issue. If the teams use the OEM cable, the connection remains consistent and has no drop outs.

We also were running with a dead battery. The first thing to give out was the digital sidecar. It resulted in the robot quickly flashing our LEDs on and off as it would brown out the DS,.causing the relays to turn on and off rapidly. The voltage went down to about 4V before I shut off the light show. The cRIO (4-slot) and the Dlink remained powered and connected the whole time with no dropouts.

We have also been testing CAN out a bit, and found some unsettling results.

If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time.

This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this.

We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again.

So that's another idea that I had as far as troubleshooting.

We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk.

I think it has to deal with calling CAN devices at different times. We have Periodic Tasks running a shooter control loop every 50ms, and then Teleop is called every 50ms. I think there is a resource conflict and timing issues and we occasionally get a bus lockup for a split second. Going back to PWM, we run at a constant of 3 to 4ms to run the Telop task.

[Mark McLeod]

Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.

You've got something else wrong going on with Teleop.

[RyanN]

Originally Posted by Mark McLeod

Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.

You've got something else wrong going on with Teleop.

I'm speaking of the actual time it takes to finish the teleop vi, not how often it is called. It is still called every 50ms as it should be. The actual processing of tr teleop vi is what takes only 3-4ms, and it should be pretty low and consitent for all teams.

[linuxboy]

[quote=RyanN;1168414
We have also been testing CAN out a bit, and found some unsettling results.

If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time.
[/QUOTE]
I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.

[quote=RyanN;1168414
This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this.
[/QUOTE]
Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"
[quote=RyanN;1168414
We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again.

So that's another idea that I had as far as troubleshooting.

We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk.
[/QUOTE]
So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

[mjcoss]

I know that in the C++ case TeleopPeriodic is triggered, by default, on the arrival of a DS packets. This is bad if the network has a lot of jitter or latency issues. You can set the period, but then of course you need to handle the case where you have stale data.

[RyanN]

Originally Posted by linuxboy

I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

[Alan Anderson]

Originally Posted by RyanN

The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.

[Jon236]

Originally Posted by Alan Anderson

There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.

Alan,

Great analysis. What I'm trying to figure out is how we've advanced, if when we used C, we would be careful about interrupt priorities so that critical functions (i.e. packet transmission) were not delayed. When I use Labview, I run sensors and actuators in separate vi's, not in the main case structure. I know that if I'm not careful, a separate process can eat up resources as well, but doesn't Labview allow you to prioritize specific processes? Could we then not protect comms? Of course, careless programming might have the robot stop because of the causes you outlined. But the comms would stay up.

[Racer26]

While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

[RyanN]

Originally Posted by linuxboy

I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

Originally Posted by 1075guy

While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

[Al Skierkiewicz]

Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

[linuxboy]

Originally Posted by Al Skierkiewicz

Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

The terminator is required by the CAN spec, but it is there to prevent "signal reflection" which messes things up, however it will operate, without it, albeit with a smaller number of devices on the bus, however, my understanding is that no terminator at the end of the bus leads to issues with the Jaguars at the beginning of the bus, and lacking a terminating resistor at the beginning of the bus leads to issues at the end of it.

I have seen pictures of CAN splitters (there is a discussion about them on here somewhere), but they need terminators on each branch, and I believe the more branches you have the shorter each cable must be. In terms of CAN not being something that should shut everything down, when it comes to error handling and timeouts it can, even if it shouldn't.

Originally Posted by RyanN

Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

I wasn't saying that CAN could have caused the problem at Bayou for you guys, I was just responding to the testing you were doing with it. I really do hope that the root cause of what happened to you guys at Bayou is found, I'd love to hear it.