Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   Electrical (http://www.chiefdelphi.com/forums/forumdisplay.php?f=53)
-   -   Intermittent connection on field only (http://www.chiefdelphi.com/forums/showthread.php?t=104713)

Jon236 01-05-2012 11:09

Re: Intermittent connection on field only
 
Quote:

Originally Posted by RyanN (Post 1164450)
Then comes the driver station. We had been using a Dell purchased this season as the DS, but during all the debugging, we switched to my laptop, a 2010 MacBook Pro (Core i5, 8GB RAM, SSD... a pretty powerful machine), and we kept on having the issues. Finally, when trying out the basic framework code, we switched to the Classmate PC, configured for this year and updated with the latest FRC updates, and no Windows Updates.



Just curious, was your wifi adapter disabled on your laptop when you connected to FMS?

techhelpbb 01-05-2012 11:11

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1165139)
Brian,
Your last post is confusing, what are you trying to say? Are you saying I act with favoritism because I don't agree with your suggestions? With your methods or with your premise?

No I am merely stating that the power supply issues, on the whole robot in point of fact, are often a great big grab bag of problems many of which I feel are beyond the resources of some teams.

We don't expect teams to solder. If they can great. We don't expect teams to engineer the cRIO. If they can great. We expect teams to follow directions, but what directions can I hand them to follow?

Where are the directions and parameters to load test that power supply? What tools do they need? What do the tests in the directions tell, or not tell about the robot?

I am not the first to ask these questions. When we ignore that some teams are far more equipped than others. When we ignore that off the competition fields these teams may be more greatly isolated. We create a situation that while I agree with you seems so simple...to them is disaster looking for a place to happen for them.

Quote:

What are you referring to in the paragraphs about students and field teams?
I am suggesting that you use your monitors on your own robot on your practice field to give First and the community some data to digest and duplicate. Are you against that?
I'll be at Monty Madness at the request of field personnel. I am not against it at all.

Quote:

As to you having something on your robot during Monty Madness, that is up to you and the event staff.
Exactly. I was willing to let FIRST shut down the idea of monitoring the D-Link power supply with my my monitors without too much fuss because it was made clear to me that off season events are the place to beta test. I agree with the KOP team at FIRST that they should have all opportunity to review a piece of test hardware they've not seen before.

I was surprised when FIRST said the cRIO couldn't monitor the power supply of the D-Link AP. In effect that means that we can't look while the robots are running on the competition field with hardware FIRST should by now know quite well.

Quote:

As to First's reaction to your hardware, I have no knowledge of what you are relating to us. For the record I am not FIRST staff, I am not a member of the GDC, I am a key volunteer.
Fair enough Al. It's just that this a FIRST sponsored forum. It's hard to ignore the authority from which some of you speak.

Quote:

As to adding monitoring to a robot, may I refer you to StangSense. We used the control system in conjunction with a custom circuit, to monitor current/voltage to critical systems, mostly motors, using a method that did not interrupt the power pathways to electrical components on our robot. We also used that system in a mobile design to aid other teams in the pursuit of problems on their robots for several years. We also described the data we collected and the subsequent issues we observed to anyone who wanted them and that data aided in the design of the current PD power circuitry. FIRST added several test bed monitors to operating robots (including ours) during the design phase as well. I assisted with at least some of the interpretation of that collected data during the collection at IRI. The boost/buck regulators on the PD are a direct result (I believe) of that research.
I have no doubt that StangSense and your work is well considered Al. The problem for me remains. That you do not personally build all of our robots. The need for teams to collect data doesn't really end in a beta or in just a few robots or even in the pits at competition.

Data collection against issues, assuming proper respect for doing it properly, should not discouraged in any way. In fact, it should be an extremely common thing for FIRST teams beyond just reading the voltage of the battery. I don't feel based on the current rules and based on the current status of things that such collection is widely encouraged. Worse, because of a lack of simple instructions I feel that it's beyond the reach of some people that really need it.

Al Skierkiewicz 01-05-2012 11:39

Re: Intermittent connection on field only
 
Quote:

Originally Posted by techhelpbb (Post 1165163)
Fair enough Al. It's just that this a FIRST sponsored forum. It's hard to ignore the authority from which some of you speak.

This is not a FIRST sponsored forum, this is a team forum and always has been. Only the First Forum is sponsored by First. For myself, as for others that may or may not be FIRST staff, we write here for the teams that need advice. As part of that responsibility we (the collective CD community) feel strongly in correcting occasional errors in posts to insure that teams get the answers they need. I write personal opinions on issues related to the community in general from my collective experience (17 years on WildStang and 43 years as a Broadcast Engineer).

I still have to ask what your reference to students on field teams means?

techhelpbb 01-05-2012 12:15

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1165173)
This is not a FIRST sponsored forum, this is a team forum and always has been. Only the First Forum is sponsored by First. For myself, as for others that may or may not be FIRST staff, we write here for the teams that need advice. As part of that responsibility we (the collective CD community) feel strongly in correcting occasional errors in posts to insure that teams get the answers they need. I write personal opinions on issues related to the community in general from my collective experience (17 years on WildStang and 43 years as a Broadcast Engineer).

Fair enough if I am mistaken. Your forum's position is well established in FIRST circles and the frankly your forum actually often comes up before FIRST forums in a variety of situations. My apologies for misinterpreting the extent of your influence.

I just want to make myself very clear again, that nothing I've written in this topic is intended to devalue your views or your experience.

Quote:

I still have to ask what your reference to students on field teams means?
It all comes down to how leadership decides to handle situations.

I see people all over the place in these topics suggesting that some of us have lost confidence in the field personnel because we speculate and try to solve the problems (funny to be asked not to solve problems in situations where it's the task of the day normally...especially difficult considering the enthusiasm that FIRST inspires in regard to troubleshooting regardless of whom you are or what you know). That somehow we have lost confidence in FIRST's ability to troubleshoot this issue on the field.

I have lost no confidence in any field personnel.
None at all. Not one field person I've seen recently seemed to me to be intent on making the playing field anything other than fair, circumstances beyond their control may influence the situation otherwise however. It was certainly clear to me on several occasions they were not fully equipped or fully prepared with quick, direct and specific resolutions to a variety of problems. Again there's a limit to what they can bring to the table, having never seen your robot before, having no access to valuable tools, and sometimes not having information about things that could and probably should have been given to them before these events. As my example in this topic, I was expressly assured by field personnel that the field was on 2.4GHz...not 5Ghz. That's a problem that needs to be addressed, it was a mere simple error I'm sure, but it's an error that masks other communications issues. Collecting anecdotal information before events is no real solution to the issue either of getting these field folk and the teams the information they need. Continuous, specific, directed data collection is a good solution.

I also brought that up because some people were genuinely thrilled that I had actually bothered to think to bring an oscilloscope to an event as the guy that didn't even know he was competition field spare parts until the evening before. People at that event in management of it made it very clear they intended to recommend that an electronics table, or at least the relevant basic tools were available to teams at events. They were thrilled that I sat there making tails for 2GO PC's with broken Ethernet ports. I even made a bunch of cables for the field and for teams. It was extremely clear to me that some of these teams had no resources to do this for themselves I wonder what would have become of them had I not stepped up.

When I wander through forums often reading but not posting to topics and I see people sort of suggesting that field personnel are somehow to deal with this. When I see people suggesting we should be quiet because the field personnel might feel we are being predatory on them. It makes me very unhappy. The field personnel have done all they could, in many cases much more than they should have been able to do...

However, they are restrained by the tools, by the quality of their instructions, and most importantly by restrictions on information that need to be removed.

I don't find any fault for these people running the fields, in the end FIRST is the central control canopy for all that happens. If the information doesn't get there. If the instructions were not given. If the tools were not there. FIRST needs to do something about it. Not have some people running around topics suggesting that other people are not patient with the field personnel or that the field personnel are the issue. There's no need for people to be silenced, but we can all see the unprecedented and absolute need that this problem...in fact all problems that drop robots dead on the field be meticulously managed and removed.

Data collection needs to be done. It needs to be done to analyze this mess at the Championship. It needs to be done within the organization. It needs to be a tool handed to every nook and cranny of the people involved and it does need to involve the people in the community. It needs to be continuous and it shouldn't be disappearing behind the curtains now. Blame is irrelevant and again purely anecdotal.

Hugh Meyer 01-05-2012 13:36

Re: Intermittent connection on field only
 
Out of band interference could also be part of the problem.

I remember at the 2007 DARPA Grand Challenge they had a huge video display setup near the starting line. The three robots closest to the screen couldn’t move until they shut down the giant display. It was about the same distance the Einstein field was from the display in the dome.

Another possible culprit is the high concentration of video and other equipment near to the FMS access point. I often see the access point sitting adjacent to a television video monitor, which I would consider as bad as putting the robot radio on top of a CIM motor.

-Hugh

Al Skierkiewicz 01-05-2012 14:21

Re: Intermittent connection on field only
 
Brian,
These students???...

Quote:

Originally Posted by techhelpbb (Post 1165133)
Furthermore, some of those students you have on that field crew are students I mentored. So let me be clear. I am extremely unhappy when I see students in my chosen profession. A profession to which I have been an apprentice. To which I have clawed my way up in versus vast nonsense. I am extremely unhappy when I watch them get reduced to tears. I am tired of FIRST using them as reason why I should watch what I say.


techhelpbb 01-05-2012 14:24

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1165252)
Brian,
These students???...

A fair number of your field crews are college students. They come back and help out.
No specific event was implied.
Further, I'm including all the personnel at an event in the crew.

Additionally (post 139):
http://www.chiefdelphi.com/forums/sh...106042&page=10
"This could have been prevented long before Einstein, but that is a discussion for other threads which we have all been following all season. I saw one FTA in tears afterwards, and I know everyone involved was doing all they could."

Al Skierkiewicz 01-05-2012 14:59

Re: Intermittent connection on field only
 
Hugh,
The LED wall does give off a fair amount of noise. It is mostly in the form of buzz and wreaks havoc with dynamic microphones. That's why you don't see anyone speaking from behind the scorer's table on the big stage I would suspect. I have some experience with RF mics and wireless "IN EAR" monitors and have used a spectrum analyzer when checking for problems in several shows here. Most are in the UHF band now that the FCC has been involved in unlicensed wireless devices in unused TV channels. Almost everyone is using Wi-Fi to interconnect various audio interfaces and haven't had any issues. I have not seen anything in the bands I am looking at so that leads me to believe that harmonics beyond 700 MHz do not exist. Each little block connects to the one next to it to assemble a large screen. Then you tell the master-controller how many blocks wide and high and if you want it to portion the screen or simply display the entire video signal. There is a CRT monitor on the scorer's table but is an analog device as I remember. There are a variety of LCD monitors as well. Most of that stuff has a fair amount of shielding internal so I don't expect anything in 5 GHz range from any of that stuff.

RyanN 09-05-2012 21:24

Re: Intermittent connection on field only
 
I've been messing around with things today and yesterday trying to break the robot.

Power and brownouts should be a non-issue. If the teams use the OEM cable, the connection remains consistent and has no drop outs.

We also were running with a dead battery. The first thing to give out was the digital sidecar. It resulted in the robot quickly flashing our LEDs on and off as it would brown out the DS,.causing the relays to turn on and off rapidly. The voltage went down to about 4V before I shut off the light show. The cRIO (4-slot) and the Dlink remained powered and connected the whole time with no dropouts.

We have also been testing CAN out a bit, and found some unsettling results.

If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time.

This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this.

We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again.

So that's another idea that I had as far as troubleshooting.

We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk.

I think it has to deal with calling CAN devices at different times. We have Periodic Tasks running a shooter control loop every 50ms, and then Teleop is called every 50ms. I think there is a resource conflict and timing issues and we occasionally get a bus lockup for a split second. Going back to PWM, we run at a constant of 3 to 4ms to run the Telop task.

Mark McLeod 09-05-2012 22:08

Re: Intermittent connection on field only
 
Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.

You've got something else wrong going on with Teleop.

RyanN 10-05-2012 12:36

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Mark McLeod (Post 1168423)
Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.

You've got something else wrong going on with Teleop.

I'm speaking of the actual time it takes to finish the teleop vi, not how often it is called. It is still called every 50ms as it should be. The actual processing of tr teleop vi is what takes only 3-4ms, and it should be pretty low and consitent for all teams.

linuxboy 10-05-2012 19:45

Re: Intermittent connection on field only
 
[quote=RyanN;1168414
We have also been testing CAN out a bit, and found some unsettling results.

If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time.
[/QUOTE]
I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.

[quote=RyanN;1168414
This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this.
[/QUOTE]
Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"
[quote=RyanN;1168414
We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again.

So that's another idea that I had as far as troubleshooting.

We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk.
[/QUOTE]
So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

mjcoss 10-05-2012 20:18

Re: Intermittent connection on field only
 
I know that in the C++ case TeleopPeriodic is triggered, by default, on the arrival of a DS packets. This is bad if the network has a lot of jitter or latency issues. You can set the period, but then of course you need to handle the case where you have stale data.

RyanN 11-05-2012 01:33

Re: Intermittent connection on field only
 
Quote:

Originally Posted by linuxboy (Post 1168628)
I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

Alan Anderson 11-05-2012 07:28

Re: Intermittent connection on field only
 
Quote:

Originally Posted by RyanN (Post 1168703)
The compressor is controlled in its own loop in periodic tasks and called every second. For no reason should CAN errors cause it to drop out like it is.

There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.

Al Skierkiewicz 11-05-2012 07:35

Re: Intermittent connection on field only
 
Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

Jon236 11-05-2012 09:48

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Alan Anderson (Post 1168716)
There actually is a possible reason for CAN errors to shut off the compressor. It's a second-order effect. The error processing and reporting uses up a lot of system resources, delaying other actions, including the communication task. The system watchdog times out because the data from the Driver Station didn't get processed in time. Every PWM and Relay output on the Digital Sidecar gets shut off. Since the compressor is controlled by a Spike relay, it stops.


Alan,

Great analysis. What I'm trying to figure out is how we've advanced, if when we used C, we would be careful about interrupt priorities so that critical functions (i.e. packet transmission) were not delayed. When I use Labview, I run sensors and actuators in separate vi's, not in the main case structure. I know that if I'm not careful, a separate process can eat up resources as well, but doesn't Labview allow you to prioritize specific processes? Could we then not protect comms? Of course, careless programming might have the robot stop because of the causes you outlined. But the comms would stay up.

Racer26 11-05-2012 09:57

Re: Intermittent connection on field only
 
While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

RyanN 11-05-2012 11:08

Re: Intermittent connection on field only
 
Quote:

Originally Posted by linuxboy (Post 1168628)
I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck.


Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms"

So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop.

Quote:

Originally Posted by 1075guy (Post 1168743)
While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.

Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

linuxboy 11-05-2012 11:48

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1168719)
Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.
Has anyone used a CAN splitter to any advantage?

The terminator is required by the CAN spec, but it is there to prevent "signal reflection" which messes things up, however it will operate, without it, albeit with a smaller number of devices on the bus, however, my understanding is that no terminator at the end of the bus leads to issues with the Jaguars at the beginning of the bus, and lacking a terminating resistor at the beginning of the bus leads to issues at the end of it.

I have seen pictures of CAN splitters (there is a discussion about them on here somewhere), but they need terminators on each branch, and I believe the more branches you have the shorter each cable must be. In terms of CAN not being something that should shut everything down, when it comes to error handling and timeouts it can, even if it shouldn't.

Quote:

Originally Posted by RyanN (Post 1168757)
Trust me. I know that wasn't the issue. It also wasn't our issue at bayou because we got rid of everything on our robot besides motors and solenoids. All using PWM, relay, and solenoid outputs, no camera.

It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down.

I wasn't saying that CAN could have caused the problem at Bayou for you guys, I was just responding to the testing you were doing with it. I really do hope that the root cause of what happened to you guys at Bayou is found, I'd love to hear it.

Al Skierkiewicz 11-05-2012 12:43

Re: Intermittent connection on field only
 
Oliver,
The nature of the signaling on the CAN bus uses switching in the transceivers to cause a device to enter a 'dominant' state for transmit and a 'recessive' state for receive. It is my understanding this is accomplished by driving the bus in a way that causes current flow in the terminating resistors. The CAN terminating resistors also provide the transmission line termination to allow 1 Mb/s traffic on a buss 40 meters long. Microchip has a nice application note, AN228, describing one of their products and the CAN signaling.

EricVanWyk 11-05-2012 12:56

Re: Intermittent connection on field only
 
Mikets did some interesting work on CAN backboning and posted about it here: http://www.chiefdelphi.com/forums/sh...d.php?t=102108

Al Skierkiewicz 11-05-2012 14:05

Re: Intermittent connection on field only
 
Thanks,
I remember that discussion now.

linuxboy 11-05-2012 14:06

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1168776)
Oliver,
The nature of the signaling on the CAN bus uses switching in the transceivers to cause a device to enter a 'dominant' state for transmit and a 'recessive' state for receive. It is my understanding this is accomplished by driving the bus in a way that causes current flow in the terminating resistors. The CAN terminating resistors also provide the transmission line termination to allow 1 Mb/s traffic on a buss 40 meters long. Microchip has a nice application note, AN228, describing one of their products and the CAN signaling.

I will talke a look at the info about that chip, thanks! My comments are based purely on observation of the behavior in certain fault conditions with the FRC control system. I don't know exactly how the internals of CAN work (yet, this is what the offseason is for!).

- Oliver

Al Skierkiewicz 11-05-2012 14:12

Re: Intermittent connection on field only
 
Oliver,
Still learning myself. Considering the actual implementation was for cars, I would suspect that the protocol would allow for devices to go off line without killing the whole thing. So I am wondering what actually would cause the effect described above with a loss of a single device. Having seen the internal destruction on some Jaguars, it is always possible that an internal failure could have taken out the transceiver and shut down (shorted) the bus. In that condition, I would guess it is possible for the Crio to keep attempting to transmit data and do nothing else. I do not have any data to suggest that is what happened, just that it is possible. I bet Jim Zondag would know or at least could render an opinion.

techhelpbb 11-05-2012 18:04

Re: Intermittent connection on field only
 
http://www.ti.com/lit/ds/symlink/sn65hvd1050.pdf

The application note for the CAN bus transceiver indicates that it has a dominant-time-out circuit that should stop a software error from simply holding the open-collector CAN transmission circuit down to ground and stopping all communication on the CAN bus. (IE the microcontroller crashes and stops doing anything of value and locks up the entire CAN bus.)

However this protection circuit has a fault. It only stops the Jaguar from stomping on the CAN bus if the TXD pin from the microcontroller goes to a logic low and stays there. It does not stop the Jaguar from going crazy and simply randomly stomping on the bus by changing the logic state of the TXD pin at bad moments. (IE the microcontroller gets overwhelmed by say interrupts from the encoder and starts running around speaking gibberish on the CAN bus.)

A quote from that datasheet:
"A dominant-time-out circuit in the SN65HVD1050 prevents the driver from blocking network communication with a hardware or software failure. The time-out circuit is triggered by a falling edge on TXD (pin 1). If no rising edge is seen before the time-out constant of the circuit expires, the driver is disabled. The circuit is then reset by the next rising edge on TXD."

If something can overwhelm the Jaguar's internal processing (like the encoder can) and cause it misunderstand it's timing on the CAN bus it could still stomp out the communications of other Jaguars occasionally or even sufficiently to lock the entire bus if the Jaguar's microcontroller can just keep picking the right moments to fail to run silent on the bus.

Dropping even a few bits here and there randomly from the protocol should be sufficient to render the expected functions of the entire CAN bus essentially erratic.

More advanced CAN solutions eliminate this possibility because they are not merely drivers. They are themselves peripherals designed not to fail in this application. If your processor were to crash while driving a controller like this (and I've done it plenty of times) the CAN bus would continue to operate regardless of that failure.

For example:
http://ww1.microchip.com/downloads/e...doc/21801d.pdf

The CAN solution in the Jaguar per the datasheet:
http://www.ti.com/lit/ds/spms047g/spms047g.pdf

Does implement a controller, but it's interrupt based. So if the processing unit is overwhelmed it may not realize that it's failed to meet it's timing requirements and then bad news (from page 569):

"15.3.14 Bit Timing Configuration Error Considerations
Even if minor errors in the configuration of the CAN bit timing do not result in immediate failure, the
performance of a CAN network can be reduced significantly. In many cases, the CAN bit
synchronization amends a faulty configuration of the CAN bit timing to such a degree that only
occasionally an error frame is generated. In the case of arbitration, however, when two or more
CAN nodes simultaneously try to transmit a frame, a misplaced sample point may cause one of the
transmitters to become error passive. The analysis of such sporadic errors requires a detailed
knowledge of the CAN bit synchronization inside a CAN node and of the CAN nodes' interaction on
the CAN bus."

Also the CAN solution in the Jaguar is adaptive for timing so if you start spewing badly timed bits onto the bus then this can cause you issues (page 559):
"Once the CAN module is initialized and the INIT bit in the CANCTL register is cleared, the CAN module synchronizes itself to the CAN bus and starts the message transfer. As each message is received, it goes through the message handler's filtering process, and if it passes through the filter, is stored in the message object specified by the MNUM bit in the CAN IFn Command Request (CANIFnCRQ) register."

mjcoss 11-05-2012 19:24

Re: Intermittent connection on field only
 
This seems like the exact situation that we had. We had one of Jaguars on our shooter motor at the Rutgers competition flood the CAN bus, and since we were/are an all CAN bus robot, we were dead in the water. Why and how it got in that state, and stayed there is not clear. Power cycling the robot did not clear it. We replaced the offending Jaguar, after we replaced the 2CAN because as far as we could tell, the whole bus was dead.

linuxboy 12-05-2012 00:55

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1168796)
Oliver,
Still learning myself. Considering the actual implementation was for cars, I would suspect that the protocol would allow for devices to go off line without killing the whole thing. So I am wondering what actually would cause the effect described above with a loss of a single device. Having seen the internal destruction on some Jaguars, it is always possible that an internal failure could have taken out the transceiver and shut down (shorted) the bus. In that condition, I would guess it is possible for the Crio to keep attempting to transmit data and do nothing else. I do not have any data to suggest that is what happened, just that it is possible. I bet Jim Zondag would know or at least could render an opinion.

It isn't a CAN protocol problem, it is a cRIO problem that causes the problems on the CAN bus to affect the entire robot. If you take a look at the traces on the Jaguar PCB you should actually see that the two RJ12 ports actually have traces connecting the pins on both (pass thorough) such that even with a unpowered jaguar the CAN signal should still be passed through (although you will still get timeout errors).

The reason the cRIO causes problems is the error handling is expensive processor time wise especially if the errors are not handled correctly (printing lots of errors to the console and such).

Al Skierkiewicz 12-05-2012 08:38

Re: Intermittent connection on field only
 
Oliver,
If we go on the assumption that the CAN is echoed through the serial port, it is not hard to imagine that something is confusing the traffic flow there. I can't tell you the priority the Crio takes when handling serial port issues but I would guess it would be one of the more important tasks. A big flaw in the Jaguar in my opinion, is the CAN connectors facing up and having a rather large target for all kinds of trash. I have seen more than a few with aluminum fluff and usually advise the team to clean out the Jag and tape over the connector. It is of course possible for stray conducting material to simulate a dominant condition from a Jag if it gets into the right places.
If I would to have any input on a rebuild on the Jag, I would add a dip switch or jumper to turn on a terminating resistor. The current method leaves a lot to be desired on such a critical item as buss termination. I have seen far more scary teminators than good ones.

Levansic 12-05-2012 16:21

Re: Intermittent connection on field only
 
Quote:

Originally Posted by linuxboy (Post 1168763)
I have seen pictures of CAN splitters (there is a discussion about them on here somewhere), but they need terminators on each branch, and I believe the more branches you have the shorter each cable must be.

Whoa a second. The splitters change the physical topology from a daisy chain to a star. This doesn't change the party line nature, nor the end of run requirement of the terminating resistors. You still need only two 100 ohm resistors. Adding more would degrade the signal.

The resistors are there to separate the potential of CAN high and CAN low signal lines. The signal is carried by the differential of these two lines. Add too many terminators, and the signal voltage bleeds off too quickly. Remember how resistors in parallel work. 100 ohm is lower than the 120 ohm CAN spec. This helps to artificially limit cable length. If you add a terminator on every split out jaguar, your cables would have to be very short.

If you use a splitter for a more robust connection (we did), only terminate at the end of the backbone (as Mikets did) or the other side of the jaguar with the longest cable. We did the latter, because our shooter cable was longer than the rest of our backbone. The other end of the bus will be terminated by the serial adapter or a 2CAN.

-- Len

linuxboy 12-05-2012 23:05

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1168900)
Oliver,
If we go on the assumption that the CAN is echoed through the serial port, it is not hard to imagine that something is confusing the traffic flow there. I can't tell you the priority the Crio takes when handling serial port issues but I would guess it would be one of the more important tasks. A big flaw in the Jaguar in my opinion, is the CAN connectors facing up and having a rather large target for all kinds of trash. I have seen more than a few with aluminum fluff and usually advise the team to clean out the Jag and tape over the connector. It is of course possible for stray conducting material to simulate a dominant condition from a Jag if it gets into the right places.
If I would to have any input on a rebuild on the Jag, I would add a dip switch or jumper to turn on a terminating resistor. The current method leaves a lot to be desired on such a critical item as buss termination. I have seen far more scary teminators than good ones.

CAN is not echoed on the Serial Port, the way the Serial bridge works is the cRIO communicates with the first Jaguar in the chain using the RS232 protocol (so the first one in the chain must be powered to do the conversion). Following Jaguars converse using CAN, but all the data that goes between the cRIO and the first jaguar is in fact transfered using rs232, however, the control flow lines are not connected, only the TX, RX, and GND lines are connected, however there are both TX and RX lines so I imagine flow control isn't too much of a problem.

I agree that the vertical ports can be a pain but I can't think of a better way to do it.

I would love to see a switch for termination, I'm doing some work with DMX and all the parts I've looked at that use that have switches for termination. There are a lot of ways for a external terminator to fail, I'd like to avoid seeing that happen.

Quote:

Originally Posted by Levansic (Post 1168941)
Whoa a second. The splitters change the physical topology from a daisy chain to a star. This doesn't change the party line nature, nor the end of run requirement of the terminating resistors. You still need only two 100 ohm resistors. Adding more would degrade the signal.

The resistors are there to separate the potential of CAN high and CAN low signal lines. The signal is carried by the differential of these two lines. Add too many terminators, and the signal voltage bleeds off too quickly. Remember how resistors in parallel work. 100 ohm is lower than the 120 ohm CAN spec. This helps to artificially limit cable length. If you add a terminator on every split out jaguar, your cables would have to be very short.

If you use a splitter for a more robust connection (we did), only terminate at the end of the backbone (as Mikets did) or the other side of the jaguar with the longest cable. We did the latter, because our shooter cable was longer than the rest of our backbone. The other end of the bus will be terminated by the serial adapter or a 2CAN.

-- Len

Good info, thanks! As you can probably tell I don't know too much about using a CAN backbone, so this was an excellent read.

- Oliver

Al Skierkiewicz 13-05-2012 09:23

Re: Intermittent connection on field only
 
Oliver,
The data is still in a string from CAN to Serial and back again. They don't act independently is what I was getting at. The DMX termination is as much for noise immunity as for correct operation of the serial data. We have DMX running throughout our big studio (85 x 105) using self raising battens. Each batten has DMX running through it to the next batten in line. The DMX terminates in several places around the studio so that we can connect in controllers where needed. We have 92 battens.

DonRotolo 13-05-2012 12:44

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1168719)
Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus.

The terminating resistors are essential to address signal reflections, just like you'd get if you had an unterminated antenna wire.

An open circuit (=high impedance) allows the signal wave to reflect back onto the wires. These reflections can interfere with the actual data. As data rates increase, the interference has a greater impact on the data. Without the terminating resistors, eventually all the echoes would increase the bit error rate beyond acceptable.
Quote:

Originally Posted by Levansic (Post 1168941)
Whoa a second. The splitters change the physical topology from a daisy chain to a star. This doesn't change the party line nature, nor the end of run requirement of the terminating resistors. You still need only two 100 ohm resistors. Adding more would degrade the signal.

The natural topology of CAN is a star, where everyone hears everything. For arbitration to work, each transmitter must be able to hear itself as it transmits, so these are full duplex transceivers. Putting them into a daisy-chain topology is just a poor implementation method.

The resistors need to add up to about 60 Ohms. Two 120's in parallel, four 240's in parallel, the bus doesn't care. In cars, there is a central "star" node which carries a single 60 Ohm resistor in it.

If you go with a class A CAN implementation, which is very slow, you don't even need terminations, but a Class C (e.g., 1 Mb/s) is very sensitive to them.

EricVanWyk 13-05-2012 13:06

Re: Intermittent connection on field only
 
Quote:

Originally Posted by DonRotolo (Post 1169064)
The natural topology of CAN is a star, where everyone hears everything. For arbitration to work, each transmitter must be able to hear itself as it transmits, so these are full duplex transceivers. Putting them into a daisy-chain topology is just a poor implementation method.

The resistors need to add up to about 60 Ohms. Two 120's in parallel, four 240's in parallel, the bus doesn't care. In cars, there is a central "star" node which carries a single 60 Ohm resistor in it.

This doesn't match my understanding of CAN topology or signal lines. Can you provide a link to a description of the star topologies in cars? All of the systems I've used CAN in were daisy chained or vampire tapped.

Al Skierkiewicz 13-05-2012 16:47

Re: Intermittent connection on field only
 
Don,
I was suggesting that the resistors have a dual purpose, both termination and signalling. Ironically, a standard twisted pair cable has a characteristic impedance of about 120 ohms. Assuming a transmission line model, the transmitter then would see two lines (in parallel) terminated in 120 ohms in most cases. There is a famous AES paper by Jim Brown for audio twisted pairs that describes this.

EricVanWyk 13-05-2012 18:27

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1169095)
Don,
I was suggesting that the resistors have a dual purpose, both termination and signalling. Ironically, a standard twisted pair cable has a characteristic impedance of about 120 ohms. Assuming a transmission line model, the transmitter then would see two lines (in parallel) terminated in 120 ohms in most cases. There is a famous AES paper by Jim Brown for audio twisted pairs that describes this.

Not ironic at all! You are exactly correct. The termination resistor needs to match the characteristic impedance of the line.

If there is interest, I could write a small white paper on CAN bus reflections. I think I even have access to the right software for it now that NI has acquired AWR and their excellent simulators.

linuxboy 13-05-2012 19:33

Re: Intermittent connection on field only
 
Quote:

Originally Posted by EricVanWyk (Post 1169112)
Not ironic at all! You are exactly correct. The termination resistor needs to match the characteristic impedance of the line.

If there is interest, I could write a small white paper on CAN bus reflections. I think I even have access to the right software for it now that NI has acquired AWR and their excellent simulators.

I'd love to see a white paper on CAN bus reflections, although I'm sure it would mostly be over my head.

Al Skierkiewicz 14-05-2012 08:38

Re: Intermittent connection on field only
 
Oliver,
What we are discussing are basic transmission lines. Lines do tend to behave in certain ways dependent on the frequency in use and the characteristic of the line. When a line has a characteristic impedance (in this case 120 ohms) and the termination at the end of the line does not match the impedance of the line, some of the energy is reflected from the termination (load) back towards the source (generator). Depending on the system in use, these reflections can add and subtract with the original signal or in this case be interpreted as additional data. For our use, the length of the line is so short that reflections, when they occur, travel on the line at almost the same instant as the generator that produced them.
Where this gets complicated is we are talking complex impedance which is different than resistance (although both use the common term 'ohm' as a unit value). At the frequency we are using for discussion (1 Mb/s or 1 MHz) and the distances we are using on a typical robot, (a few feet at most) much of the complex calculations fall out. Things don't start to get hairy until the line length gets about 1/4 wavelength and above which in our case would be more than 60 feet long.

DonRotolo 18-05-2012 20:59

Re: Intermittent connection on field only
 
Quote:

Originally Posted by Al Skierkiewicz (Post 1169202)
At the frequency we are using for discussion (1 Mb/s or 1 MHz)

Your comments are spot on Al, but don't confuse data rate with frequency, since CAN is a bursty mode. (That means CAN transmitters don't transmit all the time, but in shorter bursts of data, with some quiet time between). Our Class C automotive CAN has a fundamental at about 1.2 MHz (meaning that's the dominant frequency) with sidebands out to 9 MHz (meaning there is enough energy of higher harmonic (multiplied) frequencies to worry about), bringing critical length down to well under 10 feet.

Still not as big a problem for our robots, but you can't ignore reflections even here.

DonRotolo 18-05-2012 21:33

Re: Intermittent connection on field only
 
3 Attachment(s)
Quote:

Originally Posted by EricVanWyk (Post 1169069)
This doesn't match my understanding of CAN topology or signal lines. Can you provide a link to a description of the star topologies in cars? All of the systems I've used CAN in were daisy chained or vampire tapped.

Since CAN is a multicast / multi-master system with arbitration (see here) any network topology where everyone hears everyone is OK.

Vampire-tap is just a star network with a really long hub...

I can't give you any public documentation of how it is implemented in practice, but attached are two pieces of wiring diagram for a certain automobile with which I am very familiar that uses CAN.

In the 'white' image, the "hub" at the bottom (X30/7) is physically about 2 inches long with 10 'taps' for twisted pairs that run out to each node (I have included a small photo). The hub is used only on the low-speed* bus (125 kb/s).

For the high speed bus (500 kb/s) illustrated in the 'black' image, the 'hubs' are actually solder splices(Z37/x), which physically are a bunch of wires with their ends twisted together, crimped with a ring, and soldered.

Some implementations have termination resistors integrated in the hub, but most have them in two control units somewhere on the Bus.

*I call it Low-speed, but it is Class B, therefore "Mid-speed".

Don

Al Skierkiewicz 19-05-2012 18:19

Re: Intermittent connection on field only
 
Don,
Doesn't the cable have a significant roll off at 9 MHz? So even with the sidebands, the cabling attenuates the signals for the sidebands.

Levansic 20-05-2012 13:44

Re: Intermittent connection on field only
 
I wish I knew how to split off threads. The last few pages have been really great for CAN stuff, but don't really match the original topic. Maybe we could re-post some of this in the CAN subforum, so it is easier to find next year. Especially for rookie teams, or others who want to dive into CAN, but have no experience with it.

-- Len

DonRotolo 20-05-2012 13:52

Re: Intermittent connection on field only
 
Actually, we're getting way too deep into CAN for FRC teams here.

On the original topic: FIRST has asked one of our mentors to write a paper describing what we experienced this year, including actions we took and the results. So I have to believe they're doing the best analysis they can of issues reported.


All times are GMT -5. The time now is 17:53.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi