![]() |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
We don't expect teams to solder. If they can great. We don't expect teams to engineer the cRIO. If they can great. We expect teams to follow directions, but what directions can I hand them to follow? Where are the directions and parameters to load test that power supply? What tools do they need? What do the tests in the directions tell, or not tell about the robot? I am not the first to ask these questions. When we ignore that some teams are far more equipped than others. When we ignore that off the competition fields these teams may be more greatly isolated. We create a situation that while I agree with you seems so simple...to them is disaster looking for a place to happen for them. Quote:
Quote:
I was surprised when FIRST said the cRIO couldn't monitor the power supply of the D-Link AP. In effect that means that we can't look while the robots are running on the competition field with hardware FIRST should by now know quite well. Quote:
Quote:
Data collection against issues, assuming proper respect for doing it properly, should not discouraged in any way. In fact, it should be an extremely common thing for FIRST teams beyond just reading the voltage of the battery. I don't feel based on the current rules and based on the current status of things that such collection is widely encouraged. Worse, because of a lack of simple instructions I feel that it's beyond the reach of some people that really need it. |
Re: Intermittent connection on field only
Quote:
I still have to ask what your reference to students on field teams means? |
Re: Intermittent connection on field only
Quote:
I just want to make myself very clear again, that nothing I've written in this topic is intended to devalue your views or your experience. Quote:
I see people all over the place in these topics suggesting that some of us have lost confidence in the field personnel because we speculate and try to solve the problems (funny to be asked not to solve problems in situations where it's the task of the day normally...especially difficult considering the enthusiasm that FIRST inspires in regard to troubleshooting regardless of whom you are or what you know). That somehow we have lost confidence in FIRST's ability to troubleshoot this issue on the field. I have lost no confidence in any field personnel. None at all. Not one field person I've seen recently seemed to me to be intent on making the playing field anything other than fair, circumstances beyond their control may influence the situation otherwise however. It was certainly clear to me on several occasions they were not fully equipped or fully prepared with quick, direct and specific resolutions to a variety of problems. Again there's a limit to what they can bring to the table, having never seen your robot before, having no access to valuable tools, and sometimes not having information about things that could and probably should have been given to them before these events. As my example in this topic, I was expressly assured by field personnel that the field was on 2.4GHz...not 5Ghz. That's a problem that needs to be addressed, it was a mere simple error I'm sure, but it's an error that masks other communications issues. Collecting anecdotal information before events is no real solution to the issue either of getting these field folk and the teams the information they need. Continuous, specific, directed data collection is a good solution. I also brought that up because some people were genuinely thrilled that I had actually bothered to think to bring an oscilloscope to an event as the guy that didn't even know he was competition field spare parts until the evening before. People at that event in management of it made it very clear they intended to recommend that an electronics table, or at least the relevant basic tools were available to teams at events. They were thrilled that I sat there making tails for 2GO PC's with broken Ethernet ports. I even made a bunch of cables for the field and for teams. It was extremely clear to me that some of these teams had no resources to do this for themselves I wonder what would have become of them had I not stepped up. When I wander through forums often reading but not posting to topics and I see people sort of suggesting that field personnel are somehow to deal with this. When I see people suggesting we should be quiet because the field personnel might feel we are being predatory on them. It makes me very unhappy. The field personnel have done all they could, in many cases much more than they should have been able to do... However, they are restrained by the tools, by the quality of their instructions, and most importantly by restrictions on information that need to be removed. I don't find any fault for these people running the fields, in the end FIRST is the central control canopy for all that happens. If the information doesn't get there. If the instructions were not given. If the tools were not there. FIRST needs to do something about it. Not have some people running around topics suggesting that other people are not patient with the field personnel or that the field personnel are the issue. There's no need for people to be silenced, but we can all see the unprecedented and absolute need that this problem...in fact all problems that drop robots dead on the field be meticulously managed and removed. Data collection needs to be done. It needs to be done to analyze this mess at the Championship. It needs to be done within the organization. It needs to be a tool handed to every nook and cranny of the people involved and it does need to involve the people in the community. It needs to be continuous and it shouldn't be disappearing behind the curtains now. Blame is irrelevant and again purely anecdotal. |
Re: Intermittent connection on field only
Out of band interference could also be part of the problem.
I remember at the 2007 DARPA Grand Challenge they had a huge video display setup near the starting line. The three robots closest to the screen couldn’t move until they shut down the giant display. It was about the same distance the Einstein field was from the display in the dome. Another possible culprit is the high concentration of video and other equipment near to the FMS access point. I often see the access point sitting adjacent to a television video monitor, which I would consider as bad as putting the robot radio on top of a CIM motor. -Hugh |
Re: Intermittent connection on field only
Brian,
These students???... Quote:
|
Re: Intermittent connection on field only
Quote:
No specific event was implied. Further, I'm including all the personnel at an event in the crew. Additionally (post 139): http://www.chiefdelphi.com/forums/sh...106042&page=10 "This could have been prevented long before Einstein, but that is a discussion for other threads which we have all been following all season. I saw one FTA in tears afterwards, and I know everyone involved was doing all they could." |
Re: Intermittent connection on field only
Hugh,
The LED wall does give off a fair amount of noise. It is mostly in the form of buzz and wreaks havoc with dynamic microphones. That's why you don't see anyone speaking from behind the scorer's table on the big stage I would suspect. I have some experience with RF mics and wireless "IN EAR" monitors and have used a spectrum analyzer when checking for problems in several shows here. Most are in the UHF band now that the FCC has been involved in unlicensed wireless devices in unused TV channels. Almost everyone is using Wi-Fi to interconnect various audio interfaces and haven't had any issues. I have not seen anything in the bands I am looking at so that leads me to believe that harmonics beyond 700 MHz do not exist. Each little block connects to the one next to it to assemble a large screen. Then you tell the master-controller how many blocks wide and high and if you want it to portion the screen or simply display the entire video signal. There is a CRT monitor on the scorer's table but is an analog device as I remember. There are a variety of LCD monitors as well. Most of that stuff has a fair amount of shielding internal so I don't expect anything in 5 GHz range from any of that stuff. |
Re: Intermittent connection on field only
I've been messing around with things today and yesterday trying to break the robot.
Power and brownouts should be a non-issue. If the teams use the OEM cable, the connection remains consistent and has no drop outs. We also were running with a dead battery. The first thing to give out was the digital sidecar. It resulted in the robot quickly flashing our LEDs on and off as it would brown out the DS,.causing the relays to turn on and off rapidly. The voltage went down to about 4V before I shut off the light show. The cRIO (4-slot) and the Dlink remained powered and connected the whole time with no dropouts. We have also been testing CAN out a bit, and found some unsettling results. If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time. This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this. We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again. So that's another idea that I had as far as troubleshooting. We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk. I think it has to deal with calling CAN devices at different times. We have Periodic Tasks running a shooter control loop every 50ms, and then Teleop is called every 50ms. I think there is a resource conflict and timing issues and we occasionally get a bus lockup for a split second. Going back to PWM, we run at a constant of 3 to 4ms to run the Telop task. |
Re: Intermittent connection on field only
Teleop shouldn't be showing a loop time any faster than 19-20ms, because it's triggered by the DS packets arriving every 20ms.
You've got something else wrong going on with Teleop. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
[quote=RyanN;1168414
We have also been testing CAN out a bit, and found some unsettling results. If you have an intermittent connection where a CAN devices does not appear on the CAN network, the whole CAN network will drop out, and the digital side car will not work. The thing is that the robot remains connected the entire time. [/QUOTE] I imagine the Digital Side Car not working is not a result of the CAN device dropping out, but rather, the error handling when a CAN device drops out is expensive and therefore watchdogs start tripping because loops aren't going fast enough and packets are processed quickly enough and suck. [quote=RyanN;1168414 This was not the cause of our problem as we went back to PWM during Bayou, but I can see teams complaining about no communication due to this. [/QUOTE] Again, expensive error handling means the cRIO stops processing communication packets with the DS, and there for it "loses comms" [quote=RyanN;1168414 We originally had 4 CAN-enabled Jaguars, IDs 2, 3, 4, and 5. ID 5 was the last in the chain, and had the terminator resistor installed. We wanted to test 8 Jaguars over the RS-232 to see how well it worked. Well, we assigned IDs 6, 7, 8, and 9 to the drive motors. When we plugged it all back in, nothing worked. We found out that when we extended the network, that the Jaguar with ID 5 had a bad RJ-45 port. To work temporarily, we rearranged the cables to make that Jaguar the last in line, and it all worked again. So that's another idea that I had as far as troubleshooting. We can't use the 8 jags though... I'm not sure if we reached the limit of RS-232 driving CAN, but we were getting tons of CAN timeouts, and Drive Motors not being called fast enough warning (in teleop). I analyzed the timing of Telop, and found that it ran every 10ms on average, but 60ms to 500ms spikes were not uncommon (where we saw the timeouts). When this occurred, we would also glitch/jerk. [/QUOTE] So, since the last Jag in line's other RJ-12 (RJ-45 is 8p8c I think, whereas the CAN plugs on the jaguars are 6P6C and 6P4C) port was bad, the terminator couldn't terminate, therefore it was like having no terminator. This wouldn't be an issue in a shorter chain since there isn't as much need for a terminator (I don't know why) but in a longer chain, it would cause timeout errors, and again, the error handling is expensive, so it would slow down the loop. |
Re: Intermittent connection on field only
I know that in the C++ case TeleopPeriodic is triggered, by default, on the arrival of a DS packets. This is bad if the network has a lot of jitter or latency issues. You can set the period, but then of course you need to handle the case where you have stale data.
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Oliver,
From what I understand, the resistors are an integral part of CAN signaling in addition to terminating the bus. With an intermittent resistor, CAN devices cannot signal when they want to transmit data to the bus. Has anyone used a CAN splitter to any advantage? |
Re: Intermittent connection on field only
Quote:
Alan, Great analysis. What I'm trying to figure out is how we've advanced, if when we used C, we would be careful about interrupt priorities so that critical functions (i.e. packet transmission) were not delayed. When I use Labview, I run sensors and actuators in separate vi's, not in the main case structure. I know that if I'm not careful, a separate process can eat up resources as well, but doesn't Labview allow you to prioritize specific processes? Could we then not protect comms? Of course, careless programming might have the robot stop because of the causes you outlined. But the comms would stay up. |
Re: Intermittent connection on field only
While all this discussion of CAN is great, I know for a fact it is NOT the problem that 1114/2056 had on Einstein. They both exclusively use Victors.
|
Re: Intermittent connection on field only
Quote:
Quote:
It's just a potential problem that can come up that isn't easily diagnosable. CAN errors shouldn't cause everything to shut down. |
Re: Intermittent connection on field only
Quote:
I have seen pictures of CAN splitters (there is a discussion about them on here somewhere), but they need terminators on each branch, and I believe the more branches you have the shorter each cable must be. In terms of CAN not being something that should shut everything down, when it comes to error handling and timeouts it can, even if it shouldn't. Quote:
|
Re: Intermittent connection on field only
Oliver,
The nature of the signaling on the CAN bus uses switching in the transceivers to cause a device to enter a 'dominant' state for transmit and a 'recessive' state for receive. It is my understanding this is accomplished by driving the bus in a way that causes current flow in the terminating resistors. The CAN terminating resistors also provide the transmission line termination to allow 1 Mb/s traffic on a buss 40 meters long. Microchip has a nice application note, AN228, describing one of their products and the CAN signaling. |
Re: Intermittent connection on field only
Mikets did some interesting work on CAN backboning and posted about it here: http://www.chiefdelphi.com/forums/sh...d.php?t=102108
|
Re: Intermittent connection on field only
Thanks,
I remember that discussion now. |
Re: Intermittent connection on field only
Quote:
- Oliver |
Re: Intermittent connection on field only
Oliver,
Still learning myself. Considering the actual implementation was for cars, I would suspect that the protocol would allow for devices to go off line without killing the whole thing. So I am wondering what actually would cause the effect described above with a loss of a single device. Having seen the internal destruction on some Jaguars, it is always possible that an internal failure could have taken out the transceiver and shut down (shorted) the bus. In that condition, I would guess it is possible for the Crio to keep attempting to transmit data and do nothing else. I do not have any data to suggest that is what happened, just that it is possible. I bet Jim Zondag would know or at least could render an opinion. |
Re: Intermittent connection on field only
http://www.ti.com/lit/ds/symlink/sn65hvd1050.pdf
The application note for the CAN bus transceiver indicates that it has a dominant-time-out circuit that should stop a software error from simply holding the open-collector CAN transmission circuit down to ground and stopping all communication on the CAN bus. (IE the microcontroller crashes and stops doing anything of value and locks up the entire CAN bus.) However this protection circuit has a fault. It only stops the Jaguar from stomping on the CAN bus if the TXD pin from the microcontroller goes to a logic low and stays there. It does not stop the Jaguar from going crazy and simply randomly stomping on the bus by changing the logic state of the TXD pin at bad moments. (IE the microcontroller gets overwhelmed by say interrupts from the encoder and starts running around speaking gibberish on the CAN bus.) A quote from that datasheet: "A dominant-time-out circuit in the SN65HVD1050 prevents the driver from blocking network communication with a hardware or software failure. The time-out circuit is triggered by a falling edge on TXD (pin 1). If no rising edge is seen before the time-out constant of the circuit expires, the driver is disabled. The circuit is then reset by the next rising edge on TXD." If something can overwhelm the Jaguar's internal processing (like the encoder can) and cause it misunderstand it's timing on the CAN bus it could still stomp out the communications of other Jaguars occasionally or even sufficiently to lock the entire bus if the Jaguar's microcontroller can just keep picking the right moments to fail to run silent on the bus. Dropping even a few bits here and there randomly from the protocol should be sufficient to render the expected functions of the entire CAN bus essentially erratic. More advanced CAN solutions eliminate this possibility because they are not merely drivers. They are themselves peripherals designed not to fail in this application. If your processor were to crash while driving a controller like this (and I've done it plenty of times) the CAN bus would continue to operate regardless of that failure. For example: http://ww1.microchip.com/downloads/e...doc/21801d.pdf The CAN solution in the Jaguar per the datasheet: http://www.ti.com/lit/ds/spms047g/spms047g.pdf Does implement a controller, but it's interrupt based. So if the processing unit is overwhelmed it may not realize that it's failed to meet it's timing requirements and then bad news (from page 569): "15.3.14 Bit Timing Configuration Error Considerations Even if minor errors in the configuration of the CAN bit timing do not result in immediate failure, the performance of a CAN network can be reduced significantly. In many cases, the CAN bit synchronization amends a faulty configuration of the CAN bit timing to such a degree that only occasionally an error frame is generated. In the case of arbitration, however, when two or more CAN nodes simultaneously try to transmit a frame, a misplaced sample point may cause one of the transmitters to become error passive. The analysis of such sporadic errors requires a detailed knowledge of the CAN bit synchronization inside a CAN node and of the CAN nodes' interaction on the CAN bus." Also the CAN solution in the Jaguar is adaptive for timing so if you start spewing badly timed bits onto the bus then this can cause you issues (page 559): "Once the CAN module is initialized and the INIT bit in the CANCTL register is cleared, the CAN module synchronizes itself to the CAN bus and starts the message transfer. As each message is received, it goes through the message handler's filtering process, and if it passes through the filter, is stored in the message object specified by the MNUM bit in the CAN IFn Command Request (CANIFnCRQ) register." |
Re: Intermittent connection on field only
This seems like the exact situation that we had. We had one of Jaguars on our shooter motor at the Rutgers competition flood the CAN bus, and since we were/are an all CAN bus robot, we were dead in the water. Why and how it got in that state, and stayed there is not clear. Power cycling the robot did not clear it. We replaced the offending Jaguar, after we replaced the 2CAN because as far as we could tell, the whole bus was dead.
|
Re: Intermittent connection on field only
Quote:
The reason the cRIO causes problems is the error handling is expensive processor time wise especially if the errors are not handled correctly (printing lots of errors to the console and such). |
Re: Intermittent connection on field only
Oliver,
If we go on the assumption that the CAN is echoed through the serial port, it is not hard to imagine that something is confusing the traffic flow there. I can't tell you the priority the Crio takes when handling serial port issues but I would guess it would be one of the more important tasks. A big flaw in the Jaguar in my opinion, is the CAN connectors facing up and having a rather large target for all kinds of trash. I have seen more than a few with aluminum fluff and usually advise the team to clean out the Jag and tape over the connector. It is of course possible for stray conducting material to simulate a dominant condition from a Jag if it gets into the right places. If I would to have any input on a rebuild on the Jag, I would add a dip switch or jumper to turn on a terminating resistor. The current method leaves a lot to be desired on such a critical item as buss termination. I have seen far more scary teminators than good ones. |
Re: Intermittent connection on field only
Quote:
The resistors are there to separate the potential of CAN high and CAN low signal lines. The signal is carried by the differential of these two lines. Add too many terminators, and the signal voltage bleeds off too quickly. Remember how resistors in parallel work. 100 ohm is lower than the 120 ohm CAN spec. This helps to artificially limit cable length. If you add a terminator on every split out jaguar, your cables would have to be very short. If you use a splitter for a more robust connection (we did), only terminate at the end of the backbone (as Mikets did) or the other side of the jaguar with the longest cable. We did the latter, because our shooter cable was longer than the rest of our backbone. The other end of the bus will be terminated by the serial adapter or a 2CAN. -- Len |
Re: Intermittent connection on field only
Quote:
I agree that the vertical ports can be a pain but I can't think of a better way to do it. I would love to see a switch for termination, I'm doing some work with DMX and all the parts I've looked at that use that have switches for termination. There are a lot of ways for a external terminator to fail, I'd like to avoid seeing that happen. Quote:
- Oliver |
Re: Intermittent connection on field only
Oliver,
The data is still in a string from CAN to Serial and back again. They don't act independently is what I was getting at. The DMX termination is as much for noise immunity as for correct operation of the serial data. We have DMX running throughout our big studio (85 x 105) using self raising battens. Each batten has DMX running through it to the next batten in line. The DMX terminates in several places around the studio so that we can connect in controllers where needed. We have 92 battens. |
Re: Intermittent connection on field only
Quote:
An open circuit (=high impedance) allows the signal wave to reflect back onto the wires. These reflections can interfere with the actual data. As data rates increase, the interference has a greater impact on the data. Without the terminating resistors, eventually all the echoes would increase the bit error rate beyond acceptable. Quote:
The resistors need to add up to about 60 Ohms. Two 120's in parallel, four 240's in parallel, the bus doesn't care. In cars, there is a central "star" node which carries a single 60 Ohm resistor in it. If you go with a class A CAN implementation, which is very slow, you don't even need terminations, but a Class C (e.g., 1 Mb/s) is very sensitive to them. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Don,
I was suggesting that the resistors have a dual purpose, both termination and signalling. Ironically, a standard twisted pair cable has a characteristic impedance of about 120 ohms. Assuming a transmission line model, the transmitter then would see two lines (in parallel) terminated in 120 ohms in most cases. There is a famous AES paper by Jim Brown for audio twisted pairs that describes this. |
Re: Intermittent connection on field only
Quote:
If there is interest, I could write a small white paper on CAN bus reflections. I think I even have access to the right software for it now that NI has acquired AWR and their excellent simulators. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Oliver,
What we are discussing are basic transmission lines. Lines do tend to behave in certain ways dependent on the frequency in use and the characteristic of the line. When a line has a characteristic impedance (in this case 120 ohms) and the termination at the end of the line does not match the impedance of the line, some of the energy is reflected from the termination (load) back towards the source (generator). Depending on the system in use, these reflections can add and subtract with the original signal or in this case be interpreted as additional data. For our use, the length of the line is so short that reflections, when they occur, travel on the line at almost the same instant as the generator that produced them. Where this gets complicated is we are talking complex impedance which is different than resistance (although both use the common term 'ohm' as a unit value). At the frequency we are using for discussion (1 Mb/s or 1 MHz) and the distances we are using on a typical robot, (a few feet at most) much of the complex calculations fall out. Things don't start to get hairy until the line length gets about 1/4 wavelength and above which in our case would be more than 60 feet long. |
Re: Intermittent connection on field only
Quote:
Still not as big a problem for our robots, but you can't ignore reflections even here. |
Re: Intermittent connection on field only
3 Attachment(s)
Quote:
Vampire-tap is just a star network with a really long hub... I can't give you any public documentation of how it is implemented in practice, but attached are two pieces of wiring diagram for a certain automobile with which I am very familiar that uses CAN. In the 'white' image, the "hub" at the bottom (X30/7) is physically about 2 inches long with 10 'taps' for twisted pairs that run out to each node (I have included a small photo). The hub is used only on the low-speed* bus (125 kb/s). For the high speed bus (500 kb/s) illustrated in the 'black' image, the 'hubs' are actually solder splices(Z37/x), which physically are a bunch of wires with their ends twisted together, crimped with a ring, and soldered. Some implementations have termination resistors integrated in the hub, but most have them in two control units somewhere on the Bus. *I call it Low-speed, but it is Class B, therefore "Mid-speed". Don |
Re: Intermittent connection on field only
Don,
Doesn't the cable have a significant roll off at 9 MHz? So even with the sidebands, the cabling attenuates the signals for the sidebands. |
Re: Intermittent connection on field only
I wish I knew how to split off threads. The last few pages have been really great for CAN stuff, but don't really match the original topic. Maybe we could re-post some of this in the CAN subforum, so it is easier to find next year. Especially for rookie teams, or others who want to dive into CAN, but have no experience with it.
-- Len |
Re: Intermittent connection on field only
Actually, we're getting way too deep into CAN for FRC teams here.
On the original topic: FIRST has asked one of our mentors to write a paper describing what we experienced this year, including actions we took and the results. So I have to believe they're doing the best analysis they can of issues reported. |
| All times are GMT -5. The time now is 17:53. |
Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi