Intermittent connection on field only

[SteveGPage]

Joe,

This sounds very similar to what was happening to us at Chesapeake. We, too, thought it had to be the FMS, since we were only having the issues when we connected to the field. The FTA said, that in his experience, the type of connection issues we were having were usually due to hardware issues. We replaced several wires, swapped out our cameras, removed the cameras, etc... All approaches I see above. We re-checked every connection, taped down everything that might move, did a very thourough shock and vibe set of tests on the bot in the pits, and then, at least, everything seemed to work in our first quarterfinal, and thought we had found the issue. The next match, we were playing blind again, our cameras had dropped the frames, and the lag times between the camera and the dashboard were killing us. we asked the FTA if the logs showed any connection issues, and he said no. He then said, "You do have, however, a lot of latency showing up. A lot more than I would expect."

We decided to bring the cameras home and run tests with our practice bot to see if we could re-create the issues, but were generally unable to do so. A post here on CD, regarding the 3-13 update had a post from a member of the Killer Bees (33) talking about issues he had found in some of the LabView code, and he posted a sample code he had written that had helped with some of the latency issues they were having.
http://www.chiefdelphi.com/forums/sh...=104580&page=2
It sounded familiar to our problems, so I forwarded the info to our programmers. Other than some of the basic code, our team had written a lot of the code ourselves. When they reviewed the info in this post, they found some of the same issues mentioned. They have been optimizing the code as described, and finding that the code is now a lot more efficient. We are continuing to run tests to see if there are other issues, but are hopeful that when we get to DC, we will not see a repeat of the problems we had in Chesapeake.
In previous years we had build rather "simple" machines, with very little reliance in sensors and complex code, this year, we have two cameras, complex image processing, a complex set of code that keeps track of the state of each ball in the system, etc... I'm wondering if this year's general increase in the complexity of the bots teams are building is bring some of these code issues to light? If you all find anything else that is causing these problems, I would love to hear about it. Thanks!

Steve

[Alan Anderson]

One team at Boilermaker today had the same symptoms described here: inconsistent loss of wireless communication during a match. There didn't seem to be any pattern other than noting that it never happened on the practice field. When I was called in to help troubleshoot, I discovered the 12v-to-5v converter essentially stuck to the bottom of the D-Link. After I suggested that they move it to a less potentially interfering location, they had a string of successful matches all the way through the elimination rounds.

I remember the same problem with the same fix with another team last year.

I'm going to engage in some speculation here. The 12v radio power from the PDB comes from a boosted supply. The supply should have filtering on the output, but if something is faulty it's going to have some ripple in it from the switching. That could be causing electrical noise and RFI from the 5v converter. It might be that an insufficiently clean 5v input makes the D-Link act a bit flaky. Without taking a few hours with an oscilloscope and a collection of parts to do a careful analysis, I think that's the best theory I'm going to come up with.

[jspatz1]

Just to document other occurrences of the problem, we had this identicle trouble at Greater Kansas City in week 1. We tried all the replacements and fixes mentioned in this thread, including filtering the power from the DC adapter. To this day we do not know the problem. Just to make sure it is not the router itself, we have replaced it with a new one. One possibility is physical damage to the router PCB/components caused by shocks/jarring/vibration. This router certainly was not designed for such abuse. Our older, 2011 router which has seen more mileage and abuse exhibits the same problem but more frequently. We have mounted the new router in a location with less shock and more cushioning to try to keep it pristine. Time and another competition will tell if this solves the problem.

[MrForbes]

We had a dead robot one match in San Diego, because of a problem that we were aware of but hadn't fixed yet.

The ethernet connectors on our bridge seem to have a bad design or have been damaged or something. When the cable is plugged in and latched, it can be pulled out against the latch, and lose connection. We have tried several different cables, they all act the same, no connection if not pushed in all the way. We fixed it with some hot glue.

I doubt this is your problem, but it's worth a look.

[Al Skierkiewicz]

Max,
The 12 volt radio output is exactly 12 volts from a boost/buck regulator inside the PD. It continues to work until the battery falls to about 4.5 volts at which time it quits altogether. It does not track the battery.

[McGurky]

Can anyone confirm or deny seeing this problem during practice matches on Thursday?

[drwisley]

Originally Posted by Alan Anderson

One team at Boilermaker today had the same symptoms described here: inconsistent loss of wireless communication during a match. There didn't seem to be any pattern other than noting that it never happened on the practice field. When I was called in to help troubleshoot, I discovered the 12v-to-5v converter essentially stuck to the bottom of the D-Link. After I suggested that they move it to a less potentially interfering location, they had a string of successful matches all the way through the elimination rounds.

I remember the same problem with the same fix with another team last year.

I'm going to engage in some speculation here. The 12v radio power from the PDB comes from a boosted supply. The supply should have filtering on the output, but if something is faulty it's going to have some ripple in it from the switching. That could be causing electrical noise and RFI from the 5v converter. It might be that an insufficiently clean 5v input makes the D-Link act a bit flaky. Without taking a few hours with an oscilloscope and a collection of parts to do a careful analysis, I think that's the best theory I'm going to come up with.

On behalf of Team 1756, thank you Alan.

[PayneTrain]

This may sound silly, but I haven't seen anyone with problems explicitly mentioning communications drops with their router plugged into the regulated port for the router.

Our alliance captain apparently got through two rounds of inspection and every qualification match before their communications started dropping Saturday afternoon. The problem was quickly traced to that wiring problem, and was rectified over a series of timeouts, cool-downs, and gentle, gentle driving.

[Joe Ross]

We ran a total of 5 matches on Saturday with the new power distribution board, without a single problem. After removing the "bad" power supply, we did discover that there was a few pieces of metallic swarf on the outside. It's possible that a piece worked its way inside and caused problems. The mechanical team has been sufficiently reprimanded.

I'm still interested in the explanation of what the PD board could be doing to cause issues with the radio. Like Alan speculated, perhaps some noise that isn't filtered by the power converter.

Originally Posted by Al Skierkiewicz

Max,
The 12 volt radio output is exactly 12 volts from a boost/buck regulator inside the PD. It continues to work until the battery falls to about 4.5 volts at which time it quits altogether. It does not track the battery.

That disagrees with the data sheet for the power distribution board. http://www.usfirst.org/sites/default...on%20Board.pdf It also disagrees with our experience, both the "bad" board and the "good" board output voltage above 12v when the battery is above 12v.

12V/2A boost supply with on-board 2A PTC for over-current protection (typically for powering a WiFi adapter, the boost supply tracks battery voltage when the battery is fully charged and greater than 12V)

Don’t panic if the 12V power supply output is a bit higher than 12V. The supply tracks battery voltage when the battery is fully charged and greater than 12V.

Originally Posted by McGurky

Can anyone confirm or deny seeing this problem during practice matches on Thursday?

We had a problem in one practice match, which was initially attributed to code, but may have actually been this problem.

Originally Posted by PayneTrain

This may sound silly, but I haven't seen anyone with problems explicitly mentioning communications drops with their router plugged into the regulated port for the router.

Our alliance captain apparently got through two rounds of inspection and every qualification match before their communications started dropping Saturday afternoon. The problem was quickly traced to that wiring problem, and was rectified over a series of timeouts, cool-downs, and gentle, gentle driving.

Our radio was properly connected through the 12-5v converter and the boosted power supply. On a side note, I'm amazed that every year I find a few teams that made it through inspection without using the boosted power supply.

[cgmv123]

Originally Posted by Al Skierkiewicz

Max,
The 12 volt radio output is exactly 12 volts from a boost/buck regulator inside the PD. It continues to work until the battery falls to about 4.5 volts at which time it quits altogether. It does not track the battery.

Originally Posted by PDB Documentation

Don’t panic if the 12V power supply output is a bit higher than 12V. The supply tracks battery voltage when the battery is fully charged and greater than 12V.

http://www.usfirst.org/sites/default...on%20Board.pdf

[Steve Warner]

You said you tried running the robot with and without the camera but I'll mention this anyway. At Knoxville we had a similar problem with the robot stopping on the competition field but never having a problem anywhere else. We had very high trip times and a lot of lost packets. We were displaying the camera vidoe on the dashboard using one of the older E09 classmates. It turned out the classmate could not handle the video stream load even though it was only 4 or 5 frames per second. Disabling the video stream brought down the trip times and lost packets and the problem seems to have gone away. Check the stats in the Dashboard Log File Viewer.

[rsisk]

Originally Posted by Al Skierkiewicz

Joe,
The 12 volt output for the radio should be 12 volts +/- 0.1 volts. It should not read the same as the battery. Since you replace the PD already, I am going to guess that the 12 volt supply has a problem. However, none of this explains the action on the field vs all other trials. If the radio resets due to a power dump it takes about 50 seconds. The new FMS dashboard also reports lost packets. Were you told that there were cluster of lost packets on your data stream? They also have the ability to check emissions on your channel, have they looked at that? Normally they only check prior to the event and not during the event. It really sounds like something is interfering with your radio. I know it's not supposed to happen but it obviously is. What is the possibility that your WPA key is corrupt in the kiosk? Have you always configured at the same kiosk or did you try the second one?

I was the CSA that checked the logs. In a couple of cases, there was a flurry of dropped packets right before the robot was disabled. In other cases, there were some lost packets, but nothing unusual. There really was no pattern we could determine in the logs. Sometimes they were disabled for 4 seconds, a couple times for 30 seconds and once for the remainder of the match (> 45 secs).

[Al Skierkiewicz]

Sorry to mislead, I meant the 12 volt supply does not track the battery below 12 volts. The maximum output of the supply according to the sheet will not make more than 13 volts even with a 15 volt input.

[Joe Ross]

For anyone who's interested, I've attached ds log files from our second match on Friday, where we dropped out once prior to the match and twice during the match. It appears that upon reconnecting, the DS creates a new log file, so there are 4 files in the attached zip file.

[dbarr4011]

We have also been getting D-Link drop outs on the field. During our competition at Traverse City and at Western Michigan District we had several communications drop outs. Our robot was not driving hard and no major current draws on the battery. The Field controls personal said our radio reset. I am questioning this answer because I see the radio takes about 1 1/2 minutes to power-up and starts its wireless communications. We loss communications for about 40 seconds. After 40 seconds we are off and running again. My thoughts are we having a wireless network access point/network problem on the field control access point. We saw several other teams with the same problem at these two events.

We have checked the power to the 12V/5V converter. We have replaced the Ethernet cable with a shielded Ethernet cable. We added (4) 470uf 35V capacitors across the 5V power into the D-Link radio. We are running LabView and a camera on our robot network back to out DS. We process the camera image on the DS and send data back to the Robot through a UDP packet.

HELP.... We ordered a new D-Link radio and power converter today but I think there is an issue on the Field Control Access Point.