![]() |
Re: Intermittent connection on field only
Quote:
Others have mentioned a few more communications traps that lie in the software. All of these are applicable and of merit as well (let's not assume that all communications disconnects are D-Link AP power induced issues). Also let us all remember that voltage is just part of DC power, current being the other aspect (obviously time plays a role in AC power which we aren't discussing). Right now voltage is easier for us to tinker with without risking adding trouble that might not be obvious. I did follow up on my question with FIRST and it was decided that for many valid reasons it needs to be in the official Q&A and so that's exactly where we submitted it by Friday: https://frc-qa.usfirst.org/Questions.php (Just click search it's not officially answered yet.) At the moment it appears that the answer to using the cRIO to monitor the D-Link power supply input is not permitted based on what both Al and others have said. Then again there is clearly some dispute on this based on some of the discussions I've had. The use of my voltage monitor sensors could be considered a custom circuit, but the outlook for that being permitted at Championship seems bleak. I remain in discussions with FIRST KOP to get them approved for future application. The entire point of all that discourse was to find a way to accelerate the diagnosis and remedy of tricky power quality issues (some of them might be wiring induced) so that when issues other than power quality are present you can more quicky and clearly narrow in on them. Right now often we don't know what's going on with the power quality to certain accessories when the robots are moving. This is the sort of problem that comes up with Jaguars (because of the added complexity and because they draw the most power while moving, that's why I made those units in the first place) and again with the D-Link AP because of the complexity and issues related to moving. I will abide by whatever FIRST's response is as that is only proper on the competition field. Off the competition field we have much more latitude. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
I guess I meant to ask, if we know the Dlink is restarting does that mean there is a problem somewhere on the robot itself and not on the driver station? I am asking about our practice bot so the FMS is not involved.
|
Re: Intermittent connection on field only
If the DLink is restarting, then yes it's very, very likely a problem on the robot itself.
There may be a rare case where the DLink can be made to reboot by an overwhelming flood of network traffic, but it's not common. |
Re: Intermittent connection on field only
Quote:
Does that describe the symptom of your practice bot issue? Assuming you know your battery is fully charged: If you have a decent DMM put it on voltage min/max and attach it in parallel to the output of the DC/DC converter (even without knowing the exact voltages at which things are a problem) you shouldn't be reading less than 4.5V as a minimum. If you see something lower check your wiring over and make sure you've got the DC/DC converter plugged into the proper power on the PDB not a breaker. If all your connections are correct, then perhaps put the DMM on min/max in parallel with the output of the PDB feeding the DC/DC converter that shouldn't be less than 11.5V (again I'm approximating). If none of that finds the problem try a different D-Link AP (easy but means you need another unit). Then a different DC-DC converter (few wires and again another unit that is a little cheaper than the D-Link AP). Then a different PDB (not so easy and again you need another unit). |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Well another point to make is that if we knew what the nominal current going to the D-Link AP was from the DC-DC converter we could monitor that with the min/max function on a suitable DMM setup as an ammeter and that would actually tell us if the connector on the D-Link AP and even where it meets the PCB inside the D-Link AP is making good connection. If it wasn't making good connection the current flow would be reduced momentarily or longer. Course this requires putting the DMM in series with one side or the other of the circuit between those points so it's intrusive unless you can find a suitable clamp on DC current meter.
(For a real world example of this technique consider the resistor at the end of alarm system contacts. That resistor serves to draw a relatively fixed current from the alarm contact loop. Too little current and either the resistor is the wrong value, or there's a bad connection adding to the circuit resistance or impedance, or a contact is open (alarm). Too much current and either the resistor has had another resistor of near or lower value put in parallel with it (or there's a short), or a contact is closed (alarm). With the general exception of a contact being in alarm the resistor would warn of a tamper with that circuit. Same idea here except we're not looking for security reasons.) I just can't advocate right now that at championship we should measure current to the D-Link AP on the robots while they move. To me that introduces the risk that someone might not realize that the AndyMark breakout module could be a singled ended input. I've seen a ton of data acquistion tools wrecked by this simple error. In order to measure current through a current sense resistor by measuring voltage in parallel with that sense resistor in the path of the circuit to the D-Link AP (then using E=I*R which is Ohm's law); you'd need to make sure the other side of the voltage input across a current sense resistor was robot ground. In theory you could put the current sense resistor on the ground side of the circuit between the DC-DC converter and the D-Link AP, but that assumes the DC-DC converter is also properly connected to robot ground and as Al points out there are some issues with the connector that feeds that DC-DC converter from the PDB (which is where the robot ground for the DC-DC converter comes from). A DMM avoids this issue because the sense resistor is within the DMM and the voltage inputs are differential on that sense resistor within that DMM. My modules avoid this issue because their input is also differential. I am also unclear if a DMM somehow strapped to a competition robot would be reasonable. It would likely have it's own battery and that starts running into odd territory as well. I know a COTS computing device can have it's own battery, but can it connect to the robot like a custom circuit might be able to? Does the additional weight of the DMM (much heavier generally than my little circuits) count against the robot's fielded weight limit (because a COTS computing device does)? I would think so...but again I'm unclear on it. Really these are questions all their own. However if FIRST says no to measuring the voltage to the D-Link AP then in spirit for the sake of this topic there's no reason to bother to ask the additional questions except to set a precident for the future (besides even if we ask this today it's likely that by the time we get an official answer it'll be too late to apply to the championship). |
Re: Intermittent connection on field only
Again,
No valid data would be collected, as the radio RF output changes so does the current input. |
Re: Intermittent connection on field only
Quote:
Sure, there is the chance that the connection could be just loose enough that when the current draw for the radio should be high, the radio can only draw a current that's not lower than the lowest current it might draw normally. However, we can not merely assume that such a complexity exists we can only deal with that issue when we can quantify just exactly how much current maximum and mininum we are talking about. I would suspect since the D-Link's AP primary purpose is to power that radio that there is a relatively finite upper and lower limit to how much current it draws for the RF amplifier once that RF amplifier is in operation. I would think that the RF amplifier would be disabled for a period during resets. Since you need to test with a properly operating robot to find normal limits one could reset the min/max function on a DMM before moving when the D-Link AP is now already connected and operating. Also we can tell when the D-Link AP resets because we loose communication so we can ignore that event if we use the cRIO to check for communications. I realize that the cRIO would have no effect on a stand alone DMM, but as a data collector we could use it that way and with my circuits it can be used that way as well. Therefore we could monitor the current draw during non-reset conditions only and that should greatly reduce the variation between the upper and lower measured currents, thereby making the troubleshooting value much greater. If a connection is loose enough to prevent the D-Link AP from being able to draw the current it would normally draw (intermittently or otherwise) it should be loose enough that it pushes the current draw below the normal minimum at some point, perhaps not immediately, but definitely when it exerts itself as a problem. Otherwise the D-Link AP would always be straving for power and we know that the ratings of the FIRST provided power supply gear say that situation should not happen. Course the FIRST provided power supply parts are voltage regulated so if we measure the voltage at the same time we measure the current and we see the voltage move we should suspect a power supply component malfunction because those parts regulate the voltage and are supposed to accomodate the varying current draw while keeping the voltage where it was designed to be (within reason). I know that this methodology works because I use it troubleshooting components all the time. The key is to characterize the power requirements of the components under normal operating conditions so we know what abnormal operating conditions would look like. If we don't characterize the components then obviously we can't know what we should not see either. So we have 2 needs. We need the characterization in the relevant environments and then we need a good reliable way to catch the exceptions. My little circuits and to a lesser extent the cRIO offer the (possibly legal if FIRST says so) means to catch the exceptions on the moving robot. With proper statistics we could use both methods to characterize the components as well, never mind simple DMM readings (the more readings the more applicable the baselines to all the FIRST robots). The real problem is that to be thorough we need to do this before, between and during events. Right now usually we focus the troubleshooting during the events and that timetable degrades our ability to apply this method by lack of time, resources and quantifiable information. Once we characterize those D-Link AP power supply requirements then it's a simple question of whether the problem system falls within the requirements or outside of them and where. That's the sort of information you can tell by looking at a lit LED. |
Re: Intermittent connection on field only
tech,
What is your name as this just doesn't seem to be right not to address you with a common name? There are so many conditions that are variable with the radio you cannot come up with a 'normal' current draw. i.e. how many LEDs are lit, how often are they flashing, is there a high or low data rate on the ethernet ports, is there a high SWR on either or both of the antennae, is the radio searching for clear channels, how often is the router searching, is the 5 volt convertor at the high end of it's specified output or the low end, what frequency is the internal switching supply actually running at, is the radio deciding to increase output RF power, what is the internal temperature, how often is it resending packets, how many null packets are being sent? Need I go on? In any combination of the above, I would bet that the current demand varies at least 250 ma and maybe as much as 500 ma. With all of these things changing at any one point in time, how can anyone collect valid data? How will you interpret the data you collect? If the radio resets, is the drop in current caused by the reset or the reset by a drop in current? If the current drops is it caused by the PD input, the PD power supply, the 5 volt convertor, bad wiring in any or all of the connections or an intermittent coaxial plug? Or is it a bad battery? how do you know if the software hasn't caused a reset due to something buried in the code? |
Re: Intermittent connection on field only
Although our problem could be something surrounding power supply, we're fairly sure it's not. We've replaced that entire chain, have never actually seen power cycle on the radio, and are now chasing down something in cRio software that could cause the radio to reset as though power was cycled.
Yes, we'll try a ferrite on the power lead, and perhaps a moderate capacitor too (Eric convinced me that Big Honkin' might not be good). To Steve Warner's question: We didn't think so, but now we are not so sure. See MaxMax161's post for a description of what we're seeing now: Always fails when first connected to the field, reboot just prior to the match start and it is always good. :confused: |
Re: Intermittent connection on field only
Quote:
Quote:
More to the point we don't need to do millsecond by millisecond data logging. If the core of the D-Link is powered up it should draw at least a minimum finite amount of power as long as anything at all is running. As long as we catch that minimum anything more than that is adequate to a point (no we don't have intimate knowledge of all the current goings on in the D-Link AP but we know we delivered at least the minimum it should normally require). It wouldn't matter if more than that was 1A more or 500mA more unless that extra current exhausted the energy storage needed to make the DC-DC converter function in which case the voltage regulation would suffer. We just need to know how low that current can go normally. If we were using this to diagnose the specific functions within the D-Link AP then we'd need to worry about making all those extra measurements. I just want to use this to make sure that the D-Link should be getting at least the minimum current it should get to even sit idle. I'm willing to bet that the minimum current it normally draws with the radio powered up is still a pretty noticable amount above nearly zero current, like perhaps 100mA but probably more (and again we're not talking during the initial power-on reset when the radio is likely entirely off). Obviously the number of ports in use will have some effect but it should be towards the maximum, not the minimum. Idealy this sort of characterization testing should be done with a current probe on a high speed oscilloscope suitable for mixed signal analysis. Obviously that's asking a lot unless someone has access to a bunch of robots and the tools. However, if you use even a DMM a bunch of times and different working robots a statistical analysis should reveal a good starting point. Quote:
1. A wrongly characterized minimum (the fix for this is more checking of the minimum, the more the better). 2. The D-Link AP has reset from a software issue (more on this later). 3. The D-Link AP has bad wiring to it possibly including the barrel connector. 4. The D-Link AP is actually defective (bad connector, damaged internals). 5. The current output from the DC-DC converter dropped, but if you're watching the voltage output of the DC-DC converter at the same time you could see if it's voltage regulation related. Quote:
A bad DC-DC converter would have hard time maintaining it's output voltage with the D-Link AP connected and fail to source at least the minimum current. At first you'd see ripple, then you'd see the underlying switching clipped. I could see how a good DC-DC converter could have a hard time maintaining it's output voltage at the maximum current and above...but that's not what we're checking when we look at the minimums. We can of course check for exceeding the maximum current as well which would remove overload as well. The current power supply is supposed to be able to provide more current than the D-Link AP should draw. With that knowledge we can find sloppy power towards the minimum and overloads at the maximum. An overload would show a lower than reasonable output voltage and a higher than normal current. If you really want to push the current draw of the D-Link to a high point stick it in the best approximation of a Faraday cage you can make. It's gonna have to turn the RF amplifier's power all the way up eventually to try to communicate. If you want to really push up the link and the Ethernet switch patch the cables back to the ports (that's going to push the traffic loading to the roof unless they've trapped a loopback in the switch firmware). Do the loopback to all the ports. If that is trapped in the switch firmware. Stick a laptop and another switch in the box. Cross patch the switches and smurf (hacking term) the administration page of the D-Link AP. Quote:
1. A wrongly characterized minimum. 2. Bad wiring or a bad coaxial plug (obviously if the plug is shorted the current will be large and then you'll see the voltage regulation collapse. 3. A bad D-Link AP which is relatively easy to swap. 4. A rebooting D-Link AP which the cRIO would notice and we could log. 5. A rebooting D-Link AP because of a software problem. (There's a trend here, LOL.) Quote:
At the moment I assume that a battery collapse issue shouldn't happen because we are assuming that the FIRST field monitors the battery, but of course, we can monitor the robot battery with the cRIO analog monitor already (communications or not) and we could log it. If you wanted, of course my minimum voltage monitors could be put on the battery as well and trap that regardless of the onboard robot hardware (they can be powered by the robot battery or another battery). Of course a DMM set as a voltmeter with a min/max function would work to some extent as well. Quote:
Again the point was to quickly check the power quality, not dismantal and reverse engineer a D-Link AP in the field. I can speak from experience that FIRST provides several spare D-Link AP in the spare parts and we know the cables are trouble so there should be spares for that as well in there (not so sure, didn't need any at MARS Mount Olive so I did not look). In any case now we form a dividing line in the power quality issue between the D-Link AP and the DC-DC converter. We can draw another between the PDB and the DC-DC converter. We can draw yet another at the battery. With my little modules that's easy and not much weight. Again all of this requires characterization of the components. I thought FIRST did that before they shipped at least the KOP but maybe not. So we can do it now using data collection and all these robots with the obvious additional issue of excluding data from teams that might measure with bad tools or not understand how to do it. Enough statistics with robots that are fairly stable will reveal teams that errored in their measurements or have issues they don't know about. Not much we can do about these measurment issues except confirm measurements at events. I can see how a great number of issues people report could be power quality related. Obviously some are not. Anything that makes the likelihood of a power quality issue being or not being the problem would seem valuable to cut down the troubleshooting and guessing. |
Re: Intermittent connection on field only
Not necessarily corresponding to the recent previous posts of the D-Link, we have finally solved our weird and unique problem with losing connections.
Through the districts and Michigan State tournament, Team RUSH had been experiencing some odd connection issues with the FMS. Not affecting our performance tremendously, we had a large trip time and ms loop going back and forth between the field and the driver station. In addition, we had some lag issue too (although we never knew about it because as drivers, that is what we practiced with so we assumed it was normal). After Hybrid was over, we would experience a 500ms to 1000ms drop time in which we would loose all comms and code but then regain it for the rest of the match. We also had some issues with "flickering" coms during end game at MSC. At first, we assumed it was because we were using 2 cameras and displaying the views on the dashboard. We also were using the classmate as the driver station cpu - big mistake. The classmate with 2 camera feeds did not have nearly the processing power to do this. The switch from hybrid to teleop spiked the cpu% to 100%, therefore momentarily loosing communication. Also, our lost packets were around 40 the entire match. By switching to a different, larger, newer computer, it solved all of our lag problems (controls and video streaming) as well as communication issues. For those teams out there still using the classmate and struggling with communication, I would suggest switching your cpu for your driver station. (based on my experience this year). |
Re: Intermittent connection on field only
I'm going to summarize my post above to make it quick to follow:
Given: 1. Ability to monitor maximum and minimum voltage and current between DC-DC converter and D-Link AP. 2. Characterization of maximum and minimum voltage and current for D-Link AP (and really all FIRST parts should be characterized for at least this). 3. Reasonable characterization of the DC resistance of a properly functional D-Link AP robot power system as measured from the D-Link connector with a resistance meter and the D-Link disconnected. 4. A good reliable connector on the current and voltage monitor that allows swapping the tail that goes to the D-Link AP reliably. 5. Battery voltage monitored to be proper (logged to avoid missed measurements due to communication lost or trapped by voltage monitoring with a DMM or custom circuit). Possible issues: Exceeded high voltage or low voltage Exceeded high current or low current Exceeded nothing and communications issues remain Terminology: Power supply: The delivery method of power from the battery on the robot to the outputs of the DC/DC converter, including all components and wiring in between. Diagnoses requires a different troubleshooting flow. Troubleshooting flow: Symptom: Exceeded high voltage and high current / exceeded high voltage and normal current / exceeded high voltage and low current / exceeded low voltage and normal current / exceeded high voltage and exceeded low voltage during the same with with high current. Probable cause: Regulation issue. Fix: Diagnose power supply. If the proper functioning power supply still has issues replace D-Link AP. Symptoms: Exceeded low voltage and low current / Exceeded high voltage and exceeded low voltage during the same run with low current / Exceeded high voltage and exceeded low voltage during the same run with normal current. Probable cause: Regulation issue, wiring and/or D-Link AP issues. Fix: Diagnose power supply then check wiring. If power supply returns to normal voltage and still have low current, replace D-Link AP and connector tail. Symptoms: Exceeded low voltage and high current / normal voltage and high current Probable causes: Overload due to short or unusually high D-Link AP current usage. Fix: Remove D-Link AP and measure circuit resistance. If as characterized and there's nothing like swarf floating around replace D-Link AP with / without a new connector tail. Symptoms: Normal voltage and low current Probable causes: Bad power connections (outside the D-Link AP back to the DC-DC converter or inside D-Link AP), bad D-Link AP (draws uncharacteristically low current when radio is on), D-Link AP software induced reset. Fix: Check connections between DC-DC converter and D-Link AP. Replace D-Link AP and connector tail. Analyze field behavior with other robots for common issue. If all that fails you probably had a software induced reboot. Symptoms: Normal voltage and normal current but communications problems Probable cause: Not a power quality issue. Fix: Check Ethernet connections. Replace the D-Link AP with/without connector tail. Analyze field behavior with other robots for common issue. If all that fails you probably had a software issue. |
Re: Intermittent connection on field only
Quote:
You should track down Greg or myself next week. |
| All times are GMT -5. The time now is 17:53. |
Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi