![]() |
Re: Intermittent connection on field only
What is the nominal minimum voltage that can be provided by the DC-DC converter connected to the D-Link AP before things start to become a problem?
Keep in mind that if there's ripple on the DC power supply I need the voltage at the lowest peak of that ripple from the robot 'ground'. I know it should be 5V, but does anyone know just how far below 5V you can go before you might start having problems? There must be a window of regulation that is acceptable (for example 4.95V - 5.1V). |
Re: Intermittent connection on field only
You know we haven't correlated the possibility of RF pickup on the power wiring to the DAP yet. I haven't thought to check for that but I have seen many robots using a long power cord simply wound up/folded up and secured with a ty wrap. It is possible that large amounts of RF energy are simply walking in on the power wiring. Maybe what we should do is bring a bunch or ferrite chokes with us to St. Louis and give them a try. We should also not rule out intermittant power connectors putting noise on the power input that actually makes it through to the data circuitry.
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
I'm a little dubious right now of what I can send out there. I have a circuit I made that can be set with a small handheld setup I made, but you don't really need all that stuff if you just use potentiometers and resistors for setting it up. Even with all those bells and whistles the robot mounted part is tiny and light. Basically I'm trying to get a grasp on the range of input voltages it needs to accept. If the range is large then I should try the digital setup I made. If it's small then I can probably trim up a few common settings and use that. If I just stick with working around the 5V supply going into the D-link AP it comes down to just how low can that voltage go before we need to call that a problem. hence my question above. I should think that number is in the tenths or hundreths of 5V. So if we read this as perfectly acceptable then it comes down to actually doing it during a competition. I basically need to move my butt. Otherwise it'll have to be tested off season on a real competition field (I know it works I just never used it in a competition). The good news is that this thing is at it's heart a latching analog comparator. So even if noise on the DC power supply reduces the voltage for a split second this will see it (and if not things will need to get expensive to test because that would mean it exceeds the performance of the integrated op-amps I used). At the moment all this thing does is constantly look for the voltage below the setting and if it happens it lights an LED and keeps it that way. Seems utterly trivial but it can do it on the moving robot and much more completely than a DMM. |
Re: Intermittent connection on field only
Quote:
Quote:
|
Re: Intermittent connection on field only
Quote:
Be aware though, it doesn't log data as much as turn on an LED when the threshold is reached so it has only one equivalent bit of storage. If that bit is on you went below the calibrated voltage. If that bit is off you did not. All the digital attachment does is calibrate the set of digital potentiometers attached to a precision reference. (Sorry I saw only the top part of your post so I'll catch the rest next post.) |
Re: Intermittent connection on field only
Quote:
The problem then is that a digital measurement will always quantize the noise. So a digital data logger always runs the risk that something happens between the samples. Most of the cheap commercial units use SD memory so they are really not much faster at catching transients than DMMs (most actually quite a bit slower). The analog circuit doesn't have a quantized 'blind spot' as much as a response time. I do have an FPGA based oscilloscope I made with a bunch of computer RAM but that's not nearly as small or as light. It has a mode to essentially log data into a bucket to make a digital strip recorder (for a short time beyond that it needs to dump to USB2). It's also not nearly setup as a production test piece. I did specifically design it to be driven around on the robot. With this in mind, then I can probably rig up some small DMMs with Max/Min functionality that you could robot mount till you figure out the limits. Either that or I'm really gonna have to move my butt and try to get that FPGA oscilloscope into something I can send unsupervised. |
Re: Intermittent connection on field only
Another possibility is I have a circuit I use in a production item I worked on that can essentially hold the lowest voltage you present to it till you reset it (within reason).
It's essentially an analog minimum detector. Sort of a minimum voltage sample and hold. If I slap that together stand alone you could put it in the robot and drive around then measure the output with a voltage meter. It would tell you how low it went at any time it was operating. It wouldn't tell you so much how low the D-link AP supply went before it malfunctioned but it could tell you how low that D-Link AP supply got during the match. Actually perhaps with a few of those someone could see what the robots that kept working produced for power to the D-Link AP and what the robots that did not keep working produced for power to the D-Link AP. I think you'd have a similar process anyway with a datalogger would you not? This would be much cheaper and probably collectively faster to quantify the results. I'll dig into my stuff tonight and see what's involved with making them. |
Re: Intermittent connection on field only
This is a very long thread. I did not follow it very closely. So I apologize if I am missing the point. If the goal is to log the voltage data of the Wireless bridge during competition then why can't we do it in software? Assuming the cRIO is still up and running and only wireless bridge is losing communication, can one feed the wireless bridge's power (5V) to an analog channel and write code to log the data? Heck, even the SmartDashboard can be used for that. This year, our team has written a generic datalogger that allows us to do post mortem analysis of anything we want to log. So far, we are using it to tune PID control and evaluate data filter algorithms. But it can be used for anything.
|
Re: Intermittent connection on field only
Quote:
It would require someone to tinker with the code in the cRIO and that would add another task for the cRIO to handle (which might alter the performance of the system in general). So you'd probably need to write that software in each language you might encounter. Not to say it can't be done, but the cRIO still suffers the same issue that you don't know just how often it can check that voltage. What I'm proposing is able to watch the voltage *much* faster and without tinkering with the robot's software. Though I fully admit there are circumstances in which it might not matter. For example if you're saying you measure an analog voltage used in a PID loop feedback it probably won't matter with that because that voltage changes as the consequence of mechanical movement which by comparison to a power transient is very slow. |
Re: Intermittent connection on field only
Since we already have the datalogger (written in a C++ class), if we encounter communication issues in St Louis, we can certainly hook up an analog channel, turn on the logger and see what we get.
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
I won't disagree about the filtering I just want to point out that Team 11 apparently already had a defective DC-DC converter powering the D-Link AP that caused us quite some headaches. That was among the points being tested (it's back a few pages in the topic). Ultimately this possibility was already mentioned before and if you've got it working for you, I certainly won't suggest you shouldn't give it a shot. The more data and eyes the better. If we knew the sorts of power issues that aggravate the D-Link AP's functionality we could detect and prevent them. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
Quote:
If we don't know what the minimum voltage that causes problem with the AP we do know that many teams are not having problems with their AP when they are not on the competition playing field. We can certainly get the output voltages to the D-Link AP from a bunch of robots AP power supplies and get a feel for the deviation from AP power supply to AP power supply between robots and we can do that in the pits. If someone was willing to you could even collect AP power supply ripple data in the pits. The voltage shouldn't really be dramatically going below that measurement. Especially if we all agree that the power supply for the AP should be able to operate down to a battery voltage of around 5V. Anything wrong with this idea? Basically I could set up my little low voltage indicator board to be just below that measurement voltage (to account for some ripple). Could even do a little statistics on the measurements from a bunch of robots and reduce the limit till statistically it would be an outlier if your AP power supply was that low. You might get a few false positives but I would think that would even out really fast. The bigger the set you work with the faster you should be able to weed out minor variations in the systems especially if you start with robots with and without AP power supply issues. Actually come to think of it we could probably collect that data from just about all the teams that were willing to provide that even if they have their robots at home right now and are done for the season. In that case, however, probably best to poll if they had any communications issues that they ever noticed and where. |
Re: Intermittent connection on field only
Quote:
Quote:
|
Re: Intermittent connection on field only
I see nothing wrong with that train of thought techhelpbb.
Regarding the D-Link's immunity to power transients I would agree that it should have appropriate filtering on the input to handle short transients. What we don't know is how it handles any kind of ripple on the power input and what the characteristics are of the regulator's output under various loads. Any noise we are getting from both the PDB and the Regulator is not characterized (publicly) as well as what effect, if any, the length of wire runs from the PDB to the regulator to the AP have. The D-Link's AC-DC power supply is presumably chosen to meet the input power requirements, but are we meeting those under all robot operating circumstances? We know that there is an apparent failure mode of the regulator that causes the D-Link AP to behave anomalously. What I'm unsure of is what failure modes are we introducing in the D-Link over time? The environment we are subjecting the D-Link to is hardly what I assume to be the design environment of sitting on a shelf providing wireless connectivity. Are these the only factors or is the power input also causing a negative effect over time? Are teams that are seeing communications issues using an old D-Link from previous years or have they seen issues with a new regulator and a new D-Link? I'm really interested in performing some bench testing with the D-Link and at a minimum the regulator to help iron out some of these unknown characteristics. I have most of the measurement equipment I'd want here at home, but my one remaining issue is getting a good enough power supply to test various input power levels with differing amounts of noise, etc. I have everything I'd need to run the tests at work, but I'm unsure of my ability to do any of them (on my own time of course). |
Re: Intermittent connection on field only
Quote:
Quote:
But adding a big honkin' capacitor on the power input to the radio should filter out any millisecond transients. I need to verify that the rules permit this. I'm thinking several dozens to a hundred microfarads (whatever's in the junk box) |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
I'm going to put a website to collect measurements from the D-Link AP robot power supplies so we can perform a statiscal analysis. I'll put some direction on the website so people understand what the process to get the measurements should be so we get them to provide us the most practical data when they enter it into the website.
When I'm done I'll put the link in here. This way everyone with a 2011 or 2012 robot can help with the testing. That could net us a few thousand points of data while more evenly distributing the work load. There will obviously be some issues with test instrument calibration but odds are the issues from that will be statistically small. |
Re: Intermittent connection on field only
Tech,
I think we need to slow down here. Most of the problems are not pointing to a power supply problem with the radio. It has been our experience that power supply problems with this radio manifest themselves by total radio reset which by most accounts lasts nearly 50 seconds. With the thousands of PD's in use thus far, only a handful have ever had a real +12 volt power supply issue and of those some were from mishandling of the PD. Those power supplies are very well designed and do more than they were originally intended to do. Remember that this power supply is designed to source a lot more current than the demand from the regulator and radio combined. When the supply faults it doesn't brown out, it just goes away. As to the 5 volt regulator, this is a 25 amp device if memory serves, and it won't brown out either. What ripple might be present on the PD power supply is high frequency and easily shunted by the high frequency impedance of the input cap(s) on the DLink. The fixes I discussed earlier come from the knowledge that teams using the long power wiring that comes with the radio may run it near some high noise sources that are generating some very high frequencies. This noise can and does couple into wiring and in some cases, will actually travel on the outside of the wire and right into the radio bypassing everything you are suggesting. At this point, the really offensive failures are either repetitive (and seemingly regular) losses in packets or complete loss of communications for no other obvious reason. I have seen a few robots where all of the data tell us that the robot is working, there simply isn't any robot commands. Remember that the FMS this year is watching far more things that ever before. They know what the robot battery is doing, what packets are being transferred, when the robot is communicating with the field, even if the robot is doing nothing. If you want to look at power supply issues, you must first rule out, bent contacts in the radio, poor solder jobs on the wiring (if soldered at all), poor wire restraint and poor mounting of the radio. More than half of all radio complaints that are brought to inspectors are obvious problems with the coaxial connector supplying power to the radio. It is either loose, the wrong size, not secured, improperly insulated, the radio connector has pulled away from the board, or there is an improper termination to the 5 volt regulator. In the majority of the remaining issues (when pressed), the team had failed to use the regulator for some period of time. While in many cases the radio continues to function, the power supply has been stressed or damaged. Without doing some physical inspection of these radios, all statistical data becomes flawed. As to the application of devices between the PD and radio, the answer is there is no rule that allows anything in this path when used on the robot. I know that there are people working on the problem and that bench testing will reveal issues if they exist. |
Re: Intermittent connection on field only
Has anybody considered that the in question DAP has a cracked pc board? Cracked boards manifest all kinds of strange behavior. Since the DAPs really were not designed for the robot environment, maybe they are more fragile than anybody thought.
|
Re: Intermittent connection on field only
We have a DC to DC switching power supply feeding a DC to DC switching power supply feeding most likely another switching power supply inside the D-link. I believe that this sets up a strong possibility that under high load a instability develops and a transient of very short duration gets through to the digital parts in the D-link. Any part that is out of spec in the chain would greatly increase the chance of this happening. Probably the only way to catch this is with a very high speed circuit or a scope. Caps are a prime source of this. That's why mother board manufactures boast about Japanese caps in their power supplies. I may be wrong but don't most N stuff have gain scheduling based on background RF levels? Why shout if its quiet. This could explain why the problems are seen on the field. Phily was a very loud rf environment based on the behavior of my I-phone while down on the floor. There are 6 robots all shouting in a loud rf environment. The D-links most likely were at max gain and highest current draw. If the power supplies do turn out to be a point of failure then there are automotive power supplies that are designed to handle the nasty automotive environment. (load dump, static, ect.)
|
Re: Intermittent connection on field only
This has gotten to be a very long thread so I'd like to try and summarize the important points for anyone new who comes along. If any of this is inaccurate, incomplete, or over complete please tell me so I can edit this post.
Thread Summary Main Symptom: On the field, after the auton bell rings and before teleop ends, robots loose communication with the driver station. Main Cause 1: Camera stressing bandwidth. Manifests in many lost packets and high trip times. Very rare when getting a 320x240 image at 30fps and 30 compression. The un-updated smart dashboard defaults to the settings on the camera which default to a 640x480 image, the default dashboard defaults (say that three times fast) to 320x240 settings. Fix: Smartdash- reconfigure the settings on your camera. Stockdash- make sure you don't ask the camera for anything specific unless it's less then the default. Main Cause 2: Power problem to the radio. Manifests by taking around 40s-50s for communication to come back while radio lights turn off and then go solid while rebooting. Most often a problem with the 5v-12v converter or the PDB. Fix: Replace the 5v-12v converter and/or the PDB. Other causes: ? Other Symptom 1: Robot connects to field, robot comm dies, we reboot robot, everything is fine indefinitely. Causes: ? Other Ideas Currently Floating Around: Idea 1: ? |
Re: Intermittent connection on field only
Quote:
If I had the voltage that was too low we could easily test virtually all supply issues up to the radio with those little LED modules I have. Actually come to think of it, I don't know how you can test that voltage at all if you don't know what the requirement actually is. It wouldn't tell you exactly where the problem was (you could further isolate with more of those little modules) but it would eliminate that issue or indicate it's presence just by looking at it. So given how small the effort why not? Certainly there's no reason I can't make the site. To be clear I'm not saying your wrong, I'm just asking why not be sure there's no problem hidden there? As it stands I having been watching the current method of troubleshooting and it is dramatically extended as we hunt through that great big list you offered dancing back and forth between things that effect power quality to the D-Link AP and may be intermittent and things that could additionally be wrong. If we knew we had a power quality issue during the match that's probably half the effort right there and it's entirely possible you could have a power quality issue during the match but not any other place you can measure it during a competition. (I've sent a question about rule R47 and the legality of using a circuit to measure that D-Link AP supply voltage to someone at FIRST. Let's see what they say.) |
Re: Intermittent connection on field only
I just don't want people to get the impression that there is a power supply issue when we are quite sure that this is not the case. The issue may not even be with the radio.
|
Re: Intermittent connection on field only
Quote:
If the voltage is too low it could still be an indicator of a wiring issue not a power supply module issue (a module being the PDB or DC-DC converter). However, first you have to know that there's a problem or not there. It's that ambiguity that lets the imagination roam. I'm not looking to offload blame just find a process that's tangible and quantitative. |
Re: Intermittent connection on field only
Quote:
The power supply issue you (Tech) are tracking, transient undervoltage, has a known fault signature that does not match reported issues. It is possible that a transient spike could affect things, but I'd expect more of a buzz than a spike in this situation. Gdeaver might be on the right track with his "switching cacophony" theory. Please confirm with your own measurements - we'd love to see the numbers. But please, no more back-to-back-to-back posts! |
Re: Intermittent connection on field only
For three years we have been running c++ code build from the default program offered by FIRST. In match 10 at the Michigan championship we moved in hybrid and came to a complete stop with loss of packet from the FMS. The only change was an added camera at low FPS rate. Turns out the CRIO was throwing away the FMS control packet because it was overloaded with the empty code stubs for the continious teleop. This was a totally unexpected consequence of adding a camera that had nothing to do with the control program except the CRIO would see those packets and throw them away. That was enough. After reading this set of symptoms I wonder if the same continous blocks are also in labview not being used and are having the same intermittent effect.
Considering how many years I have been doing programming it was a sobering revelation to see something so simple cause the problem. Every program I help the students write have performance timers in them and according to our data the code was being executed every 20-100millisec but not the TCP packet handler.. When everything has been ruled out look at the impossible.. |
Re: Intermittent connection on field only
Dave,
How is your camera connected? Do you connect it through the crio or direct to the radio? |
Re: Intermittent connection on field only
our cameras were connected to the wireless and not through the crio.
It was only when we added the second camera that the problem really appeared. |
Re: Intermittent connection on field only
Was your 2nd camera connected to your c-rio or did you configure it as something like 10.32.34.12 and also get it through the dashboard?
|
Re: Intermittent connection on field only
This subject has been covered pretty thoroughly but just to make sure I understand: Can a Dlink restart be caused by ANYTHING other than loss of voltage or low voltage at the DC input to the Dlink?
|
Re: Intermittent connection on field only
Quote:
Others have mentioned a few more communications traps that lie in the software. All of these are applicable and of merit as well (let's not assume that all communications disconnects are D-Link AP power induced issues). Also let us all remember that voltage is just part of DC power, current being the other aspect (obviously time plays a role in AC power which we aren't discussing). Right now voltage is easier for us to tinker with without risking adding trouble that might not be obvious. I did follow up on my question with FIRST and it was decided that for many valid reasons it needs to be in the official Q&A and so that's exactly where we submitted it by Friday: https://frc-qa.usfirst.org/Questions.php (Just click search it's not officially answered yet.) At the moment it appears that the answer to using the cRIO to monitor the D-Link power supply input is not permitted based on what both Al and others have said. Then again there is clearly some dispute on this based on some of the discussions I've had. The use of my voltage monitor sensors could be considered a custom circuit, but the outlook for that being permitted at Championship seems bleak. I remain in discussions with FIRST KOP to get them approved for future application. The entire point of all that discourse was to find a way to accelerate the diagnosis and remedy of tricky power quality issues (some of them might be wiring induced) so that when issues other than power quality are present you can more quicky and clearly narrow in on them. Right now often we don't know what's going on with the power quality to certain accessories when the robots are moving. This is the sort of problem that comes up with Jaguars (because of the added complexity and because they draw the most power while moving, that's why I made those units in the first place) and again with the D-Link AP because of the complexity and issues related to moving. I will abide by whatever FIRST's response is as that is only proper on the competition field. Off the competition field we have much more latitude. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
I guess I meant to ask, if we know the Dlink is restarting does that mean there is a problem somewhere on the robot itself and not on the driver station? I am asking about our practice bot so the FMS is not involved.
|
Re: Intermittent connection on field only
If the DLink is restarting, then yes it's very, very likely a problem on the robot itself.
There may be a rare case where the DLink can be made to reboot by an overwhelming flood of network traffic, but it's not common. |
Re: Intermittent connection on field only
Quote:
Does that describe the symptom of your practice bot issue? Assuming you know your battery is fully charged: If you have a decent DMM put it on voltage min/max and attach it in parallel to the output of the DC/DC converter (even without knowing the exact voltages at which things are a problem) you shouldn't be reading less than 4.5V as a minimum. If you see something lower check your wiring over and make sure you've got the DC/DC converter plugged into the proper power on the PDB not a breaker. If all your connections are correct, then perhaps put the DMM on min/max in parallel with the output of the PDB feeding the DC/DC converter that shouldn't be less than 11.5V (again I'm approximating). If none of that finds the problem try a different D-Link AP (easy but means you need another unit). Then a different DC-DC converter (few wires and again another unit that is a little cheaper than the D-Link AP). Then a different PDB (not so easy and again you need another unit). |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Well another point to make is that if we knew what the nominal current going to the D-Link AP was from the DC-DC converter we could monitor that with the min/max function on a suitable DMM setup as an ammeter and that would actually tell us if the connector on the D-Link AP and even where it meets the PCB inside the D-Link AP is making good connection. If it wasn't making good connection the current flow would be reduced momentarily or longer. Course this requires putting the DMM in series with one side or the other of the circuit between those points so it's intrusive unless you can find a suitable clamp on DC current meter.
(For a real world example of this technique consider the resistor at the end of alarm system contacts. That resistor serves to draw a relatively fixed current from the alarm contact loop. Too little current and either the resistor is the wrong value, or there's a bad connection adding to the circuit resistance or impedance, or a contact is open (alarm). Too much current and either the resistor has had another resistor of near or lower value put in parallel with it (or there's a short), or a contact is closed (alarm). With the general exception of a contact being in alarm the resistor would warn of a tamper with that circuit. Same idea here except we're not looking for security reasons.) I just can't advocate right now that at championship we should measure current to the D-Link AP on the robots while they move. To me that introduces the risk that someone might not realize that the AndyMark breakout module could be a singled ended input. I've seen a ton of data acquistion tools wrecked by this simple error. In order to measure current through a current sense resistor by measuring voltage in parallel with that sense resistor in the path of the circuit to the D-Link AP (then using E=I*R which is Ohm's law); you'd need to make sure the other side of the voltage input across a current sense resistor was robot ground. In theory you could put the current sense resistor on the ground side of the circuit between the DC-DC converter and the D-Link AP, but that assumes the DC-DC converter is also properly connected to robot ground and as Al points out there are some issues with the connector that feeds that DC-DC converter from the PDB (which is where the robot ground for the DC-DC converter comes from). A DMM avoids this issue because the sense resistor is within the DMM and the voltage inputs are differential on that sense resistor within that DMM. My modules avoid this issue because their input is also differential. I am also unclear if a DMM somehow strapped to a competition robot would be reasonable. It would likely have it's own battery and that starts running into odd territory as well. I know a COTS computing device can have it's own battery, but can it connect to the robot like a custom circuit might be able to? Does the additional weight of the DMM (much heavier generally than my little circuits) count against the robot's fielded weight limit (because a COTS computing device does)? I would think so...but again I'm unclear on it. Really these are questions all their own. However if FIRST says no to measuring the voltage to the D-Link AP then in spirit for the sake of this topic there's no reason to bother to ask the additional questions except to set a precident for the future (besides even if we ask this today it's likely that by the time we get an official answer it'll be too late to apply to the championship). |
Re: Intermittent connection on field only
Again,
No valid data would be collected, as the radio RF output changes so does the current input. |
Re: Intermittent connection on field only
Quote:
Sure, there is the chance that the connection could be just loose enough that when the current draw for the radio should be high, the radio can only draw a current that's not lower than the lowest current it might draw normally. However, we can not merely assume that such a complexity exists we can only deal with that issue when we can quantify just exactly how much current maximum and mininum we are talking about. I would suspect since the D-Link's AP primary purpose is to power that radio that there is a relatively finite upper and lower limit to how much current it draws for the RF amplifier once that RF amplifier is in operation. I would think that the RF amplifier would be disabled for a period during resets. Since you need to test with a properly operating robot to find normal limits one could reset the min/max function on a DMM before moving when the D-Link AP is now already connected and operating. Also we can tell when the D-Link AP resets because we loose communication so we can ignore that event if we use the cRIO to check for communications. I realize that the cRIO would have no effect on a stand alone DMM, but as a data collector we could use it that way and with my circuits it can be used that way as well. Therefore we could monitor the current draw during non-reset conditions only and that should greatly reduce the variation between the upper and lower measured currents, thereby making the troubleshooting value much greater. If a connection is loose enough to prevent the D-Link AP from being able to draw the current it would normally draw (intermittently or otherwise) it should be loose enough that it pushes the current draw below the normal minimum at some point, perhaps not immediately, but definitely when it exerts itself as a problem. Otherwise the D-Link AP would always be straving for power and we know that the ratings of the FIRST provided power supply gear say that situation should not happen. Course the FIRST provided power supply parts are voltage regulated so if we measure the voltage at the same time we measure the current and we see the voltage move we should suspect a power supply component malfunction because those parts regulate the voltage and are supposed to accomodate the varying current draw while keeping the voltage where it was designed to be (within reason). I know that this methodology works because I use it troubleshooting components all the time. The key is to characterize the power requirements of the components under normal operating conditions so we know what abnormal operating conditions would look like. If we don't characterize the components then obviously we can't know what we should not see either. So we have 2 needs. We need the characterization in the relevant environments and then we need a good reliable way to catch the exceptions. My little circuits and to a lesser extent the cRIO offer the (possibly legal if FIRST says so) means to catch the exceptions on the moving robot. With proper statistics we could use both methods to characterize the components as well, never mind simple DMM readings (the more readings the more applicable the baselines to all the FIRST robots). The real problem is that to be thorough we need to do this before, between and during events. Right now usually we focus the troubleshooting during the events and that timetable degrades our ability to apply this method by lack of time, resources and quantifiable information. Once we characterize those D-Link AP power supply requirements then it's a simple question of whether the problem system falls within the requirements or outside of them and where. That's the sort of information you can tell by looking at a lit LED. |
Re: Intermittent connection on field only
tech,
What is your name as this just doesn't seem to be right not to address you with a common name? There are so many conditions that are variable with the radio you cannot come up with a 'normal' current draw. i.e. how many LEDs are lit, how often are they flashing, is there a high or low data rate on the ethernet ports, is there a high SWR on either or both of the antennae, is the radio searching for clear channels, how often is the router searching, is the 5 volt convertor at the high end of it's specified output or the low end, what frequency is the internal switching supply actually running at, is the radio deciding to increase output RF power, what is the internal temperature, how often is it resending packets, how many null packets are being sent? Need I go on? In any combination of the above, I would bet that the current demand varies at least 250 ma and maybe as much as 500 ma. With all of these things changing at any one point in time, how can anyone collect valid data? How will you interpret the data you collect? If the radio resets, is the drop in current caused by the reset or the reset by a drop in current? If the current drops is it caused by the PD input, the PD power supply, the 5 volt convertor, bad wiring in any or all of the connections or an intermittent coaxial plug? Or is it a bad battery? how do you know if the software hasn't caused a reset due to something buried in the code? |
Re: Intermittent connection on field only
Although our problem could be something surrounding power supply, we're fairly sure it's not. We've replaced that entire chain, have never actually seen power cycle on the radio, and are now chasing down something in cRio software that could cause the radio to reset as though power was cycled.
Yes, we'll try a ferrite on the power lead, and perhaps a moderate capacitor too (Eric convinced me that Big Honkin' might not be good). To Steve Warner's question: We didn't think so, but now we are not so sure. See MaxMax161's post for a description of what we're seeing now: Always fails when first connected to the field, reboot just prior to the match start and it is always good. :confused: |
Re: Intermittent connection on field only
Quote:
Quote:
More to the point we don't need to do millsecond by millisecond data logging. If the core of the D-Link is powered up it should draw at least a minimum finite amount of power as long as anything at all is running. As long as we catch that minimum anything more than that is adequate to a point (no we don't have intimate knowledge of all the current goings on in the D-Link AP but we know we delivered at least the minimum it should normally require). It wouldn't matter if more than that was 1A more or 500mA more unless that extra current exhausted the energy storage needed to make the DC-DC converter function in which case the voltage regulation would suffer. We just need to know how low that current can go normally. If we were using this to diagnose the specific functions within the D-Link AP then we'd need to worry about making all those extra measurements. I just want to use this to make sure that the D-Link should be getting at least the minimum current it should get to even sit idle. I'm willing to bet that the minimum current it normally draws with the radio powered up is still a pretty noticable amount above nearly zero current, like perhaps 100mA but probably more (and again we're not talking during the initial power-on reset when the radio is likely entirely off). Obviously the number of ports in use will have some effect but it should be towards the maximum, not the minimum. Idealy this sort of characterization testing should be done with a current probe on a high speed oscilloscope suitable for mixed signal analysis. Obviously that's asking a lot unless someone has access to a bunch of robots and the tools. However, if you use even a DMM a bunch of times and different working robots a statistical analysis should reveal a good starting point. Quote:
1. A wrongly characterized minimum (the fix for this is more checking of the minimum, the more the better). 2. The D-Link AP has reset from a software issue (more on this later). 3. The D-Link AP has bad wiring to it possibly including the barrel connector. 4. The D-Link AP is actually defective (bad connector, damaged internals). 5. The current output from the DC-DC converter dropped, but if you're watching the voltage output of the DC-DC converter at the same time you could see if it's voltage regulation related. Quote:
A bad DC-DC converter would have hard time maintaining it's output voltage with the D-Link AP connected and fail to source at least the minimum current. At first you'd see ripple, then you'd see the underlying switching clipped. I could see how a good DC-DC converter could have a hard time maintaining it's output voltage at the maximum current and above...but that's not what we're checking when we look at the minimums. We can of course check for exceeding the maximum current as well which would remove overload as well. The current power supply is supposed to be able to provide more current than the D-Link AP should draw. With that knowledge we can find sloppy power towards the minimum and overloads at the maximum. An overload would show a lower than reasonable output voltage and a higher than normal current. If you really want to push the current draw of the D-Link to a high point stick it in the best approximation of a Faraday cage you can make. It's gonna have to turn the RF amplifier's power all the way up eventually to try to communicate. If you want to really push up the link and the Ethernet switch patch the cables back to the ports (that's going to push the traffic loading to the roof unless they've trapped a loopback in the switch firmware). Do the loopback to all the ports. If that is trapped in the switch firmware. Stick a laptop and another switch in the box. Cross patch the switches and smurf (hacking term) the administration page of the D-Link AP. Quote:
1. A wrongly characterized minimum. 2. Bad wiring or a bad coaxial plug (obviously if the plug is shorted the current will be large and then you'll see the voltage regulation collapse. 3. A bad D-Link AP which is relatively easy to swap. 4. A rebooting D-Link AP which the cRIO would notice and we could log. 5. A rebooting D-Link AP because of a software problem. (There's a trend here, LOL.) Quote:
At the moment I assume that a battery collapse issue shouldn't happen because we are assuming that the FIRST field monitors the battery, but of course, we can monitor the robot battery with the cRIO analog monitor already (communications or not) and we could log it. If you wanted, of course my minimum voltage monitors could be put on the battery as well and trap that regardless of the onboard robot hardware (they can be powered by the robot battery or another battery). Of course a DMM set as a voltmeter with a min/max function would work to some extent as well. Quote:
Again the point was to quickly check the power quality, not dismantal and reverse engineer a D-Link AP in the field. I can speak from experience that FIRST provides several spare D-Link AP in the spare parts and we know the cables are trouble so there should be spares for that as well in there (not so sure, didn't need any at MARS Mount Olive so I did not look). In any case now we form a dividing line in the power quality issue between the D-Link AP and the DC-DC converter. We can draw another between the PDB and the DC-DC converter. We can draw yet another at the battery. With my little modules that's easy and not much weight. Again all of this requires characterization of the components. I thought FIRST did that before they shipped at least the KOP but maybe not. So we can do it now using data collection and all these robots with the obvious additional issue of excluding data from teams that might measure with bad tools or not understand how to do it. Enough statistics with robots that are fairly stable will reveal teams that errored in their measurements or have issues they don't know about. Not much we can do about these measurment issues except confirm measurements at events. I can see how a great number of issues people report could be power quality related. Obviously some are not. Anything that makes the likelihood of a power quality issue being or not being the problem would seem valuable to cut down the troubleshooting and guessing. |
Re: Intermittent connection on field only
Not necessarily corresponding to the recent previous posts of the D-Link, we have finally solved our weird and unique problem with losing connections.
Through the districts and Michigan State tournament, Team RUSH had been experiencing some odd connection issues with the FMS. Not affecting our performance tremendously, we had a large trip time and ms loop going back and forth between the field and the driver station. In addition, we had some lag issue too (although we never knew about it because as drivers, that is what we practiced with so we assumed it was normal). After Hybrid was over, we would experience a 500ms to 1000ms drop time in which we would loose all comms and code but then regain it for the rest of the match. We also had some issues with "flickering" coms during end game at MSC. At first, we assumed it was because we were using 2 cameras and displaying the views on the dashboard. We also were using the classmate as the driver station cpu - big mistake. The classmate with 2 camera feeds did not have nearly the processing power to do this. The switch from hybrid to teleop spiked the cpu% to 100%, therefore momentarily loosing communication. Also, our lost packets were around 40 the entire match. By switching to a different, larger, newer computer, it solved all of our lag problems (controls and video streaming) as well as communication issues. For those teams out there still using the classmate and struggling with communication, I would suggest switching your cpu for your driver station. (based on my experience this year). |
Re: Intermittent connection on field only
I'm going to summarize my post above to make it quick to follow:
Given: 1. Ability to monitor maximum and minimum voltage and current between DC-DC converter and D-Link AP. 2. Characterization of maximum and minimum voltage and current for D-Link AP (and really all FIRST parts should be characterized for at least this). 3. Reasonable characterization of the DC resistance of a properly functional D-Link AP robot power system as measured from the D-Link connector with a resistance meter and the D-Link disconnected. 4. A good reliable connector on the current and voltage monitor that allows swapping the tail that goes to the D-Link AP reliably. 5. Battery voltage monitored to be proper (logged to avoid missed measurements due to communication lost or trapped by voltage monitoring with a DMM or custom circuit). Possible issues: Exceeded high voltage or low voltage Exceeded high current or low current Exceeded nothing and communications issues remain Terminology: Power supply: The delivery method of power from the battery on the robot to the outputs of the DC/DC converter, including all components and wiring in between. Diagnoses requires a different troubleshooting flow. Troubleshooting flow: Symptom: Exceeded high voltage and high current / exceeded high voltage and normal current / exceeded high voltage and low current / exceeded low voltage and normal current / exceeded high voltage and exceeded low voltage during the same with with high current. Probable cause: Regulation issue. Fix: Diagnose power supply. If the proper functioning power supply still has issues replace D-Link AP. Symptoms: Exceeded low voltage and low current / Exceeded high voltage and exceeded low voltage during the same run with low current / Exceeded high voltage and exceeded low voltage during the same run with normal current. Probable cause: Regulation issue, wiring and/or D-Link AP issues. Fix: Diagnose power supply then check wiring. If power supply returns to normal voltage and still have low current, replace D-Link AP and connector tail. Symptoms: Exceeded low voltage and high current / normal voltage and high current Probable causes: Overload due to short or unusually high D-Link AP current usage. Fix: Remove D-Link AP and measure circuit resistance. If as characterized and there's nothing like swarf floating around replace D-Link AP with / without a new connector tail. Symptoms: Normal voltage and low current Probable causes: Bad power connections (outside the D-Link AP back to the DC-DC converter or inside D-Link AP), bad D-Link AP (draws uncharacteristically low current when radio is on), D-Link AP software induced reset. Fix: Check connections between DC-DC converter and D-Link AP. Replace D-Link AP and connector tail. Analyze field behavior with other robots for common issue. If all that fails you probably had a software induced reboot. Symptoms: Normal voltage and normal current but communications problems Probable cause: Not a power quality issue. Fix: Check Ethernet connections. Replace the D-Link AP with/without connector tail. Analyze field behavior with other robots for common issue. If all that fails you probably had a software issue. |
Re: Intermittent connection on field only
Quote:
You should track down Greg or myself next week. |
Re: Intermittent connection on field only
Eric-
Since you know the design of the FRC control systems components extremely well, is it fair to say that the current PDB 12V -> 5V DC-DC Regulator -> D-Link AP is well characterized (publicly or privately) such that all the components are appropriately spec'd when it comes to power ignoring failed components, bad wiring, etc? I'd hate for people (myself included) to go down the path of characterizing something that has already been done. I assume the homework has been done but I know things have changed slightly in the control system setup since it was originally designed. |
Re: Intermittent connection on field only
We tried the idea of putting a volt meter on the output of the DC-DC converter to catch the minimum voltage. With the meter connected we could not reproduce the problem of the Dlink restarting. We were also using a different driver station PC. Could the meter have smoothed out any voltage spikes that might have been present? What size ferrite and/or capacitor should be used to test this? Would they be legal?
|
Re: Intermittent connection on field only
Quote:
I see no reason why ferrite beads would be illegal since they are passive and don't physically connect to the actual conductors in the wire. Appropriately sizing them would depend on the noise you are looking to reject, etc and I couldn't provide a recommendation at the moment. Capacitors on the other hand may be illegal in the current rules. Unless you've had power related issues with the D-Link I would carefully consider adding anything/changing anything at this point. |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
I would expect a bad DC-DC converter pumping out switching frequency instead of DC would drive the DC reading of a DMM into strange territory. A low frequency signal like 60Hz AC usually under reads on the positive side and you might see the negative side occasionally (quantification by the DMM's DC voltmeter sampling), but I seriously doubt the internal switching frequency of the DC-DC converter would be lower than 1kHz and might be much higher. Also there's the chance that as the DC-DC converter reaches beyond it's ability to pump energy into it's output storage because of feedback internally it'll allow it's regulation to creap up because it'll keep saturating it's switching MOSFET for extra cycles to feed the energy storage. Depending on what safety systems are in the DC-DC converter you might even see some slow peek modulation imposed on the output as it tries to restrain it's output voltage and then gives up a bit and tries again in a loop That would have the effect of bringing the output ever closer to the input (so you'd see peaks between 12V and 5V...like 7V). However, if you looked with an oscilloscope you'd notice that it's not 7VDC, it's really more an AC small signal. As the D-Link's input capacitance is already within the circuit it'll not easily be able to impact such a wild input. In probability, however, the D-Link AP will have at least a standard linear regulator in there with some capacitance beyond it and that'll provide at the minimum a few dB of damping on the AC improperly feeding it. I would except misbehavior...perhaps due to complexity it might not be apparent but I bet a really thorough review would find it. There's also the possibility that there might be a measurement difference from a cheap DMM and a true RMS DMM even though true RMS DMM usually impact the AC performance of the test equipment not the DC (I've seen some really cheap DMMs in my time). I'm still chasing down the DC-DC converter Team 11 found damaged on their robot and when I find it, I'll put an oscilloscope on it. I'm not sure, however, that the voltage it was producing was too high on a DMM. So if someone does encounter a unit that produces 7V according to a DMM, please put an oscilloscope on it and describe the output signal for us. If I'm correct and the peeks are merely 7V the effect of the low pass filter by the D-Link's internal regulation (which would probably be low-drop out) would be to drop that 7V (not really DC) by a few dB. In effect it'll be like pumping more like 5VDC or so into the input instead a lot of the time. Occasionally it might peek over but probably not all that often which would explain why it keeps running at all...of course doing this is still not a great idea and you're probably right eventually either the DC-DC converter might give it a hard shot of 9V-12V or the regulation within the D-Link AP will fail (if it's linear the result would be thermal damage). As that applies to your quote of my earlier post, since I suspect that the DC-DC converter's output would then be AC small signal and not DC under those circumstances I feel I stand by my presented idea. The best voltage and current monitoring solution for the robot would approximate an oscilloscope not a DMM DC voltmeter. Something close to an analog oscilloscope like a voltage comparator would see the instantaneous low current and low voltage. It would not effectively take snapshots a few thousand times a second like a DMM would. Since the input capacitance essentially forms part of the output energy storage of the DC-DC converter we could see currents and voltages that are during the length of a time frame exactly as I've stated. Also, we could see them during the length of the waveform as too high in voltage and still too low in current which if you look back is still a regulation failure symptom. I will conceed, however, that if you put a circuit with an LED for exceeding the voltage maximum and an LED for exceeding the voltage miminum fed by an analog comparator you could see both voltage monitoring LED lit and a current that's too low with a failure mode like this. In effect the comparator would be fast enough to at one moment see one fault and then shortly later see the other. Latching the results would produce both lit. I'll add that mode to my cheat sheet on the page before but there's already a course of troubleshooting for it starting with diagnosing the robot power supply up to that monitoring. In any event, it doesn't negate the value of at least attempting to investigate the power quality feeding the D-Link AP, it just adds another possible observation you might encounter. |
Re: Intermittent connection on field only
Quote:
Quote:
|
Re: Intermittent connection on field only
Quote:
Now all we have is speculation and swapping parts until we remedy the initial engineering process oversight (and that's all we can have given the missing quantifiable information). Given I got the question into the FIRST Q&A forum I'm well on my way to removing the impediment to correcting that oversight. Also I've read every post in this topic so that point is a straw man argument http://en.wikipedia.org/wiki/Straw_man. Nevermind it ignores that my team did have this problem and it cost us 3 matches chasing phantoms with the help of field personnel while not being able to really do much on the field (and finally after chasing software and other unrelated issues they replaced the bad DC-DC converter they found in part because of my suggestion). Nevermind that reading Al's post does not tell us whether the DC/DC converter was loaded or not or what tool was used. Must have set foot in a political forum by mistake. Sorry. Thanks for letting me know what my time is worth. Here I was thinking we were trying to be thorough. |
Re: Intermittent connection on field only
Quote:
Quote:
We value your input and your time - we need to make effective use of everyone's time. To do that, I'll pull from one of my mentor's phrasebook: Be Brief, Be Brilliant. |
Re: Intermittent connection on field only
Quote:
Quote:
The question above remains the same. No straw man argument can distract from it. Do we have access to the quantified data I seek about the D-Link AP or must we as community engage in a process to get that data? I was fully prepared to start community data collection at my financial cost before it was suggested I 'slow down'. All the other issues from the software and otherwise are valid. They deserve recognition but the format of this forum is not within my ability to control. If you're concerned that I'm burying these additional topics can someone (because I am unable to) provide a topic split? |
Re: Intermittent connection on field only
Quote:
The 12->12 is very well known, the 12->V is well specified, but the D-Link's input stage doesn't have the same level of documentation available. We've cracked them open and V&V-ed from the inside, but unfortunately reverse engineering can never create a data-sheet: We can only say that all the samples we tested worked well with good margin. It may be worth while to take an oscilloscope to a working and a non-working setup. The frustrating part is that we haven't been able to reliably reproduce the issue in a lab, so we have to drag a subset of our equipment to competitions. |
Re: Intermittent connection on field only
Quote:
It might be true that you can't necessarily reverse engineer every aspect of the original design expectations. It's often the case that the actual product behaves sufficiently to serve the original design priorities but parts of it perform outside of the original expectations. The point being that we don't hand a customer the design expectations and tell them this must be what they got because the product seems to meet the critical design priorities. We test and hand data backed by real measurements to the customer (hundreds of different measurements). Sometimes we don't even perform enough tests so we do some more because there's a problem with the RF modules after the fact. Often involving the amount of power these RF modules can consume when added to other circuits which toggle their states of operation (these are usually cell phone RF circuits and WiFi chipsets). All veracity of all these test results requires effectively confirmation by quantity. The more of the things you test the more likely you are to notice the tolerances. That's where statistics come in. None the less, the problem is that now that I've slowed down to allow other ideas to pass I've lost time to collect data relevant to the concerns I expressed from a wider selection of working and non-working robots. It's not likely that by middle of this week a whole lot of teams will take the time to perform a bunch of measurements to add to the measurements we can get at competition. In a real way the delay has removed a key validation aspect from the characterization process by time limiting the testing and the quantity at the Championship event. Between the fact that I need a characterization to use my modules, and the fact that FIRST has yet to even answer the question about the legality of the modules for this year in their forum. There's little sense sending these items to the Championship. I grow increasingly concerned that the necessary characterization parameters will not be sufficiently determined as a result of this contrived delay's impact and therefore my circuit will produce dubious results by misapplication. It greatly bothers me that FIRST might have tested an unspecified number of D-Link APs as samples from unspecified lots from D-Link and merely tested for the margin from overloading the DC-DC converter on the bench. It has partially blinded us at the troubleshooting phase and now I have basically run out of time to remedy this using the community and we are left merely with the same troubleshooting process that left us pecking and hunting for 3 matches while stuck like a brick on the field. I will be asking FIRST to approve my units for testing. It seems (contrary to forum commentary I read when I proposed making these very modules for the Jaguar in the first place) that FIRST does not in fact have suitable test equipment to test on any arbitrary moving robot. That's what these circuits I made addressed as a primary design attribute and I want them to get a fair evaluation in the environment of quantified results that they deserve. In the mean time, the good news is that if FIRST does answer my question in the official FIRST Q&A forum then they'll open the door to allowing the cRIO to act as a datalogger as well (at least for voltage) while connected to the D-Link AP's power connection. This means that it will make it entirely clear whether or not Mikets's Team's C++ class can be used on a fielded competition robot (see page 8 of this topic). So while that's not the most comprehensive solution to my concerns it's something that we can at least do for robots that run C++ as he has offered it (in theory all FIRST robots can do this with their software but you'll need to figure out how to do it for each language in each robot). Luckily as a datalogger the actually characterized component information won't be to critical. Too bad we have no idea how fast the cRIO can actually sample that voltage and my question to FIRST doesn't include sampling current (my reasons for that decision already stated). |
Re: Intermittent connection on field only
OK Brian,
Let me repeat in a different way. Modeling the current drawn by the Dlink is a waste of time for the reasons I already outlined. When I spoke of the variable current demands I was not thinking in terms of msec changes but in terms of average current drawn. The reality is that that type of data you want to collect in a real world test, would give you absolutely nothing useful. The average current delivered will (I guarantee) change with with variables that are not under your control. (and there is no way for many people to control those variables outside of the manufacturer) The more variable of these is input RF signal processing dependent on the number of active channels, output power, SWR, modulation usage, packet repeats, and data transfer. Do you have an RF proof anechoic chamber at your disposal? The radio link does not fail with changes in current except when the wiring has high series resistance. I am glad that a suspect 5 volt convertor turned out to be the issue. However, let's face facts, it is far easier to replace a suspect device with a know failure mode than attempt to shoot holes in an otherwise working system. To remind those who are lurking. 1. A failure of the 12 volt PD power supply occurs when the battery supply falls to 4.5 volts or less. With poor mechanical design and/or electrical primary wiring, this is a real world possibility. The result is full radio reboot of 50 sec, or a full Crio reboot of more than 30 sec or both. this supply is capable of three times the current of the Dlink. 2. The five volt supply will fail when the input voltage is less than 7 volts. It is designed to regulate at least 20 times the current demand of the Dlink. 3. The permanent failure mode of the 5 volt regulator makes it look like a big resistor. The output is about 7 volts with no movement on the robot and it tracks the input when it is pulled down under varying load. There is no strict data on this failure as it is simply the series resistance of molten silicon in an individual device. 4. It is well documented that there are a few things that can take the data flow down. This is not an FMS disconnect as many people report. It is either a maximum CPU usage in the Crio, the DS computer, or an overload due to high data throughput experienced when the camera(s) is used at high resolution. 5. Let us not forget that there are certain safety protocols in use to insure that when the field is no longer capable of passing control data (i.e. robot enable) or the match is stopped, all robot function must also stop. This takes priority over everything else. Of the odd failure modes, many are attributable to mishandling by the team. Number one of these is wiring the radio backward, followed by wiring the 5 volt convertor backwards. The next most common is the team who failed to wire the 5 volt convertor at all. While the radio may work for a while with no regulator, it will start to fail in unusual ways even when the regulator is correctly wired in place. As to Crio related failures, these are very common and many are well documented. Please search for things like "feeding the watchdog", odd behavior with certain LabView implementations that are marked as unused. finally check to see if your Crio is logging activity and how often this is occurring. High loop times is good indicator that something is wrong in this area. If you suspect something in the radio circuitry, first watch the RSL to see what is happening on the robot. If the Crio reboots, you will see it there. If you robot always fails with a particular robot move, watch the lights on the PD (this is why you should mount where we can see it) and see if they go dim or go out. Watch the LEDs on the Dlink to see if they go out or if they go dim. The robot will talk to you but you have to listen. Finally, check with the FTA when failure occurs on the field. They are logging things like that battery voltage, lost packets and field connection. I have had only one unexplained failure in the last three years that I have seen and couldn't explain. However, that was when we didn't have CSAs checking to see if teams have shot themselves in the foot. All the people I respect on this forum tell me that the power supply to the radio cannot be the problem and my gut tells me the same thing. As to what ferrite beads may work in keeping RF out of the radio when used on the power wiring or ethernet connections. I know that there are a couple of snap on chokes available from Radio Shack that could work. Others that I know of are available from Ham Radio Internet sites but is unlikely at this late date you get them in time. |
Re: Intermittent connection on field only
Quote:
Quote:
Quote:
The inside of a microwave oven is an RF shielded box (with some nice leaks around the door). It'll reflect and absorb 2.4GHz well on it's way up to 5GHz. The metalized plastic stirrer fan would do what it does to the microwaves from the magnetron and form an interference pattern. You don't need that sort of test equipment to characterize the current input on the DC power supply. You just need to max out the radio current draw. You'd need a anechoic chamber if you wished to reverse engineer the details of the radio to specifics and I've never asked for that at all. Quote:
Quote:
As I feel I'm being effectively bullied on this subject in a way that can't produce anything of value for FIRST. Since I've publicly asked for the topic to be split (and I can't even start a topic myself as I've pointed out) to remove the cover distraction that I'm interfering in the discussion and my request has been ignored. Since it appears that the parties involved with the most access don't have the data I require. Since I have now been stalled to prevent my use of the community to get that data. Since some seem intent on ignoring my posts while complaining they are too long. I'm going to just drop the subject which was apparently the mission statement of the day here. That doesn't really serve the engineering or scientific interest of FIRST but it does serve to let me get some of the information I wanted without any further opportunities for interference. If FIRST allows the use of the cRIO to log that data on the Championship field I hope someone actually does it. You know the sort of funny part is that it doesn't matter if FIRST uses my little comparator modules or not officially. They have so many applications that getting money to develop them seems no issue at all. I wonder why there's so much interest in an idea people claim can't be applied. |
Re: Intermittent connection on field only
Brian,
You already posted that high frequency changes in current would not be visible due to all the low frequency filters in circuit. I agree. You won't see them so that gives you no data as I pointed out above. There are features that you cannot control and you cannot predict so again there is no way to collect meaningful current data. Microwave oven? It is not an RF proof chamber it is simply a cavity with an opening on one side. Most ovens use a 1/4 wave shorted stub to prevent leakage. And you will not be able to test real world activity since you won't have another radio that is connected to the wireless. The SWR that the Dlink will experience in this environment is likely to be extreme and changeable with a small movement within the cavity. Above all, I don't want it to seem that I am bullying you. I do have a vested interest in preventing the others who read this discussion to be distracted from collecting meaningful in situ data and understanding the real problems when they exist. If there is a real problem I (and FIRST) need more than the sampled data of the current draw on the radio collected in a sea of variables. We (Inspectors, First Engineering, National Instruments and CSAs) require hard data. i.e. did the radio stop transmitting, did the Crio reboot, did the radio reboot, did the CPU hit max usage, did the battery fall below 4.5 volts, what language is the team using to program the radio, can we see the code? Collecting this data will point us to a particular component failure or to a software issue that hopefully will be repeatable under real conditions. |
Re: Intermittent connection on field only
Quote:
Quote:
Quote:
Quote:
Quote:
Quote:
|
Re: Intermittent connection on field only
Well let me add some more hay :) And this thread seems like a good place to report some information.
I've been concerned about the field network for the past few years because it *seemed* that *which* robots were on the field was a contributing factor to the overall responsiveness, jitter and lag of the network. For example, the number of robots on the field that had onboard ip cameras seemed to be important. So this year, after our first district event, I started asking questions, and then at the MAR championship I brought a hardware packet sniffer to take a look at the field network, and I feel I should share what I discovered. I originally believed that each robot was going to be assigned a separate channel and that they were using 802.11n @ 5 Ghz. The network at the MAR championship was indeed using 802.11n @ 5Ghz, and they are using a wide channel, however all robots were sharing the same channel and as such were sharing a total theoretically bandwidth of 300 Mbits. At the Mt. Olive competition, I was told that the robots were running on channel 6. If true, this would have meant that they were running in the 2.4 Ghz range, with over a dozen other 802.11g networks, this would have considerably reduced the theoretical bandwidth. In general, due to a number of different factors, if you get half the theoretically bandwidth on a wireless network, you're doing well. So let's assume that 150 Mbits is our expected available bandwidth on the field, if you're using a wide 802.11n channel @ 5 Ghz. Much less if you're using 802.11n @ 2.4 Ghz as the interference will be awful. I've asked but haven't heard whether or not they have established any QoS or bandwidth limitations on each SSID in the Cisco access point that they are using. Without any controls, it will be a free-for-all for the available bandwidth. This bandwidth is used by the robots in a number of different ways, and here I’m just talking about communication between the robot and the driver station laptop, as I’m not sure what the FMS is using: VIDEO streaming If you have an onboard camera, and send a video stream from robot to driver station at 640x480 in 24 bit color @ 30 frames a second, that's ~200 Mbits uncompressed raw bits. MotionJPEG will compress that down on average to around 10 - 15 Mbits. H.264 will do better. I've heard that some teams have 2 cameras onboard. The Axis camera supports a number of options to reduce the bits but there is no rule about how you configure the cameras, and they would work fine in your local build environment and even in the practice fields at competitions where you’re the only user of that wireless channel. Dashboard data There is the "normal" dashboard that is part of the default code, and the default dashboard sends data, if I remember correctly, at about 10 times a second. For reasons that I can't remember at this point, we are actually sending the data at 40 times a second from our robot. This is a relatively small amount of data, but it doesn't have to be. With the addition of Smart Dashboards, and other custom dashboards, and no guiding principle on the volume of data this could be a significant amount of data or just a dribble. In our case, we're sending ~1 Kbits per update, or 40 Kbits per second. Driver station data This is data packaged by the driver station application provided by FIRST and sends the values for the input devices attached to your driver station to the robot. I've never looked into how much data is being sent, or the frequency with which it's sent but it's not a lot of data probably on the order of 40 - 50 Kbits per second. Other network traffic There are several network ports that are open for teams to use for communication between the robot and the driver station. In our case, we ran a UDP server on our robot to collect the results of vision processing performed by our driver station. We sent the results of the calculations back to the robot at the rate of 10 times a second. The data is small (72 bits) so we're sending only 720 bits per second. So for us, our network utilizations was small ~1 Mbits for the camera - we're using gray scale, 320x240 and 30 frames a second with MotionJPEG compression, and at best another 1 Mbit for the remaining traffic. But that is due to choices that we made. I could easily imagine making other choices, and given that I was operating under the belief that we had a full channel to ourselves, I might have gone down totally different path. The thing about this is that while there is the catastrophic failure mode where the field network crashes. There are many other situations where the latency and jitter can spike, and dip badly. VxWork’s IP stack is not particularly robust, and for some teams that stack has to handle all of the time sensitive CAN bus traffic, as well as driver station, dashboard and custom traffic. Further, unless you change your default Iterative robot code (at least in C++), you're periodic functions are synchronized with the arrival rate of the packets from the driver station. Now it a well behaved network, the arrival rate should be pretty stable. But if your code assumes stable packet arrivals, you can run into all sorts of timing issues. In addition, both the camera traffic, and the driver station packets are using TCP which can be very unfair when it comes to sharing bandwidth. A greedy application can ramp up its utilization of the bandwidth, causing starvation of others. And then there's retransmissions, etc. Is it possible to saturate the network? You betcha. Is it service impacting? Yes to everyone, including you. Is there anything that can be done? Yes. When I examined the network at the MAR championship, I saw a number teams that were having problems associating with the field. There were repeated attempts by the robot's DLINK to associate with the field access point. I also saw many corrupt frames. Our DLINK in Mt. Olive simply gave up completely, rebooting during several of our matches. It had been fine, and then just started reboot when we hit another robot or field element Of course to our drive team it looked like we lost communication to the field (which we did) but it was the DLINK that was rebooting. And no there weren’t any loose wires except maybe inside the DLINK housing. I heard from a team that they had a DLINK that only worked when it was standing on edge. Lay it flat and it didn't work. I think they are cheaply made, and are really not meant for the hostile environment of a FIRST robotics competition. A ruggedized access point/bridge would be a beautiful thing. I don't know why FIRST chose not to have 6 separate access points to provide a channel for each robot. Maybe they just figured that 150 Mbits/6 = 25, and who’d need more than that. I don't know if they are configuring QoS to ensure a fair share of the network. I will be looking at the network in our next off season competition, and try to come to some conclusion about what exactly is really going on. |
Re: Intermittent connection on field only
Quote:
Quote:
|
Re: Intermittent connection on field only
I mentioned about the networks at MAR Mount Olive district event appearing all to be on channel 6 (course I couldn't look at the 5GHz networks with my phone) back on Page 3. RufflesRidge assured me they were not shortly after.
I never got an answer to my question about whether the 2.4GHz sections of the D-Link are turned off when the fields configure the D-Link APs (post #48 on page 4). If they are turned off, then in theory the channel overlap in 2.4GHz should not be a problem. However, if the second band remains on the radio will attempt to interact with the 2.4GHz networks as well as the 5GHz+ networks. I got confirmation of my information from people who helped set up the field at the MARs Mount Olive District, not sure where RufflesRidge got his information. |
Re: Intermittent connection on field only
I had a separate source who assured me that at Mt Olive, they were on channel 6. I did not check to ensure whether that was true or not. But to me having no controls on the apportioning of bandwidth and having robots sharing bandwidth means that the idea that which robots are on the field can impact the communications between the robots and their driver stations is possible. And steps could be taken to mitigate these issues so that there's a level playing field. ;)
|
Re: Intermittent connection on field only
I wrote a several page document on how to address what I suspect is Al's concern regarding my previous troubleshooting flow described on page 10, post 149 of this topic.
This is about checking the power quality to the D-Link AP. I suspect Al is concerned about a 'blind spot' (see my post #144, page 10) in my troubleshooting flow which I can resolve with: A. At least 1 troubleshooting D-Link APs provided in spare parts at a competition with the barrel connectors removed and tail of stranded wire soldered to the PCB. That wire should have a good mechanical strain relief and come with a ferrite RF choke to remove noise. This has been asked for at past events and in past years as well. B. At least 1 low temperature coefficient resistive load (physically large resistor probably with fan/heatsink) that could be provided in spare parts at a competition with a maximum and minimum current monitor in series with it. It's value would draw the same current as the D-Link AP can draw maximum at all times. The current monitor limits are set using Ohm's Law (E = I x R) and the maximum and minimum voltage limits we need to characterize from the DC-DC converters we are using to power the D-Link AP. I've asked 2 times to have this topic split (because I can't) so that this matter would not distract from the other valuable observations and suggestions but since it hasn't been done I'll post the detailed version elsewhere. Suffice it to say I think this resolves Al's concerns as best as I understand his communications at this time. I'll post the link later when I have time. I think anyone that's really interested will figure my solution out from the minimum of detail I just provided. |
Re: Intermittent connection on field only
Interesting observation, as of right now the questions asked after the one about monitoring the D-Link have been answered in the official Q&A forum but this question, now days old, has not.
So much effort to avoid checking a simple voltage with already approved hardware. The amount of energy expended to avoid removing all doubt now greatly exceeds that of just doing it. |
Re: Intermittent connection on field only
We had the same problem in spokane...try setting your camera to 6fps
|
Re: Intermittent connection on field only
Quote:
That being said, our comms issue went away in our last 2 matches at MAR CMP and in 8 matches here in STL has not returned. Nothing was changed, so we don' t know why. Yet. |
Re: Intermittent connection on field only
Quote:
First off, I offered merely a solution to remove power quality issues for the D-Link AP not a guaranteed solution to everything that effects the communications. Please quote me in context. Secondly: Team 11 certainly did have power issues with the radio systems. Other teams certainly did have power issues with the radio systems. Telling me there is no power issue with the radio denies Team 11 had a power issue with the radio subsystem and so did others. It denies that I have here a bad DC-DC converter that took 3 matches worth of dysfunction on the field in Philly to find, and others have seen the problem go away when the PDB was replaced. I'm sure it's a real problem if our team is stuck on the field for 3 matches just like anyone who has problems besides the power quality issues. Just as I am sure it's a real problem if your team is stuck because of a software problem. The fact is there is obviously more than one thing not quite right with the radios and the systems that power them. That was my entire point. Outright denying a power problem can and sometimes does exist...either because of the parts or because of the wiring refutes the evidence that even those who do so present. It also misdirects the people that do actually have power problems effecting them. A loose connector on the D-Link is a power quality issue. Why are we intent on saying that doesn't happen when in point of fact even Al notes it often does? Not all power quality issues are about the components sometimes it's just the wiring in the power system. So far as how I troubleshoot the problem: how would you like me to troubleshoot a problem that per the posts in this topic seems to exist most often only on a competition field...when i don't have a competition field to work with? Even when I worked at spare parts at MAR Mount Olive it's not like I can put things on that competition field that FIRST won't allow. Furthermore thanks to my efforts we no longer can dispute that FIRST will not allow it, they officially said no (as I originally stated they would before everyone led me to believe I was incorrect...again why is this my problem when I was correct?). We do have a field but we certainly don't have the competition field parts. If the issue was about me distracting this topic I openly offered to take this conversation out of this topic. Such an argument is a straw man. I do respect you and the others Don. However, expending so much effort on scapegoating me to cover for a power quality issues that sometimes do exist is not a reasonable thing to do. The reasonable thing to do if you all feel it distracts is offer me another place to discuss this issue like another topic (I can't create topics myself). If you tell people to troubleshoot for a software issue and they have a power quality issue then like Team 11 they'll spin their wheels rifling through the software looking for what might just be bad power supply to the D-Link. Then what? Who's fault is it then? The guy offering the solution to help rule power quality issues out quickly? I think not. Furthermore, I asked repeatedly in this topic if anyone load tested the D-Link power supply and no one replied. So if you wonder how the bad wiring or bad component (no matter the rarity) might slip through you need look no further than that. Additionally, it's clear that despite the general assurance we'll find all the problems if I just stop communicating, Championships are over and obviously these issues remain. So what have we solved? How does the situation today differ from the situation at the start of March? |
Re: Intermittent connection on field only
Quote:
There's more going on here than just power issue, but that's not saying that everyone that has communication problems isn't due to power related issues. I'll admit that we did have power issues early on at Bayou, or at least it seemed to be, when we smacked the Dlink, the power would be reset. We corrected it by using the OEM power cable and hot gluing the outside of the plug to the radio to avoid it moving during vibration and hits. Bandwidth is another thing I don't believe in. While we had a couple of advanced teams at Bayou, it was nothing compared to what we saw at the LSR. Only a handful of robots at the Bayou used live streaming video from the robot to the DS, at LSR, I would say that number doubled or tripled. Interference? Maybe or maybe not. I'm not an expert in radio interference, so I cannot say whether or not that was the issue. I did notice at least half a dozen APs at Bayou and the LSR, so yea... not sure... Driver station overloading? Yes, it will cause issues if you are having saturating the CPU on the DS or the cRIO, but that wasn't our issue. We tried 3 different driver stations (Classmate, Dell M5040, and 2010 MacBook Pro), all plenty capable of running the DS software. Our cRIO ran at around 65% usage on average, and for a real time system, that's good. So yea, my point is that no one can say "This is the problem everyone is having." unless NI or FIRST comes out with a statement saying what the issue is. We have an open ticket with FIRST and they will be investigating the logs from Bayou and LSR to determine what was going on, but as I understand it, the logging features of the field software, the driver station software, and the cRIO software is still pretty underdeveloped, so the cause may never be determined. There are things that we... should have done, could have done, wish we would have done, etc... but we didn't. These issues for us were just coming up on CD from the Florida regional, where the Bayou field was the week earlier. I wish I would have checked for things such as other routers with our ID running in the pits, and I wish Team Fusion would have kept on an engineering approach and changed one thing at a time when we were trying to solve the issues. The issues with our field connection put all of our volunteering mentors on edge. Emails were shot out to FIRST with infuriation over the handling of our issues at Bayou from our mentors. I feel that we proved to the FTA that the issue was not us. While the same FTA that kept blaming us for the issues was packing up the practice field, Team Fusion was running on it, wirelessly, using the hardware they provided to us. We ran our robot for a full 20 minutes without a single drop of communication. I talked with the FTA and asked what he thought the issue was since it runs perfectly everywhere but on his field. He was straight up rude (I could call him a different name...) and said that the issue is not the field, but is still with our robot. If he KNOWS that the problem was with our robot, he should have told us what the problem was, because I spent the three days of the regional ruling out everything but their system. But I believe he said that to cover his butt. That's what he's supposed to do, right? We packed the same broken robot up into the bag and brought it to LSR, and experienced no issues whatsoever. How is that an issue with our robot and not the field? And the extent of our debugging went down to taking everything off our robot. We put back the given arcade drive code to just drive our base, with the default dashboard software on the classmate PC, with no camera feedback, and still couldn't run. CJ, our CTA at the event spent a ton of time in our pit going over things with us, and I believe he came to the same conclusion as us that the issue was not related to us. I know that some of this is offending to the FTAs from the Bayou, but I have said nothing that was untrue from my point of view. I'm extremely upset with FIRST over this problem, and so are all of the other mentors for Team Fusion. For many of the students on the team, this was their first event to attend. I can tell you that none of them were impressed with what they saw. -Ryan Nazaretian |
Re: Intermittent connection on field only
As long as the methods used to troubleshoot are anecdotal not quantifiable there's going to be increasingly tension between everyone.
There's too many possible issues, combinations of issues, and things we can't test off a competition field, and that makes sense because the robots are all different and there are different fields. It also makes sense because as we move the robots and the fields around we risk disturbing things. The field guys are told it's not the field it's the robots. The robot guys have exhausted all the tools they usually have, including extensive testing, and the problems continue. This problem seems to continue into situations impacting the performance of the best of the best by elimination. It's no longer about team reputation, veteran status and whose word can be trusted (in a level playing field it should not be about that at all anyway). There still seems to me only one fix to this problem. To find ways to test and get quantifiable evidence about each system. The field. The robots. Each and every time the problems appear while they appear. FIRST has made great strides in field and robot monitoring since Team 11 first saw communications issues last year at an off-season event and I sincerely do appreciate their efforts. I hope that FIRST will continue to seek quanitifable information of all kinds to insure that these events move quickly, move cleanly and move with the sort of direction that can only do credit to everyone. It serves no purpose to point fingers at anyone. This is the world of science and engineering it's about the numbers and the evidence. I still intend to let FIRST evaluate my voltage monitors and even if they only fix a small percentage of the problems by volume...that's a small percentage closer to the goal. |
Re: Intermittent connection on field only
I agree. There are a lot of pieces of the system that can go wrong, and the only way to get a stable competition environment is to have a way to get solid data on a real field. I intend to bring some network tools to view the field at the off season competitions that my team is going to, so that I can take some measurements, and get data. I *believe* the network configuration is part of the problem with the field. Stories of robots turning off their cameras, and have connectivity issues go away isn't proof, just more anecdotal evidence. I also believe that the DLINK is a weak link.
It would be nice to get to the point where the doubts about the field network can be laid to rest. It *may* just be a robot issue, but as long as we don't have data to back that assertion, it's just opinion. |
Re: Intermittent connection on field only
Quote:
We still seem to have a communication issue so I will repeat in as clear a series of sentences as I can. There is no "obvious" power problem with the radios. There are issues that teams should be aware of that compromise the power delivered to a radio used on an FRC robot. Namely, the power connector can become noisey or intermittent due to team handling. There are rare occasions where the 5 volt regulator is defective, again in most cases due to team handling. And on still rarer occasions (less than ten that I know of in the whole time we have used the PD) the PD power supply may not function correctly. With the 3500 teams, 60+ events, hundreds of matches, thousands of practice runs at both events and home fields, these radios have worked as intended, powered as designed. I personally witnessed failures over this past weekend that were not radio related even though many observers may think so. They were attributable to faults on the robot either in software or hardware that while intermittent were eventually corrected. We have a good and cost effective competition system that works. There are occasional issues that are as yet unexplained but the FIRST staff is working hard on solutions. They are as committed as I am to insure everyone has the ability to run when they are scheduled to play. Again, there is no indication these issues had anything related to power on the robot radio. You can continue to look at your own robot and spend your time chasing a power issue that you believe is the only problem that exists. I am convinced (more now that I observed from the scorer's table thousands of matches) that radio power is not the panacea that you believe it is. While I do not wish to stifle teams who are trying to expand our understanding I am firmly committed to preventing misleading lines of thought so that teams will keep their minds open to other possible failure modes. |
Re: Intermittent connection on field only
The CMP proved that there is a problem. Robots go dead during a match. The big problem as I see it is that there are many points of failure. Power and the power connector are just one. Isn't FIRST about the engineering mind set? Let's Give the students a real world example of how to kill a technical problem.
I would like to suggest the people working on this problem need allot more info than they have now. First they need a list of robots and matches were there was a failure. They need specific info about the robots on the field. Everything from radio mounting, language, video stream details, DS configuration, DS laptop details. Failure description (did the robot come back to life) and many more robot specific configuration details. Any solutions they tried. This could be sent out to teams as a questionnaire and the responses tabulated. An analysis may yield points to explore. The dropped coms issue has been around for some time. It needs to be attacked and killed. |
Re: Intermittent connection on field only
Quote:
Your argument is the equivalent of you have it handled and there's nothing to see here. I think a national broadcast of that was something to see there. No disrespect to you intended you are in point of fact overwhelmed. It was clearly apparent on the faces of each and every senior person on that field. It was the direct result of doing everything that could be done to sandbox any argument that differs from your own. In point of fact you continue to do exactly that. It is literally unbelievable. From your own post: Quote:
Quote:
Again, even if a small number of teams are impacted by power quality issues who is anyone to claim a fair and level playing field when you deny them the opportunity to find their problems quickly? |
Re: Intermittent connection on field only
Good Morning All,
Being new to FRC this year I heard about the Bayou issue from a lead NI engineer. So FIRST did send them to LSR I think for free? Knowing this then there is an issue somewhere that needs attention. I agree we shouldn’t chase some things, however, I would have figured that those running the contests would have some type of monitoring software. If not maybe this is where it should start. Have "HARD" evidence for the FTA staff showing the connection rates etc. and radio strength to prove or support technical difficulties occured during matches (document! Document! DOCUMENT!). As a mentor/coach I would have a hard time selling to my kiddos about their mistakes if I didn’t have something to back it up with when it seems to work fine at home and not at the conest. Also, I watched the World championship matches and was really wondering what happen with 1114 & 2056 last matches. They averaged 40+ pts. And then to see a sudden drop makes me think of another issue and then the e-mail from headquarters this morning. I know we try to instill GP across the board and this should be a lesson for all parties involved on how to improve. |
Re: Intermittent connection on field only
When a team shows up at an event and the only people welcome to troubleshoot on the competition field are your and your group. When your word is gospel. When it can't be questioned.
When representatives of your organization tell a team over and over to go find problems in their robot, and it's clear that your organization field personnel are comprised of people with relationships to certain teams. Worse it's getting increasingly clear that the field personnel don't have any more resolutions than you as a team. Exactly how do you think your argument does not lead directly to the feeling that certain teams have an advantage by having an 'in' with your organization? Marginalizing teams and people with troubleshooting knowledge is entirely incompatible with the mission statements of FIRST. Furthermore Alan spent a decent portion of this topic harping on essentially the idea that Team 11 doesn't know how to measure a battery or build a properly functional robot. It didn't matter what I wrote. The evidence is there for all to see. So you tell me. Why should I not interpret the continual actions to silence me and the continual actions to down play our team's ability as a direct example of anything other than a very unlevel playing field? Just how far is FIRST as an organization willing to go to avoid troubleshooting from the bottom up? Are you going demand next I be removed as a mentor because I stood up and pointed out that this troubleshooting process is tragically flawed when the tragedy has already claimed it's victims? How far is FIRST willing to go to protect this failure? I will not go back in my little box Al. Expending all this effort to put me back in that box only demonstrates that there's more wrong in FIRST than a mere failure to troubleshoot some WiFi and field issues. |
Re: Intermittent connection on field only
Quote:
It does not appear that FIRST values the concept of a community effort. They want a few loud voices to shout over the rest. All the excuses about my contributions to this topic distracting are also straw man arguments because I asked to be given another place to discuss the matter. They refused. They don't want to prevent distraction. They want my message to troubleshoot from the foundation up, and myself as the messenger silenced publicly. |
Re: Intermittent connection on field only
Okay Brian,
Since you don't know me or my skill set, let me explain. My real job is troubleshooting electronic failures down to the component level. A position at which I excel. I know that power can be a definitive issue with many problems that exist and I know both how to diagnose those issues, and how to correct them when they fail or are designed improperly. Power supply noise in digital systems is nothing compared to the noise generated in analog audio systems where microphone levels are in the -60 to -90 dBm (that's millwatt for those who are wondering) range, can require as little 4 microamps from the power supply and where noise is considered bad when it is only 20 db above the theoretical noise floor. I have been telling you for weeks that power is not the issue you have deemed it to be but you have failed to believe me or the others in this forum that are trying to get you to accept that fact. While there are problems (and they have yet to be diagnosed), power is not the greatest of these. How can you even think that those people who are researching every possible failure point have not considered looking at the power supply first? So to borrow from others, stop muddying the waters, perform your tests and bring us real data that can be duplicated in the lab or field and actually correlates to failures of the robot wireless link. Until that time I will only respond to others seeking real answers. Edit: Now that I have have seen your most recent post, let me assure you that Team 11 is a long time friend and a team I certainly respect. When at competitions, I would work as hard helping them compete as my own team and I expect all my inspectors to also do the same. |
Re: Intermittent connection on field only
Quote:
You know very well that I can't test a competition field environment without a competition field. So you're telling people that I should basically test something you are insuring can not be tested. Furthermore, each and every robot on that field is wired differently. Your argument that you've tested this out fully simply can not be true. You have only tested the ones you've tested. So your lots are smaller and the results are always rushed because you most often do those tests on team's robots when you access them during a competition and you only test the ones that do not work. Worse you don't have a complete tool set to test on the field during the matches. The point again, for all to read, is not that all issues are power quality issues. The point is that when you do have power quality issues you are effectively putting those teams at a disadvantage. I fully admit...again and again in this topic there are more problems than merely the power quality. The only thing I offered was a way to help identify and remove those power quality issues when they exist I left the rest to you to figure out. Note: this next part was posted before Al edited his post. So far as this not being a real issue...you prove once again that Team 11 and other teams finding bad power supply components feeding the D-Link AP is not considered a real problem. It's not a real problem why? Perhaps because Team 11 and these other teams weren't on the Einstein field? Going to be hard to get to that field when our not so 'real' problems get in the way. |
Re: Intermittent connection on field only
Quote:
As I've mentioned many times before... Team Fusion, at the Bayou Regional, could not run a single full match due to communication problems. Our first suspect was power, and sure enough, we did have a power issue. The student that purchased the connector for the router purchased one that 'fit' but didn't fit properly, leaving us to have a potentially bad connection. We could pick up and drop, shake, etc... our robot, but it wouldn't drop, but if we put focus to abuse on the router, and picked up, dropped, shook, slammed the router, then we would cause a reset. That was our first step. We had a spare router, with a spare power cable, so we took the OEM plug and used that. After replacing and gluing the plug in place, we could not brown out the router. We did stress testing to other power components of the router as well, including the 12-to-5 regulator, and the PD board. We replaced the 12-to-5 regulator for comfort, and we hit on the PD board a bit, but could not replicate the issue. And again, as the title of this thread mentions, the teams with this issue had a "Intermittent connection on (the) field only." We worked great in the pits, on the practice field, during kickoff, practicing at home with our router, everywhere, but the dang Bayou field. The only thing the FTA would tell us is that it was a robot problem. No specifics on what is failing, or why, just that it was an individual problem with us. Here are the two things I cannot get around:
There's something external to the robot with the issue we experienced. With what I saw on Einstein, the issues experienced by all of those teams was the same issue we experienced at the Bayou Regional, a Week 3 regional. As far as router placement, our original location isn't the best place, and is within a foot of some noisy shooter motors, but we relocated it during Bayou to a better location, obviously not fixing our issue. As far as code goes... besides the fact that our code didn't change from Bayou and LSR, we went back to the basic framework code, no camera (unplugged), no CAN, no PID, nothing. We simply had solenoid and motor outputs, no sensor inputs at all. Then comes the driver station. We had been using a Dell purchased this season as the DS, but during all the debugging, we switched to my laptop, a 2010 MacBook Pro (Core i5, 8GB RAM, SSD... a pretty powerful machine), and we kept on having the issues. Finally, when trying out the basic framework code, we switched to the Classmate PC, configured for this year and updated with the latest FRC updates, and no Windows Updates. Basically, at the end of Bayou, we had a kitbot control system, but still couldn't maintain a connection, even sitting still. All of this proving, in our case, to me, that this isn't an issue with Team Fusion 364's 2012 Robot, Aiminite. Our issue was not a robot issue. That's all I have concluded from the work I have done diagnosing the issues. I'm not sure if we were the only one during the regionals to pack up a broken robot and pull out a working robot, but FIRST should look at our case for some good information. Team Fusion has concluded that the issue is not our robot. Now to the big question that everyone would like to know... what is the issue? No one knows at this point. Everyone is just speculating. Interference I've been speculating that the issue is interference from the audience carrying hundreds of smart phones with WiFi enabled... but that even has errors in the hypothesis. Lone Star had a much larger crowd than was at Bayou. It's much larger event. The Lone Star regional is also held in a bigger city (and actually in the city). Bayou was right next to the river, with just a small road leading up to it, not part of a major city arterie. It seems like LSR had the potential to be worse as far as interference goes. So interference seems to be a non-issue. And why would interference just target us? Things don't add up for this case. What about the WiFi channel used? What does happen with the Dlink is the 2.4GHz is left enabled? Field Hardware Issues Again, if it truly was FMS messing it, at Bayou, it would have affected everyone, not just Team Fusion. We had been placed in multiple spots on the field, but we had problems in all of them. I'll be down on the Coast next week, and can test some things, but I need a list of things to test. First off, I want to break communication. What can we do to break communication? We'll need to have two routers and use the same system FIRST uses. I'm curious as to what will happen is we power up two of our robots, with the same IP address and everything. How will the system handle this? If it works, how do our logs look? Then what happens if we run some noisy equipment next to the 'field' router. Can we run a Skil saw next to it and still maintain connection? How about running a FP motor at full speed next to it? What happens? How do the logs look with this? Did Bayou have wireless lighting controls? Were their frequencies within the range of our channel provided? I can't tell. Anyway, I want to help figure out this issue because it really messed us up in Bayou, and we have no explanation as to what happened. |
Re: Intermittent connection on field only
Quote:
A level playing field is just that. It should not matter if Team 11 is friends with you. It should not matter if Team xxxx who just walked on that field you've never met is friends with you. Troubleshooting is troubleshooting. We must insure that guidance for it is uniform. We also must insure that if a relatively unknown team does in fact eliminate all other problems we respect the effort and investigate. |
Re: Intermittent connection on field only
Quote:
This year, I was running throughput testing for a demo board. At 10-30 feet in a quiet environment (as shown on a spectrum analyzer), we were achieving about 120 Mb/s out of a theoretical 300 Mb/s using IXChariot software. If all robots are sharing one channel, that is pretty unnerving, especially given the throughput requirements for a single video feed. |
Re: Intermittent connection on field only
RyanN:
I deeply respect that power quality issues are not the only issue on the field. That's why I asked to separate the issue repeatedly. Obviously when component in the power supply to the D-Link AP goes bad it can do so at anytime. Team 11 drove our robots extensively before the competition in Philly even at other competitions. All of a sudden the DC-DC converter decided to be a problem. We, like apparently most people were not instructed to load test the D-Link AP supply, no process was provided nor even basic specifications. I did ask repeatedly in this topic whom else load tested that supply with no responses. This was a power quality issue that was intermittent on the field only seemingly, but just because the timing of the failure coincided with the punishing match schedule. Not all power quality issues are wiring problems. Not all power quality issues are component issues. As others have said, in the absence of procedure and tools it's very hard to walk a troubleshooting process to work that out especially under pressure and especially when there are real WiFi communications issues that keep undermining the basic troubleshooting process. It's entirely conceivable in the current test environment for a team to check their wiring, replace the D-Link AP. Fail on the field. Replace their DC-DC converter and fail again on the field. Replace their PDB (which can easily consume all the time between matches) and fail on the field. Then after all of that diligence, they fail on the field for other reasons which may, or may not have been there all along as well. So in effect that's about 4 matches of effort right there where a robot might be totally or mostly non-functional. That's a great deal of ground to loose. Compound that with the fact that your robot's mechanisms might not be top tier and those 4 or 5 matches might put you out of the competition entirely. I know we are here to above all have fun. How much fun is it for a rookie team to possibly loose 4 matches due to an unclear troubleshooting path, have to go home and find more sponsorship to come back next year, and while they are at it very likely be entirely unable to repair the issue once they leave that environment? Doesn't sound like fun to me and please be aware I was one of the people that founded Team 11. Given the opportunity to treat this issue separately that's what should be done. I respect that your team expended all possible tools and efforts to eliminate your concerns. However, until someone helps me to separate these 2 topics properly I am left little choice to respond within the limited venue I've been offered. If people were trying to respect this basic foundational concern they'd help me move out of the way instead of demand my silence. Perhaps more importantly, demanding my silence here is symptomatic of the fact that we have had communications issues with this system for years and in the past the finger was pointed at the robots. The teams had no reason to believe that the problem could be anything but them because the authorities on the subject insisted that it could not be the field. People should stop insisting, open the floor, divide the topics and if you want me to shut up and test myself let me get onto a competition field and do what I need to do. |
Re: Intermittent connection on field only
Quote:
I believe we were hit with this last year, but have not seen it documented or mentioned elsewhere. We had 2 unexplained no-coms at NYC regional last year, FTA said it was a bad battery. On a whim we removed CAN and switched to PWM for the next regional, and did not have a problem. Has anyone else been hit with the 'Joysticks disconnect' problem this year? It would happen mid-match, and the fix was swapping the joysticks in the DS and swapping them back... |
Re: Intermittent connection on field only
FRC 2012 - Known Issues List
Quote:
|
Re: Intermittent connection on field only
Unfortunately, which robots are on the field is part of the "environment", and so working in one regional and not in another could simply be the mix of robots. I believe that the DLINK is certainly one point of failure, from the battery to the power connection on the box, and internal to the DLINK. I also believe that there are issues with how the network is configured.
Maybe at one of the off season competitions - our first one is at Monty Madness, we can get a group together and poke and prod the field setup, and network with a mix of different robots. If we get enough robots together we might be able to map out some of these issues, on a real field. I think that we would all like to have a level playing field for all robots. And to do that we need data on real fields, and I would like to be involved in the analysis, not simply told that it's "fixed" because I'll be back to the "there doesn't seem to be any issue with my robot, in the pit, or at our build site, and there is at competition". And once again, left to tell the kids that I can only theorize about what is wrong because the conversation is closed when it comes to discussing details about the field with FIRST |
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
@Mark - So that's an issue with Labview and not Java? (We were using Java)
|
Re: Intermittent connection on field only
Quote:
|
Re: Intermittent connection on field only
Quote:
The LabVIEW default framework is constructed to kill the Autonomous thread at the end of the mode. The good part of this is you can't easily get stuck in Auto like you can in the other languages with sloppy code. The bad part is, with sloppy LabVIEW CAN code, you can end up with corrupted CAN. |
Re: Intermittent connection on field only
Quote:
I suggest you look back and reread what I have said to you and to others. You believe your team obviously has a power issue but you refuse to perform the tests that would point to a power issue on your own robot. Do so and bring us the results. You do not need a competition field to do that (try using FMS Lite). Your robot remains a stand alone component that if power is an issue, it should show up that problem on your home field. If you find something that others have not, then and only then can FIRST actually decide to pursue an investigation using other methods. You must admit that what you are asking requires a rather large expense on First to accomplish. Bring valid data to support your claim and First can weigh your data against the need to expend further money and resources in the pursuit of a radio power supply issue. As for myself and all other robot inspectors, we take our job very seriously. When we don the shirt and hat we take on every team as our own, bar none. When they fail on the field, we fail. When they go home with a loss, so to do we. We feel strongly that the success of an event lies our in our hands and the happiness of the competitors is our highest measure of success. We cheer for everyone because that is what we are required to do in this competition. When we see something wrong we try to fix it or we find someone who can. We do this at every competition for every team, all weekend, without regard to team number or qualifying slot. Should an inspector or Lead not meet these goals, I want to know about it. Teams need to be able to trust and believe in the assistance of their inspectors. Our stated goal is to insure you play (within the rules) and not to prevent you from playing. I can tell you that robots not running is a common discussion topic during volunteer meals for inspectors. We pass along what we find so that all teams can fully participate. As to your finding a bad power component at one of your competitions, that is a known issue with a very small number of regulators. (a very few have been found defective from the factory) When questioned, most often the root cause is a team wiring issue that was later corrected by the team. For the record, that can take the form of the input and output of the regulator being swapped, the input or the output wiring reversed, the input to the radio reversed, or some of the wiring not fully insulated and shorting to the frame. As to other issues with robot radio, we have found the use of the wrong size coaxial plug (either the inside socket or the outside barrel or both), a team fix on a broken plug, a broken wire internal to the cable due to stress or bending, improperly applied hot glue that forced the connector open or un-soldered the power jack. More than half (~100 this year) of the power problems we find is teams wiring the radio directly to an unregulated 12 volt output on the PD. Other issues include blocking all the cooling holes in the radio, using damaged ethernet cables or after having broken a connector hot gluing same into an ethernet output on the radio. Mounting the radio upside down against a metal plate or mounting the radio on top of 2 CIMs in a transmission are also common. And finally teams mounting the radio deep inside the robot to "protect" it from harm. So to repeat for everyone's homework, if you suspect a power issue with your robot radio, test and document the conditions under which the link fails. i.e. a power supply droop, noise (how much and what frequency), radio reboot or not, disabling of certain inputs or outputs, do you have a camera and does it stop working, does the robot stop after auto or after a specific length of time coming out of auto, etc. Make your findings public or send them to me, so others can duplicate the problem. As far as your last paragraph above, I never said this was a non-issue. I said, in my experience I have not found any evidence that the power supply issues (that you suggest are occurring) have caused link issues. So far you haven't shown any evidence of this either. Your team found a defective regulator that for you and me is a smoking gun in your robot. That is a known problem, stated multiple times over the past two years by me (and many others) in several posts here on CD and in the LRI forum for all inspectors. For the record any problem is an issue for me. If you have one, the chances of someone else having it are good. However, I can only pursue problems if you can tell me what it is and repeat it. If you were closer I most certainly would suggest I spend a Saturday at your facility pursuing this. |
Re: Intermittent connection on field only
Quote:
The solution to the problem is past tense. It was found. Other teams did have a power issue. They replaced the PDB and it did fix it. Past tense and in the next part I accept that you publicly acknowledge this. Quote:
Furthermore, some of those students you have on that field crew are students I mentored. So let me be clear. I am extremely unhappy when I see students in my chosen profession. A profession to which I have been an apprentice. To which I have clawed my way up in versus vast nonsense. I am extremely unhappy when I watch them get reduced to tears. I am tired of FIRST using them as reason why I should watch what I say. I should like to point out that the very first e-mail I ever got in private from anyone on ChiefDelphi was Eric telling me to watch what I say because 40,000 students are watching. Yes they are and they need leadership and answers. Some of you are their leaders. You dictate the limits of their tools (you certainly shot down my offer of a tool to which you have little alternate). You dictate the information and facts they are presented. I feel given the sheer lack of students in this topic (especially among those telling me to go away) that the leadership of this matter is using them as a shield. Not that you Al are doing so...but I see it all over the place. Where we tell people don't judge them so poorly. I am not judging those students, sometimes my students poorly. I am demanding they be given an environment that is never allowed to come to this end! Quote:
In a competition where I had students that did not even know what the oscilloscope was nor it's application on the robot. Considering that I offered test equipment for mounting on a moving robot, or even to open the door to using the cRIO which was shot down in the official FIRST Q&A forum after an unusually long delay. How you figure that so many of the people you ask have the tools to deliver on your request? Especially, again, when I point out that the tools to eliminate this issue on the active field...when it's the most disastrous they do not have. They can't even use the cRIO on the competition field as a data logger on that power supply. Even if I show you what we already know. That power issues are the bottom of the troubleshooting tree and that we all admit they effect people. The point has and remains that what a robot does on my test field sometimes means nothing once I toss it in a bag. Ship it all over creation. Smack it into walls. Have pieces torn out of it by other robots accidentally and expose it to just general wear and tear. The entire point of the monitors I created or even using the cRIO as a data logger on the D-link AP power supply was to prevent incidental damage as the result of wear and tear from blind siding teams that have otherwise done everything right and just don't realize the cumulative damage they are taking. We at Team 11 had no reason nor any approved method to track that damage. That's not unique to Team 11 at all. There's no approved method to get into that point while moving, and certainly no example I can find of anyone even load testing that supply and again I did ask openly in this very topic. Quote:
I have every reason to believe based on the pattern I've been a personal witness to for the last week that I need to say that paragraph above publicly. I personally over a year ago offered to provide nearly the exact same thing LinuxBoy has been working on and the feeling I got personally was that I should go away. Do not do that to LinuxBoy, do not involve him in what seems to be an issue some of you have with me. |
Re: Intermittent connection on field only
Brian,
Your last post is confusing, what are you trying to say? Are you saying I act with favoritism because I don't agree with your suggestions? With your methods or with your premise? What are you referring to in the paragraphs about students and field teams? I am suggesting that you use your monitors on your own robot on your practice field to give First and the community some data to digest and duplicate. Are you against that? As to you having something on your robot during Monty Madness, that is up to you and the event staff. As to First's reaction to your hardware, I have no knowledge of what you are relating to us. For the record I am not FIRST staff, I am not a member of the GDC, I am a key volunteer. As to adding monitoring to a robot, may I refer you to StangSense. We used the control system in conjunction with a custom circuit, to monitor current/voltage to critical systems, mostly motors, using a method that did not interrupt the power pathways to electrical components on our robot. We also used that system in a mobile design to aid other teams in the pursuit of problems on their robots for several years. We also described the data we collected and the subsequent issues we observed to anyone who wanted them and that data aided in the design of the current PD power circuitry. FIRST added several test bed monitors to operating robots (including ours) during the design phase as well. I assisted with at least some of the interpretation of that collected data during the collection at IRI. The boost/buck regulators on the PD are a direct result (I believe) of that research. |
Re: Intermittent connection on field only
First has a serious problem with dead robots on the field. The radio power supply and cabling has known issues. Given the severity of this problem and most likely there are many points of failure all points need to be looked at. The power supply issue needs to be looked at closely to see if there are problems other than cabling. I mentioned that there are potential problems with a switching power supply feeding a switching power supply feeding a switching power supply. I have never head of anybody that has characterized the load dump and switching transients on the PD. Does this affect the power supply to the radio? One transient that gets through is all that is needed to potentially lock up the radio electronics. It would be a very good idea to get some real on the field data. This needs to be done at a competition. A competition like Monty Madness should be RF noisy enough to put the radio into a high power state. ( gain scheduling) Team 1640 will be at Monty. We would be willing to have data logging put on to our robot. Can't load code on our c-rio. It's busy enough. Has to be stand alone. We have a swerve drive using 8 motors plus 4 more motors for ball handling and shooting. Should be a good heavy load environment for some testing. The testing should be able to see short transients. So if this want to be done our robot is available. Contact me if our robot is needed.
|
| All times are GMT -5. The time now is 17:53. |
Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi