Re: Intermittent connection on field only
 

Eric-

Since you know the design of the FRC control systems components extremely well, is it fair to say that the current PDB 12V -> 5V DC-DC Regulator -> D-Link AP is well characterized (publicly or privately) such that all the components are appropriately spec'd when it comes to power ignoring failed components, bad wiring, etc? I'd hate for people (myself included) to go down the path of characterizing something that has already been done. I assume the homework has been done but I know things have changed slightly in the control system setup since it was originally designed.

Re: Intermittent connection on field only
 

We tried the idea of putting a volt meter on the output of the DC-DC converter to catch the minimum voltage. With the meter connected we could not reproduce the problem of the Dlink restarting. We were also using a different driver station PC. Could the meter have smoothed out any voltage spikes that might have been present? What size ferrite and/or capacitor should be used to test this? Would they be legal?

Re: Intermittent connection on field only
 

The meter should not have much, if any effect on the circuit given its high input impedance. Depending on the quality of the meter however, you probably will not be able to catch any quick transients with it. An oscilloscope would be much better at catching those.

I see no reason why ferrite beads would be illegal since they are passive and don't physically connect to the actual conductors in the wire. Appropriately sizing them would depend on the noise you are looking to reject, etc and I couldn't provide a recommendation at the moment. Capacitors on the other hand may be illegal in the current rules.

Unless you've had power related issues with the D-Link I would carefully consider adding anything/changing anything at this point.

Re: Intermittent connection on field only
 

The usual symptom of a bad 12v-to-5v converter is a failure to regulate its output at all. With 12 volts on the input, it puts approximately 7 volts on the output. There is no immediately obvious effect on the communication, but it's probably stressing the D-Link's power input and I would expect it to fail, perhaps slowly.

Re: Intermittent connection on field only
 

When you say 7V are you measuring that with a DMM on DC volts?

I would expect a bad DC-DC converter pumping out switching frequency instead of DC would drive the DC reading of a DMM into strange territory. A low frequency signal like 60Hz AC usually under reads on the positive side and you might see the negative side occasionally (quantification by the DMM's DC voltmeter sampling), but I seriously doubt the internal switching frequency of the DC-DC converter would be lower than 1kHz and might be much higher. Also there's the chance that as the DC-DC converter reaches beyond it's ability to pump energy into it's output storage because of feedback internally it'll allow it's regulation to creap up because it'll keep saturating it's switching MOSFET for extra cycles to feed the energy storage. Depending on what safety systems are in the DC-DC converter you might even see some slow peek modulation imposed on the output as it tries to restrain it's output voltage and then gives up a bit and tries again in a loop That would have the effect of bringing the output ever closer to the input (so you'd see peaks between 12V and 5V...like 7V). However, if you looked with an oscilloscope you'd notice that it's not 7VDC, it's really more an AC small signal. As the D-Link's input capacitance is already within the circuit it'll not easily be able to impact such a wild input. In probability, however, the D-Link AP will have at least a standard linear regulator in there with some capacitance beyond it and that'll provide at the minimum a few dB of damping on the AC improperly feeding it. I would except misbehavior...perhaps due to complexity it might not be apparent but I bet a really thorough review would find it.

There's also the possibility that there might be a measurement difference from a cheap DMM and a true RMS DMM even though true RMS DMM usually impact the AC performance of the test equipment not the DC (I've seen some really cheap DMMs in my time).

I'm still chasing down the DC-DC converter Team 11 found damaged on their robot and when I find it, I'll put an oscilloscope on it. I'm not sure, however, that the voltage it was producing was too high on a DMM. So if someone does encounter a unit that produces 7V according to a DMM, please put an oscilloscope on it and describe the output signal for us.

If I'm correct and the peeks are merely 7V the effect of the low pass filter by the D-Link's internal regulation (which would probably be low-drop out) would be to drop that 7V (not really DC) by a few dB. In effect it'll be like pumping more like 5VDC or so into the input instead a lot of the time. Occasionally it might peek over but probably not all that often which would explain why it keeps running at all...of course doing this is still not a great idea and you're probably right eventually either the DC-DC converter might give it a hard shot of 9V-12V or the regulation within the D-Link AP will fail (if it's linear the result would be thermal damage).

As that applies to your quote of my earlier post, since I suspect that the DC-DC converter's output would then be AC small signal and not DC under those circumstances I feel I stand by my presented idea. The best voltage and current monitoring solution for the robot would approximate an oscilloscope not a DMM DC voltmeter. Something close to an analog oscilloscope like a voltage comparator would see the instantaneous low current and low voltage. It would not effectively take snapshots a few thousand times a second like a DMM would. Since the input capacitance essentially forms part of the output energy storage of the DC-DC converter we could see currents and voltages that are during the length of a time frame exactly as I've stated. Also, we could see them during the length of the waveform as too high in voltage and still too low in current which if you look back is still a regulation failure symptom.

I will conceed, however, that if you put a circuit with an LED for exceeding the voltage maximum and an LED for exceeding the voltage miminum fed by an analog comparator you could see both voltage monitoring LED lit and a current that's too low with a failure mode like this. In effect the comparator would be fast enough to at one moment see one fault and then shortly later see the other. Latching the results would produce both lit. I'll add that mode to my cheat sheet on the page before but there's already a course of troubleshooting for it starting with diagnosing the robot power supply up to that monitoring.

In any event, it doesn't negate the value of at least attempting to investigate the power quality feeding the D-Link AP, it just adds another possible observation you might encounter.

Re: Intermittent connection on field only
 

I don't know what instrumentation Al Skierkiewicz was using when he characterized the failure.

It might help if you did a little more reading of what people are actually reporting and a lot less wall-of-text speculation.

Re: Intermittent connection on field only
 

I know that characterization of those values should have been done already.
Now all we have is speculation and swapping parts until we remedy the initial engineering process oversight (and that's all we can have given the missing quantifiable information).
Given I got the question into the FIRST Q&A forum I'm well on my way to removing the impediment to correcting that oversight.

Also I've read every post in this topic so that point is a straw man argument http://en.wikipedia.org/wiki/Straw_man.

Nevermind it ignores that my team did have this problem and it cost us 3 matches chasing phantoms with the help of field personnel while not being able to really do much on the field (and finally after chasing software and other unrelated issues they replaced the bad DC-DC converter they found in part because of my suggestion).

Nevermind that reading Al's post does not tell us whether the DC/DC converter was loaded or not or what tool was used.

Must have set foot in a political forum by mistake. Sorry.
Thanks for letting me know what my time is worth. Here I was thinking we were trying to be thorough.

Re: Intermittent connection on field only
 

We are trying very hard to find the needle in the haystack. Brian, you are adding a lot of extra hay to the top of that stack: the signal-to-noise makes it really hard to follow. Please constrain yourself to posting data. When you voice speculation as fact, it impedes the ability of the community to find the needles.

We value your input and your time - we need to make effective use of everyone's time. To do that, I'll pull from one of my mentor's phrasebook: Be Brief, Be Brilliant.

Re: Intermittent connection on field only
 

Point of reference Alan just seemingly explained to me that my posts are full of speculative statements (complete with quoted evidence from them) so I fail to see how I can be voicing the precise opposite. If the details of my concerns are challenged why should I not explain them before the next tactic is to claim that my failure to respond indicates my initial proposition was without merit?

Okay, here's the brief solution someone provide the quantified numbers I've asked for and FIRST should already have and I won't have to defend my reasoning in lengthy posts (for those that read things into every detail looking to find error, like claiming over and over that Team 11's robot battery was discharged despite me saying otherwise multiple times) I'll just do it and hand you the simple quantified results (which for all we know vindicate everything you've all been saying and the power quality issues are infrequent).

The question above remains the same. No straw man argument can distract from it. Do we have access to the quantified data I seek about the D-Link AP or must we as community engage in a process to get that data? I was fully prepared to start community data collection at my financial cost before it was suggested I 'slow down'.

All the other issues from the software and otherwise are valid. They deserve recognition but the format of this forum is not within my ability to control. If you're concerned that I'm burying these additional topics can someone (because I am unable to) provide a topic split?

Re: Intermittent connection on field only
 

My responsibilities shifted more towards FLL and FTC, so I'm not as involved in the day to day of FRC as I used to be. I designed the PDB, but FRCHQ did the verification work for the 5V DC -> D-Link this year.

The 12->12 is very well known, the 12->V is well specified, but the D-Link's input stage doesn't have the same level of documentation available. We've cracked them open and V&V-ed from the inside, but unfortunately reverse engineering can never create a data-sheet: We can only say that all the samples we tested worked well with good margin.

It may be worth while to take an oscilloscope to a working and a non-working setup. The frustrating part is that we haven't been able to reliably reproduce the issue in a lab, so we have to drag a subset of our equipment to competitions.

Re: Intermittent connection on field only
 

I partially write software that is used by gallium arsenide wafer fabricators to characterize their wafer lots and when they run prototype fabrications. There are the design expectations and then there are the actual data results for thousands and thousands of units on several wafers. Unless changes are made because the actual data fails to satisfy some critical parameter of the design expectations. They are reverse engineering to create what becomes the support evidence for those semiconductor's data sheets.

It might be true that you can't necessarily reverse engineer every aspect of the original design expectations. It's often the case that the actual product behaves sufficiently to serve the original design priorities but parts of it perform outside of the original expectations.

The point being that we don't hand a customer the design expectations and tell them this must be what they got because the product seems to meet the critical design priorities. We test and hand data backed by real measurements to the customer (hundreds of different measurements). Sometimes we don't even perform enough tests so we do some more because there's a problem with the RF modules after the fact. Often involving the amount of power these RF modules can consume when added to other circuits which toggle their states of operation (these are usually cell phone RF circuits and WiFi chipsets).

All veracity of all these test results requires effectively confirmation by quantity. The more of the things you test the more likely you are to notice the tolerances. That's where statistics come in.

None the less, the problem is that now that I've slowed down to allow other ideas to pass I've lost time to collect data relevant to the concerns I expressed from a wider selection of working and non-working robots. It's not likely that by middle of this week a whole lot of teams will take the time to perform a bunch of measurements to add to the measurements we can get at competition. In a real way the delay has removed a key validation aspect from the characterization process by time limiting the testing and the quantity at the Championship event.

Between the fact that I need a characterization to use my modules, and the fact that FIRST has yet to even answer the question about the legality of the modules for this year in their forum. There's little sense sending these items to the Championship. I grow increasingly concerned that the necessary characterization parameters will not be sufficiently determined as a result of this contrived delay's impact and therefore my circuit will produce dubious results by misapplication.

It greatly bothers me that FIRST might have tested an unspecified number of D-Link APs as samples from unspecified lots from D-Link and merely tested for the margin from overloading the DC-DC converter on the bench. It has partially blinded us at the troubleshooting phase and now I have basically run out of time to remedy this using the community and we are left merely with the same troubleshooting process that left us pecking and hunting for 3 matches while stuck like a brick on the field.

I will be asking FIRST to approve my units for testing. It seems (contrary to forum commentary I read when I proposed making these very modules for the Jaguar in the first place) that FIRST does not in fact have suitable test equipment to test on any arbitrary moving robot. That's what these circuits I made addressed as a primary design attribute and I want them to get a fair evaluation in the environment of quantified results that they deserve.

In the mean time, the good news is that if FIRST does answer my question in the official FIRST Q&A forum then they'll open the door to allowing the cRIO to act as a datalogger as well (at least for voltage) while connected to the D-Link AP's power connection. This means that it will make it entirely clear whether or not Mikets's Team's C++ class can be used on a fielded competition robot (see page 8 of this topic). So while that's not the most comprehensive solution to my concerns it's something that we can at least do for robots that run C++ as he has offered it (in theory all FIRST robots can do this with their software but you'll need to figure out how to do it for each language in each robot). Luckily as a datalogger the actually characterized component information won't be to critical. Too bad we have no idea how fast the cRIO can actually sample that voltage and my question to FIRST doesn't include sampling current (my reasons for that decision already stated).

Re: Intermittent connection on field only
 

OK Brian,
Let me repeat in a different way. Modeling the current drawn by the Dlink is a waste of time for the reasons I already outlined. When I spoke of the variable current demands I was not thinking in terms of msec changes but in terms of average current drawn. The reality is that that type of data you want to collect in a real world test, would give you absolutely nothing useful. The average current delivered will (I guarantee) change with with variables that are not under your control. (and there is no way for many people to control those variables outside of the manufacturer) The more variable of these is input RF signal processing dependent on the number of active channels, output power, SWR, modulation usage, packet repeats, and data transfer. Do you have an RF proof anechoic chamber at your disposal?
The radio link does not fail with changes in current except when the wiring has high series resistance. I am glad that a suspect 5 volt convertor turned out to be the issue. However, let's face facts, it is far easier to replace a suspect device with a know failure mode than attempt to shoot holes in an otherwise working system.
To remind those who are lurking.
1. A failure of the 12 volt PD power supply occurs when the battery supply falls to 4.5 volts or less. With poor mechanical design and/or electrical primary wiring, this is a real world possibility. The result is full radio reboot of 50 sec, or a full Crio reboot of more than 30 sec or both. this supply is capable of three times the current of the Dlink.
2. The five volt supply will fail when the input voltage is less than 7 volts. It is designed to regulate at least 20 times the current demand of the Dlink.
3. The permanent failure mode of the 5 volt regulator makes it look like a big resistor. The output is about 7 volts with no movement on the robot and it tracks the input when it is pulled down under varying load. There is no strict data on this failure as it is simply the series resistance of molten silicon in an individual device.
4. It is well documented that there are a few things that can take the data flow down. This is not an FMS disconnect as many people report. It is either a maximum CPU usage in the Crio, the DS computer, or an overload due to high data throughput experienced when the camera(s) is used at high resolution.
5. Let us not forget that there are certain safety protocols in use to insure that when the field is no longer capable of passing control data (i.e. robot enable) or the match is stopped, all robot function must also stop. This takes priority over everything else.


Of the odd failure modes, many are attributable to mishandling by the team. Number one of these is wiring the radio backward, followed by wiring the 5 volt convertor backwards. The next most common is the team who failed to wire the 5 volt convertor at all. While the radio may work for a while with no regulator, it will start to fail in unusual ways even when the regulator is correctly wired in place.
As to Crio related failures, these are very common and many are well documented. Please search for things like "feeding the watchdog", odd behavior with certain LabView implementations that are marked as unused. finally check to see if your Crio is logging activity and how often this is occurring. High loop times is good indicator that something is wrong in this area. If you suspect something in the radio circuitry, first watch the RSL to see what is happening on the robot. If the Crio reboots, you will see it there. If you robot always fails with a particular robot move, watch the lights on the PD (this is why you should mount where we can see it) and see if they go dim or go out. Watch the LEDs on the Dlink to see if they go out or if they go dim. The robot will talk to you but you have to listen. Finally, check with the FTA when failure occurs on the field. They are logging things like that battery voltage, lost packets and field connection. I have had only one unexplained failure in the last three years that I have seen and couldn't explain. However, that was when we didn't have CSAs checking to see if teams have shot themselves in the foot.
All the people I respect on this forum tell me that the power supply to the radio cannot be the problem and my gut tells me the same thing.

As to what ferrite beads may work in keeping RF out of the radio when used on the power wiring or ethernet connections. I know that there are a couple of snap on chokes available from Radio Shack that could work. Others that I know of are available from Ham Radio Internet sites but is unlikely at this late date you get them in time.

Re: Intermittent connection on field only
 

Average current drawn wouldn't be of value. That's why I designed a circuit fast enough to see momentary changes in the input voltage like an oscilloscope.

Turn on or off all the features you like. There's still a maximum and minimum current the unit can draw. I'm not characterizing what it draws at each operation, only what the maximum or minimum peak is.

Don't need anything like this to test what I've asked. Could do it in an old microwave with the electronics removed and even reuse the metalized stirrer fan from the magnetron section if you want to get funky with it.

The inside of a microwave oven is an RF shielded box (with some nice leaks around the door). It'll reflect and absorb 2.4GHz well on it's way up to 5GHz. The metalized plastic stirrer fan would do what it does to the microwaves from the magnetron and form an interference pattern.

You don't need that sort of test equipment to characterize the current input on the DC power supply. You just need to max out the radio current draw. You'd need a anechoic chamber if you wished to reverse engineer the details of the radio to specifics and I've never asked for that at all.

I wasn't looking to test the details of the radio link this is another straw man. I was looking to see if the radio is on and how much it and the rest of the unit can draw above that point to look for wiring issues and to generally make power quality issues locatable. I detailed how concisely already and clearly it was all ignored.

I bet you didn't intend this statement the way it appears from my perspective. Quantifiable data isn't a popularity contest. Making it into one and silencing alternate opinions isn't a great idea. In fact it's the worst possible idea once we know that the data we need, from the environment we need it in, doesn't exist so you are speculating as much as you might claim I am (and more importantly the very people who's opinions you respect have effectively been the reason why I know the data I seek wasn't considered by their own communications in this topic).

As I feel I'm being effectively bullied on this subject in a way that can't produce anything of value for FIRST. Since I've publicly asked for the topic to be split (and I can't even start a topic myself as I've pointed out) to remove the cover distraction that I'm interfering in the discussion and my request has been ignored. Since it appears that the parties involved with the most access don't have the data I require. Since I have now been stalled to prevent my use of the community to get that data. Since some seem intent on ignoring my posts while complaining they are too long. I'm going to just drop the subject which was apparently the mission statement of the day here. That doesn't really serve the engineering or scientific interest of FIRST but it does serve to let me get some of the information I wanted without any further opportunities for interference.

If FIRST allows the use of the cRIO to log that data on the Championship field I hope someone actually does it.

You know the sort of funny part is that it doesn't matter if FIRST uses my little comparator modules or not officially. They have so many applications that getting money to develop them seems no issue at all. I wonder why there's so much interest in an idea people claim can't be applied.

Re: Intermittent connection on field only
 

Brian,
You already posted that high frequency changes in current would not be visible due to all the low frequency filters in circuit. I agree. You won't see them so that gives you no data as I pointed out above.
There are features that you cannot control and you cannot predict so again there is no way to collect meaningful current data.
Microwave oven? It is not an RF proof chamber it is simply a cavity with an opening on one side. Most ovens use a 1/4 wave shorted stub to prevent leakage. And you will not be able to test real world activity since you won't have another radio that is connected to the wireless. The SWR that the Dlink will experience in this environment is likely to be extreme and changeable with a small movement within the cavity.

Above all, I don't want it to seem that I am bullying you. I do have a vested interest in preventing the others who read this discussion to be distracted from collecting meaningful in situ data and understanding the real problems when they exist. If there is a real problem I (and FIRST) need more than the sampled data of the current draw on the radio collected in a sea of variables. We (Inspectors, First Engineering, National Instruments and CSAs) require hard data. i.e. did the radio stop transmitting, did the Crio reboot, did the radio reboot, did the CPU hit max usage, did the battery fall below 4.5 volts, what language is the team using to program the radio, can we see the code?
Collecting this data will point us to a particular component failure or to a software issue that hopefully will be repeatable under real conditions.

Re: Intermittent connection on field only
 

The high frequency isn't what I'm worried about, it's the transients that drop the current because for a moment a loose wire would disconnect the circuit or block it with a high resistance. That's why I used the alarm circuit example. I'm not interested in measuring RF bleed-in or anything exotic like that if I was I'd bring my spectrum analyzer and the near field probes.

All those features do so far as I'm concerned given all the filtering on the point in the circuit I'm interested in is consume power and I'm only looking at the maximum and minimum peaks...I don't care if the window between is 1A or not. I don't need to know exactly when anything in the D-Link turns on or off *except* when the radio first turns on during the initial robot power-up sequence (I only need to know about that because for a bit of time the D-Link as a unit will draw much lower power).

I know that's the point. I want to max it out so it draws maximum current. The whole point is when you remove the microwave outer case and the magnetron assembly you'll be left with an opening in the case (usually a grid of holes) and sometimes a plastic metalized stirrer fan. The D-Link AP inside that cavity would have to drive it's RF section into the max region to drive a connection to a wireless device outside the microwave past the stirrer fan which itself would cause interference. Of course some microwaves don't have metalized plastic stirrer fans but any metal bladed fan would work, even moving tin foil on a motor. In any case during the test you leave the outside case of the microwave off...the point is not to contain all the RF signal but to make the D-Link AP work very very hard to communicate to the wireless network outside (a point not served by putting the outer lid back on and effectively building an RF shield around the interior shield).

I completely understand and hence my request to be offered a different topic so we can discuss this specific issue without in any way ignoring the other valuable advice you wish to insure is communicated.

If you look back in this topic to page 10 towards the bottom I made a nice summary of exactly what I thought this data would do for you. I prefer to use my circuit to do this test for reasons I've stated repeatedly including there's no data to trawl through just maybe 4 LEDs...but if I can't get that I'd love to see the data log later and I'll be interested.

Exactly and what I'm doing is collecting one subset of data that is particularly of value in the context I already feel I summarized quickly last night back towards the bottom of page 10.