View Full Version : COMM lost way to many times
We competed at Buckeye this past weekend and suffered comm issues in most of our later matches on Friday and Saturday.
We had lots of folks trying to help us figure it out from the FTA's and other field personnel to mentors from other teams.
Here is what I know.
From looking at logs and reported observations from the FTA's they see the radio rebooting. Immediately after they see the radio return they see the RIO so the belief is that the RADIO is the only think rebooting as the RIO takes longer than the radio and would not show up immediately after the radio if it was also booting.
Even so. The initial thought, before we found out the above, as that we had loos battery connections. We went over all our batteries and did find some that were marginal so we tightened ALL our battery connections regardless if they were loose or not. This did not solve the problem. The problem could not be duplicated in the pit no matter what we wiggled and banged on. We build a practice bot and had no issues with that all preseason. We had the radio from that along so we swapped it in and no joy with that either.
We pulled and re-seated all the wires in the weidmuller connectors on the PDP and VRM and tug tested. Again, no joy. By this time we were past our second match of elims and got subbed out. Not a problem, I would have done the same thing.
Before we bagged we replaced the radio power cable with the one we used on our practice bot and replaced the VRM with a brand new out of the box new one.
Have not had a chance to test since.
Does anybody have any idea what else we should try if this does not work?
Thanks
have you guys checked the mini automotive fuse. my team just unbagged our robot yesterday and it drove fine until we tried to go over any defenses. those fuses can come loose but still look like they are seated. hope this helps.
We had a similar problem and discovered that one of the fuses on the PDP was loose. (I believe one of the small fuses at the bottom of the PDP goes to the VRM?)
After making sure it was securely pushed in, we didn't have further problems.
That might be something to look at
Did you check that the yellow and red fuses were pressed all the way into the PDP? When they are inserted fully, there should only about 1/4 inch of the plastic showing. In Kansas City, there were a lot of robots with similar issues. The CSAs at the event checked all of the robots Friday afternoon and a lot of the roboRIO and Radio issues went away.
we had several folks look at the fuses and they were deemed OK but I will pull them and insert new ones Thursday.
Thanks
MrRoboSteve
21-03-2016, 12:14
When does the radio reboot occur? Can you post links to matches where the radio is rebooting?
1. Since you've replaced everything else, consider replacing the wire from the PDP to the VRM. Be sure you're using stranded wire.
2. Make sure that all of the wire along the path is tacked down with zip ties (use zip tie bases if needed). You want to minimize wiggling.
3. How is your radio fastened to the robot?
Joe Johnson
21-03-2016, 12:24
I know I am a flame thrower on this topic so I should probably just sit on my hands and watch this thread, but I will say this much, as a fan in the stands at UMass Dartmouth over the weekend, it was galling to watch team after team (and probably more veteran teams than rookies) sit dead on the field for close to a minute while the radio reboots (or reconnects or whatever -- their dead and that orange light isn't flashing so you can be sure that they are not coming back alive for half the match if at all).
I know, I know, I KNOW, it is almost always a power issue for the radio. I hear you, FIRST, but seriously, you need to stop blaming the teams on this matter.
When you run a remote controlled robot tournament, the first and primary contract you make with the participants that you promise them a solid data link to their robot.
I believe that FIRST is failing in this promise.
I argue that we are at the point where we need to either
find an idiot proof solution (that a team can do whatever they want and the radio link stays alive)
or
inspect in the set of behaviors that gets to a solution (e.g. every robot going on the field gets inspected for the proper wiring connections, strain reliefs, etc.).
Blaming teams was never a great plan but this year with such long reconnection times, it is just unsupportable.
Dr. Joe J.
The problem is solved with hot glue.
Thad House
21-03-2016, 12:40
I know I am a flame thrower on this topic so I should probably just sit on my hands and watch this thread, but I will say this much, as a fan in the stands at UMass Dartmouth over the weekend, it was galling to watch team after team (and probably more veteran teams than rookies) sit dead on the field for close to a minute while the radio reboots (or reconnects or whatever -- their dead and that orange light isn't flashing so you can be sure that they are not coming back alive for half the match if at all).
I know, I know, I KNOW, it is almost always a power issue for the radio. I hear you, FIRST, but seriously, you need to stop blaming the teams on this matter.
When you run a remote controlled robot tournament, the first and primary contract you make with the participants that you promise them a solid data link to their robot.
I believe that FIRST is failing in this promise.
I argue that we are at the point where we need to either
find an idiot proof solution (that a team can do whatever they want and the radio link stays alive)
or
inspect in the set of behaviors that gets to a solution (e.g. every robot going on the field gets inspected for the proper wiring connections, strain reliefs, etc.).
Blaming teams was never a great plan but this year with such long reconnection times, it is just unsupportable.
Dr. Joe J.
I would LOVE to see FIRST legalize opening up the Radio and soldering a power cable directly to the PCB pins. Not having a solid, locking connector is the biggest issue with the radio, and if teams could solder to create their own locking connector, maybe with a powerpole that would be great.
Lil' Lavery
21-03-2016, 12:41
1712 had too many simultaneous fixed to truly isolate which mattered and which didn't (we didn't have time to only change one variable at a time in the heat of competition). But I do believe that getting better strain relief on wires was a big part of us solving our roboRIO power issues, and may help you with your radio issues. Ensure your wiring has a sufficient bend radius near either end, and then get it tied down securely. Even momentary losses of power can cause reboots or brown outs. Check (and possibly replace) both your power and cat5 connectors. We also added an elastic band (hairtie) around our radio to help all the connectors stay plugged in. Further still, we cushioned our radio mount with foam to help absorb shock.
Conor Ryan
21-03-2016, 12:49
This sounds like a power quality issue. Bad wires, bad crimps, bad connections, bad barrels all can contribute to this.
I would replace all of the wires from the batter to the radio, especially the black wires that go directly into the radio. Then I would reprogram your firmware on the radio in particular. Inspect your fuses, many people don't notice when they are bad. Drive the robot around at home, drive over bumps see if you can replicate these issues. Make sure all of your wires are anchored down appropriately and nothing can shake loose. Make a good effort to replicate the driving conditions on the field, move around fast and crash frequently.
With how difficult the field is on robots this year, i'm not surprised that we are seeing a rise in power quality related issues on the communications.
protoserge
21-03-2016, 12:57
What is the battery voltage showing on the logs? How old are your batteries?
[Bad advice retracted].
Try driving the robot in a stall condition by putting it against a wall and driving forward. See if your radio restarts.
Thad House
21-03-2016, 13:03
If you haven't already, pick your battery up by the cables. If it pulls the wires out, your crimps are bad. Don't make a habit of lifting batteries by the cables though.
Please don't do this. Even doing this once is enough to damage the terminals on the battery even if the crimps are lose. Its much better to take them off of the battery, and test the crimps off the battery.
Also teams, if you are crimping your own Andersen connectors, you can actually purchase a hydraulic crimper from Harbor Freight for not too much money. We bought it a few years ago, and love it. Haven't had a lose crimp in years, and we even cut a few apart to inspect the test crimps and they are really solid.
http://www.harborfreight.com/hydraulic-wire-crimping-tool-66150.html
protoserge
21-03-2016, 13:12
Please don't do this. Even doing this once is enough to damage the terminals on the battery even if the crimps are lose. Its much better to take them off of the battery, and test the crimps off the battery.
Also teams, if you are crimping your own Andersen connectors, you can actually purchase a hydraulic crimper from Harbor Freight for not too much money. We bought it a few years ago, and love it. Haven't had a lose crimp in years, and we even cut a few apart to inspect the test crimps and they are really solid.
http://www.harborfreight.com/hydraulic-wire-crimping-tool-66150.html
That is a valid point. I would recomend pull testing them a better way.
When does the radio reboot occur? Can you post links to matches where the radio is rebooting?
1. Since you've replaced everything else, consider replacing the wire from the PDP to the VRM. Be sure you're using stranded wire.
These wires are only about 6 inches long an pretty hard to wiggle but I will look at that
2. Make sure that all of the wire along the path is tacked down with zip ties (use zip tie bases if needed). You want to minimize wiggling.
3. How is your radio fastened to the robot?
with velcro not the regular stuff that kind that sticks to itself whatever that is called.
Been doing that for years
The problem is solved with hot glue.
hot glue where?
This sounds like a power quality issue. Bad wires, bad crimps, bad connections, bad barrels all can contribute to this.
I would replace all of the wires from the batter to the radio, especially the black wires that go directly into the radio. Then I would reprogram your firmware on the radio in particular. Inspect your fuses, many people don't notice when they are bad. Drive the robot around at home, drive over bumps see if you can replicate these issues. Make sure all of your wires are anchored down appropriately and nothing can shake loose. Make a good effort to replicate the driving conditions on the field, move around fast and crash frequently.
With how difficult the field is on robots this year, i'm not surprised that we are seeing a rise in power quality related issues on the communications.
ya but the robot is in the bag
What is the battery voltage showing on the logs? How old are your batteries?
[Bad advice retracted].
Try driving the robot in a stall condition by putting it against a wall and driving forward. See if your radio restarts.
we have all kinds of batteries but we lost comms some times when we were just going under the low bar. No big current draws, no big bumps.
We had some issues with our radio and/or FMS at the Tech Valley Regional this weekend.
One of them was on us (tug test failed on our radio power connector), the other was indeterminate. (CSA/FTA was blaming robot power, as a whole, but we have video showing no power loss on the RoboRio.)
Regardless of what the problem is, even as a rookie team, the instrumentation provided by the RoboRIO/PDU/FMS/DS completely fails during problem events.
During the aftermath of our issues, we attempted to use the DS logs to trace voltage, power draw, comms, and log messages from the RoboRIO. The problem is, you can make "guesses" about your voltage state in the fraction of a second leading up to the event, but once it happens, you're blind.
We're already redoing our robot code library for next year to address some of these shortcomings. We'll be logging voltage and PDU stats, as well as bridge/radio and FMS connectivity via ethernet directly on the RoboRIO. In addition we'll be using a cached-logger, such that if DS communication is lost we can back-fill in our logs.
We are attempting to address the issue of the complete lack of FMS logging, and the logging dropout during any communication loss event. We're also pushing the logging closer to the source of the data. Our goal is to be able to document the voltages, RoboRIO, PDU, and Radio/FMS/DS status in an effort to diagnose issues.
Do a pull test on all wiring connections. That means where a wire goes into a Weidmuller or Wago connector, where a wire goes into a crimp lug, where a wire goes into an Anderson connector and where the wires go into the plug for the RoboRio power. If wire slips out, re-do the connection.
Check all screw/lug connections such as the battery cables on the battery and on your main breaker. The wire and lug should not be able to rotate relative to the battery or breaker. Tighten the screws if they move.
Use a flashlight and examine closely where the wires go into the small Weidmuller connectors (i.e. RoboRio power on your PDP). Look for stray strands on any wire that is not in the connector. These stray strands can touch adjacent wires which is always the opposite polarity on the PDP, VRM and PCM. If necessary, cut off the "crinkly" stripped ends, strip to the length shown in the Users Manual for the PDP, VRM and PCM, twist the exposed strands so they lay neatly together then re-insert the wire in the Wiedmuller connector carefully and ensuring that ALL the strands go in.
Lil' Lavery
21-03-2016, 13:52
We had some issues with our radio and/or FMS at the Tech Valley Regional this weekend.
One of them was on us (tug test failed on our radio power connector), the other was indeterminate. (CSA/FTA was blaming robot power, as a whole, but we have video showing no power loss on the RoboRio.)
What in the video establishes that there was no power loss on the RoboRIO? If you're just looking at the Robot Signal Light, it's possible that your RoboRIO browned out due to power loss, but didn't completely shut off.
hot glue where?
We solder and heat shrink where we can, and hot glue everywhere else. So the connections to the Voltage Regulator are all hot glued. We were seeing all sorts of radio drops out just as you've described on both our comp and practice bots. Since hot gluing the radio wires into the voltage regulator we haven't lost comms. (~19 matches).
We use hot glue on nearly every connection, including our encoder cables, limit switches, arduino plugs, etc.
If you want something less permanent electrical tape tends to work well enough, but I haven't seen anything better than hot glue this year.
So everyone knows, the DS logs are able to tell you if the roboRIO rebooted.
When a connection is established between the roboRIO and the DS the roboRIO will send a message saying how many seconds have passed since it last booted up, the DS then stores this information in the log. Look at the messages when communication is reestablished to find this information.
Alan Anderson
21-03-2016, 15:18
I would LOVE to see FIRST legalize opening up the Radio and soldering a power cable directly to the PCB pins. Not having a solid, locking connector is the biggest issue with the radio, and if teams could solder to create their own locking connector, maybe with a powerpole that would be great.
Why is everyone missing the obvious solution? Use passive Power Over Ethernet (POE) on the port that supports it. The Ethernet connector snaps firmly into place. I've never seen one come loose due to bounces or vibration this year.
On the other hand, more than half the robots I checked had loose terminals either on the battery or the main circuit breaker. All of the "lost communication" events I followed up on at both of the events I worked at were associated with a faulty power connection. All.
(I'm counting only ones where there was actually a loss of communication, not the few the drivers called "lost comm" but where the problem was that robot code crashed or a joystick came disconnected or the Driver Station was being interfered with by a software update process.)
Why is everyone missing the obvious solution? Use passive Power Over Ethernet (POE) on the port that supports it. The Ethernet connector snaps firmly into place. I've never seen one come loose due to bounces or vibration this year.
On the other hand, more than half the robots I checked had loose terminals either on the battery or the main circuit breaker. All of the "lost communication" events I followed up on at both of the events I worked at were associated with a faulty power connection. All.
(I'm counting only ones where there was actually a loss of communication, not the few the drivers called "lost comm" but where the problem was that robot code crashed or a joystick came disconnected or the Driver Station was being interfered with by a software update process.)
I'm not familiar enough with PoE to be comfortable without some details. I sure don't want to fry a radio. With a quick search I haven't found out the best way to wire a PoE for this this radio. I assume you can use the 12 v port on the VRM into a spliced ethernet cable but what pins are used and which port does it go into in the router. I apologize if this has already been discussed. I didn't see it in my searching, but might have been looking for the wrong things.
FIRST should consider allowing the use of a small secondary battery to power the radio should it lose connection with the VRM/PDP. The battery would be used exclusively for the radio and prevent any reboots due to power loss or brownouts elsewhere on the bot.
Consider that batteries and the PDP are generally well-hidden on the bot (the battery to lower CG and the PDP to shorten total wire lengths. However the radio needs good clearance from metal and motors to ensure a solid RF connection, leading to longer wires and generally less secure mounting (subject to more vibrations potentially). This all introduces risk of wire/connector failure. Allowing a small battery placed alongside the radio as a parallel backup would be beneficial.
Alan Anderson
21-03-2016, 16:37
With a quick search I haven't found out the best way to wire a PoE for this this radio.
All the information I needed I found through the OpenMesh web site and a simple search for "passive POE". I summarized it here (http://www.chiefdelphi.com/forums/showpost.php?p=1560595&postcount=12).
hot glue where?
Hot glue at the connection between the power plug and the radio.
I was the mentor from 379 that was trying to help. At the time we probably couldn't wait for a hot glue gun to heat up We reset all wires at the VRM. Checked other connections, inspected contacts, and zip tied the battery connection. Not knowing the robot, it was hard to inspect.
I do feel that it's something dealing with an electrical connection. Try replacing the Anderson battery/PDB connection, wire and all, from the robot. I wish we could have figured it out, you guys have a great design, good luck this season.
KrazyCarl92
21-03-2016, 16:54
Use passive Power Over Ethernet (POE) on the port that supports it.
Which port on the radio supports POE?
Bryan Herbst
21-03-2016, 16:59
I highly recommend against hot gluing the plug into the radio (and I also advocate against hot gluing or taping ethernet cables and power cables pretty much anywhere). It makes replacing those wires later more difficult.
The radio's plug is particularly problematic with hot glue. You can no longer easily detach the radio from your robot to program it at events, and swapping out radios should yours need to be replaced for any reason suddenly becomes much more difficult.
Furthermore, the issues I have seen are generally not that this plug comes loose. The problems are generally that the wires going into the Weidmuller connections on the VRM have too much copper showing and are shorting out, or that there are loose wires elsewhere in the chain.
I had a team that had mummified their Ethernet ports on both the radio and roborRIO along with the USB cables going into the roboRIO and the power plug on the radio with electrical tape drop out during a match. Seriously, there was a large mound of the stuff over everything. They asked me if I thought they should add more electrical tape since they were still dropping out. At that point I demonstrated that the wires going to their radio were just falling out of the VRM with very little effort on my part.
Understand the problem before trying to apply bandaids. You might not even be putting the bandaids in the right spot.
What in the video establishes that there was no power loss on the RoboRIO? If you're just looking at the Robot Signal Light, it's possible that your RoboRIO browned out due to power loss, but didn't completely shut off.
I was going on the basis that our battery voltages were very good (and trending up), and our load was low at the time (and trending down). It was the CSA & FTAs opinion that we lost power completely and the RoboRIO rebooted, however the video of the lights on the RoboRIO (and RSL) didn't concur with that assessment.
Is it possible it was a brown-out, sure, but my point is that there isn't enough data because of the streaming-only nature of the logging mechanisms.
I highly recommend against hot gluing the plug into the radio (and I also advocate against hot gluing or taping ethernet cables and power cables pretty much anywhere). It makes replacing those wires later more difficult.
The radio's plug is particularly problematic with hot glue. You can no longer easily detach the radio from your robot to program it at events, and swapping out radios should yours need to be replaced for any reason suddenly becomes much more difficult.
Furthermore, the issues I have seen are generally not that this plug comes loose. The problems are generally that the wires going into the Weidmuller connections on the VRM have too much copper showing and are shorting out, or that there are loose wires elsewhere in the chain.
I had a team that had mummified their Ethernet ports on both the radio and roborRIO along with the USB cables going into the roboRIO and the power plug on the radio with electrical tape drop out during a match. Seriously, there was a large mound of the stuff over everything. They asked me if I thought they should add more electrical tape since they were still dropping out. At that point I demonstrated that the wires going to their radio were just falling out of the VRM with very little effort on my part.
Understand the problem before trying to apply bandaids. You might not even be putting the bandaids in the right spot.
Great callout Bryan. As he said, you should not have to apply hot glue on the connection between the radio and the plug, and that can definitely be problematic. We used hot glue to aid the Weidmuller connectors on the voltage regulator, but it certainly has drawbacks.
I'm a big fan of Alan's PoE idea for this particular problem
Alan Anderson
21-03-2016, 18:19
Which port on the radio supports POE?
Both Ethernet ports on the OM5P-AN support POE, each in a different way. The one nearest the power connector is labeled "18-24v POE", but the bottom of the device says "PoE Input: 12-24V Passive PoE" and it definitely works with the 12 volt output of the VRM. The other connector works with the 802.3af POE standard, which uses 48 volts -- higher than what is permitted by the robot rules.
It's a shame the radio programming kiosk specifies the second port. It would have been handy to use a cheap passive power "injector" and only require teams to connect the Ethernet cable in order to make it work.
Alan Anderson
21-03-2016, 18:49
We used hot glue to aid the Weidmuller connectors on the voltage regulator.
I imagine am certain that a wire properly inserted into a Weidmuller connector will hold more strongly than hot glue.
I just tested it by putting an 18 gauge wire in the +12Vin connection of a VRM, gluing the wire to a scrap of Lexan, waiting several minutes for the glue to harden, then easily pulling it off by lifting the VRM. I did it again with a piece of aluminum. I had no problem holding on to the hardened blob of glue and using the VRM to apply enough tension to pull the wire through the glue.
Most of the wires I saw falling out of Weidmuller connectors were not stripped nearly long enough. It should be 8mm (about 5/16") of exposed conductor.
If you have one of the power cables for the radio that has a 90 degree plug (right angle plug), you can wrap a long zip-tie around the radio and the plug. That would be better than gooping up the power plug with hot glue.
One common problem (that we had last year) which can lead to temporary power loss long enough to restart your radio is the main breaker. From what I've heard, there are a few every year that have this issue.
Try tapping your finger on the red button on the breaker, (the one you would use to power off your robot) if it causes your robot to power cycle, that's your issue - at least one of them. :)
I was going on the basis that our battery voltages were very good (and trending up), and our load was low at the time (and trending down). It was the CSA & FTAs opinion that we lost power completely and the RoboRIO rebooted, however the video of the lights on the RoboRIO (and RSL) didn't concur with that assessment.
Is it possible it was a brown-out, sure, but my point is that there isn't enough data because of the streaming-only nature of the logging mechanisms.
Does that video also show the Radio not losing power and rebooting? How long was the robot down for? What was the RSL doing at the time? A radio reboot is about 50ish seconds and then 10-15 or so more for the roboRIO to attach to the DS.
Do you have a link to the video itself? I'd like to see it for my own knowledge when I'm monitoring the field for my last 3 events this year.
stoner77
22-03-2016, 01:41
Not for sure if somone already said this; apparently the breakers that came in some of the kit if parts are bad. Our team had the exact same problem along with a couple others. We didn't know until a team informed us on our last match, we swapped it, then it worked pirfectly. Wiggle the red switch button to test it if all/some of your communications go down.
We were having this same issue at the Australia Regional. After a while, we narrowed it down to the following 2 things:
a) The Circuit Breaker (and the wire that connects it to the PDP)
b) The Anderson Battery Connectors going from the Breaker/PDP to the Battery's Anderson Connector.
Replacing these seemed to solve our problem for all our future matches. Some breakers can be very sensitive to sudden stops (i.e. crashing into a defense), and faulty Anderson connectors was discussed in another thread.
EDIT: We diagnosed this by looking at the DS. If the comms go down, then comms go up and (later) code goes up (checking DS logs to confirm the signal loss was indeed with the radio), it means the total system did a reboot (radio AND rio), so you can usually diagnose that as the main power feed to the PDP (or the PDP itself), as the RIO and VRM (connecting the radio to power) are on separate connectors.
Al Skierkiewicz
22-03-2016, 08:03
OK here goes a list of problems and fixes...
Battery terminals even if tight can be intermittent for a variety of reasons. A simple fix is to add a #10, external tooth star washer between the wire terminal and the battery terminal when assembling the battery wiring. This simple addition will bite through any surface crud on the terminals and lock the two terminals together so that they cannot rotate. Use the provided locking hardware in addition to this one lock washer.
Push the PD fuses all the way in. When properly installed the fuse will be just 1/8" above the PDP surface. It should be hard to remove.
Strip the wires that are inserted into the various Weidmuller connectors to the required length. Often teams make these too short and the result is the wire is not properly retained. Do not tin the wire, these connectors are intended for bare stranded wire. If you pull the wire and it pulls out of the connector suspect the connector. I have seen these damaged by aggressive installers.
A small number (<1 in 100,000) of main breakers have a manufacturing defect. While the robot is on, lightly tap the red button. If your lights flicker, replace the main breaker.
Make sure that the terminals you use to feed the PDP input seats all the way down into the PDP terminals. Some terminals are a little too wide to fit the terminal and will not seat all the way down. This raises the series resistance considerably. Inspect these terminals with a bright light.
Use proper strain relief at each end of the wire. We use self adhesive wire tie blocks or wire tied directly to frame to keep everything from moving around.
This year's radio has antennas behind the vents on either side of the radio. Do not mount your radio near metal objects so that the vents are near the metal objects on your robot.
Observe when your radio reboots. If you have the issue when traversing the defenses, then it is likely a poor connection. If it occurs when turning, then it is a brownout issue.
Never, never, not ever, lift the battery by the wiring. The terminals on the battery are simply soldered in place inside the battery. This is a manufacturing method that allows attaching various terminal styles to the same basic battery. Lifting the battery by the wire will break the solder connection internal to the battery. You will never know you have permanently damaged the battery until you really need it.
I do not recommend hot glue for many reasons. It might give you a false sense of security and will make you look elsewhere when the problem is really under the hot glue all along. If you think about it for a few minutes you can come up with a much better solution.
If you do not use the power plug that came with the radio, then it is very likely you have the wrong diameter plug. They look alike but the center pin varies.
Try tapping your finger on the red button on the breaker, (the one you would use to power off your robot) if it causes your robot to power cycle, that's your issue - at least one of them. :)
Oh god this brings back bad memories from a couple weeks ago... At Q31(?) of the NYC regional we had some mounting issues with our top plate. Before we had proper time to fix it, we had to queue up for Q37(?), so we decided to go the match without it. After our partner pushed us out of the moat, our pressure release valve (which was normally mounted to the top plate) swung and hit the red button on the breaker.
That was a sucky way to lose a match, but hey. It happens.
OK here goes a list of problems and fixes...
...snip...
This is a great list, and pretty much the way I trouble shoot robots at competition with "loss of COMMS".
I will also add, Check the Ethernet connection on the laptop. IE, give it a wiggle. I found this was the source of intermittent comms drop for a couple teams at Centerline this past weekend.
This laundry list of items shows why it is such a frequent problem. While the folks at CTRE and NI and FIRST have done a great job at reducing the possible Idiot proofing, there are still so many ways that you can get an issue that low probability of many many things turns into a reasonably high probability that something will get you.
I.E. a 10% chance on any one item, but 10 items with that 10% chance turns into a pretty high probability that something will go wrong.
Pro-tip, when forming your alliance, ask if you can go through your partners robot with a fine tooth comb, and ask them to do the same for you. Often an extra set of eyes can help find issues and errors. Make you own checklists and review them often. Not sure what to put on the list? How about the good advice being dispersed in this thread.
These events the robots are going over are pretty violent. Even the weight of a wire or cable flopping around can be significant when the robot see a 20G pulse (1 lb acts like 20 lbs). Proper restraint and strain relief goes a long way.
One common problem (that we had last year) which can lead to temporary power loss long enough to restart your radio is the main breaker. From what I've heard, there are a few every year that have this issue.
Try tapping your finger on the red button on the breaker, (the one you would use to power off your robot) if it causes your robot to power cycle, that's your issue - at least one of them. :)
is it possible that the main breaker problem you reference can cause the radio to reboot without the RIO rebooting?
All indications are that our RIO is not rebooting.
We competed at Buckeye this past weekend and suffered comm issues in most of our later matches on Friday and Saturday.
We had lots of folks trying to help us figure it out from the FTA's and other field personnel to mentors from other teams.
Here is what I know.
From looking at logs and reported observations from the FTA's they see the radio rebooting. Immediately after they see the radio return they see the RIO so the belief is that the RADIO is the only think rebooting as the RIO takes longer than the radio and would not show up immediately after the radio if it was also booting.
Even so. The initial thought, before we found out the above, as that we had loos battery connections. We went over all our batteries and did find some that were marginal so we tightened ALL our battery connections regardless if they were loose or not. This did not solve the problem. The problem could not be duplicated in the pit no matter what we wiggled and banged on. We build a practice bot and had no issues with that all preseason. We had the radio from that along so we swapped it in and no joy with that either.
We pulled and re-seated all the wires in the weidmuller connectors on the PDP and VRM and tug tested. Again, no joy. By this time we were past our second match of elims and got subbed out. Not a problem, I would have done the same thing.
Before we bagged we replaced the radio power cable with the one we used on our practice bot and replaced the VRM with a brand new out of the box new one.
Have not had a chance to test since.
Does anybody have any idea what else we should try if this does not work?
Thanks
we had this issue at wpi 2 weeks ago, in all of our qualification matches except for two we lost communication we changed and checked everything, fortunately we were alliance captains, and going into our final quals match we simply decided to move the radio to a different spot and didn't have an issue with it after that something so simply cause suck grief .
All the information I needed I found through the OpenMesh web site and a simple search for "passive POE". I summarized it here (http://www.chiefdelphi.com/forums/showpost.php?p=1560595&postcount=12).
Ok I read that link and I want to make sure I understand how you are using this.
You take a port saver cord, open the cable somewhere and solder a red wire to the blue and white with blue stripe wires and a black wire to the brown and white with brown stripe wires then you stick those wires into the VRM, plug the port saver into the POE port on the radio and the ethernet cable from the RIO into the end of the port saver cable you just modified so you are using only ONE of the ethernet ports on the radio, correct?
Do you cut the wires in the port saver so power only goes to the radio or do you just solder to those wires so power goes to both the radio and the RIO?
Thanks
pfreivald
22-03-2016, 10:54
We had this exact problem at TVR in our first four matches.
It wasn't the fuses.
It wasn't the wire connections.
After isolating and ruling out all other issues (and reproducing the fault by giving the PCP a shake), we swapped the PCP and the problem didn't repeat. [Side note: the PCP, barely used since coming out of the kit, rattled like a maraca when shaken. Back home, the simplest wiring setup possible, very secure, would lose voltage when shaken.]
In our second-to-last qualifying match, half of our pneumatic system failed to actuate. We checked our wiring and all seemed fine, and it worked in the pit (even after some jostling)--we couldn't reproduce the failure. It failed the next game as well, and we couldn't reproduce the failure. After QF 1 we swapped the PCM for a new one and it worked fine through our last two matches. We have not at this time tried the PCM in other situations.
It's aggravating to have failures in KOP devices that we're required to use on our robot, but cannot repair or upgrade to something more reliable. FRC requires robust components, and expecting teams to deal with internal failures of KOP parts used in a manner consistent with their intended purpose as "part of the game" is not acceptable--quality control on FRC KOP components should be top notch, and it simply isn't.
(The "tested" sticker on the back of the PCP was a bit of an insult-to-injury chunk of irony. They said it was tested...but never said whether or not it worked.)
Alan Anderson
22-03-2016, 11:10
Ok I read that link and I want to make sure I understand how you are using this.
You take a port saver cord, open the cable somewhere and solder a red wire to the blue and white with blue stripe wires and a black wire to the brown and white with brown stripe wires then you stick those wires into the VRM, plug the port saver into the POE port on the radio and the ethernet cable from the RIO into the end of the port saver cable you just modified so you are using only ONE of the ethernet ports on the radio, correct?
We plug the port saver into the roboRIO, not the radio. The power wires only need to reach from near the roboRIO to the VRM. Usually the radio is mounted some distance from the rest of the control system components, and it's convenient to run a single cable from the port saver up to the radio.
The Ethernet port next to the coax power input is the only port on the radio that accepts this scheme.
Do you cut the wires in the port saver so power only goes to the radio or do you just solder to those wires so power goes to both the radio and the RIO?
The blue and brown pairs from the "plug" side of the port saver are indeed disconnected in the one we have installed on the robot, but that was mostly for ease of construction. I have used a different "passive injector" during testing and verified that having power going to those pins on the roboRIO as well does not break anything.
is it possible that the main breaker problem you reference can cause the radio to reboot without the RIO rebooting?
All indications are that our RIO is not rebooting.
bump
RufflesRidge
22-03-2016, 11:46
bump
Extremely unlikely. Can you elaborate on the indications? The implication you were provided by the FTA\CSA earlier in the thread that the roboRIO takes longer to boot than the radio is false.
Extremely unlikely. Can you elaborate on the indications? The implication you were provided by the FTA\CSA earlier in the thread that the roboRIO takes longer to boot than the radio is false.
Correct that the roboRIO boots quicker, but if both of them went down, then communications between the roboRIO and DS aren't going to come back until 10-15 seconds after the radio is fully back up (50 seconds or so itself). Net is 60-65 seconds before the robot moves again. That may be what the FTA was trying to say.
Joe Johnson
22-03-2016, 13:06
Correct that the roboRIO boots quicker, but if both of them went down, then communications between the roboRIO and DS aren't going to come back until 10-15 seconds after the radio is fully back up (50 seconds or so itself). Net is 60-65 seconds before the robot moves again. That may be what the FTA was trying to say.
Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?
I get the feeling that the problems are worse this year but I freely admit that I am a biased observer. If FIRST has data that shows that this problem is small and getting smaller then, well, I won't say I'll be happy, but I will at least feel better knowing that the ball is moving toward the right end zone.
Dr. Joe J.
mdballard
22-03-2016, 13:46
We had issues with our development robot. It would drop the radio almost any time we went over a defense. All of our other components stayed powered. The RIO never rebooted. We did several things to harden the control system:
1) made an improvised shock plate to mount the radio to. This dampens the shock transferred to the radio itself. There seems to be a strong indication that high G shock, as opposed to vibration, is an issue for the new radio.
2) We use a plug with screw terminals for the radio power. Once the wires were screwed into the terminals on the plug adapter, we soldered the wires in and insulated the leads. We verified isolation between the leads. Then zip tied the plug adaptor to the shock plate such that: 1) the tension relief moved as a system with the shock plate and radio, 2) would not move relative to the radio itself. We also zip tied the Ethernet cables to the shock plate.
3) For all wiring connections, we made tension reliefs, out of semi-rigid lexan and non-conductive foam tape, that could be zip tied to the component with the connections. This greatly reduced the movement of the wires in the connectors relative to their connected components.
4) We made sure the radio had physical separation from other CAN bus connected components. We found, as some other teams, that if the radio was placed too close to the PCM it appeared to interfere with the CAN transceiver in the device. At least that is how the issue was manifesting from observation without opening the PCM itself. When we relocated the radio, the issues went away. This was not a problem with last years radio.
5) We use 45amp power pole connectors from our TalonSRX controllers to the motors.
6) We use star rings on the main breaker and ensure the connections are tight. We fortunately did not have a faulty main breaker, but that is something that can be checked as described by others in this thread.
7) We perform a check on the robot before it leaves the pit prior to every match.
In the STL regional we didn't have any loss of communication with the robot during any of our matches.
We did have a problem before our first qualification match where the Ethernet port on our driverstation laptop was accidentally broken. But that had nothing to do with the robot. We fixed that issue with a new laptop that uses a full size port as opposed to the low profile. We also incorporate an Ethernet dongle that is fastened to the carrier.
All that said, the game has been fairly brutal this year. Definitely need to keep inspecting and repairing after every match to avoid failures due to damage.
Matt
Schnabel
22-03-2016, 14:16
.....
It's aggravating to have failures in KOP devices that we're required to use on our robot, but cannot repair or upgrade to something more reliable. FRC requires robust components, and expecting teams to deal with internal failures of KOP parts used in a manner consistent with their intended purpose as "part of the game" is not acceptable--quality control on FRC KOP components should be top notch, and it simply isn't.
(The "tested" sticker on the back of the PCP was a bit of an insult-to-injury chunk of irony. They said it was tested...but never said whether or not it worked.)
Out of curiosity, have you contacted CTR-E to see if they can help determine the root cause for the PDP and PCM? They are very easy to work with and very smart, I wouldn't be surprised if they end up helping find a root cause for both and then helping you get it fixed. Additionally how long did it take for you to replace what you assumed (and later verified) are bad components? I'd like to take this time to remind everyone that Spare Parts has multiple of these required components specifically for teams who are having issues. It's understood that you cannot open or modify them, so swap them out for an event and see if that helps.
Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?...
Joe, I get you are trying to stay unbiased in this, but you are also trying to point fingers at FIRST for any dropped communication issues that appear on the field. That isn't a fair assessment. There's still a level of responsibility for the teams to take care of their own hardware and ensure that their robots are capable of playing the game at hand. Teams that take the time to make their wiring clean and ensure proper connections and relief and provided to their electronics rarely have issues. For example, one of the teams who dropped comms constantly at a previous event I attended had a rats nest of wires and relied on gravity and double sided tape to hold pieces together. Every time they lost comm we were able to determine what had come loose. Sometimes comms drop and a cause cannot be determined, that's just a fact of life when you are on the field. Even on my high school team this happened, eventually we found the issue or found a work around to prevent it (looking at you, 2009 bot who wouldn't boot properly every time). My point is, there are a lot of troubleshooting steps teams can and should take prior to entering the field to play. These include tug tests, checking for loose connectors, etc. You wouldn't believe the number of teams I've told to do that very thing (even at the drivers meeting) and they don't.
I don't have any data to back me up, but here are my general impressions of the control system year over year. Note that when a robot drops I attempt to do everything in my power (and allotted time) to find the issue on the field itself. I hate phantom issues showing up in the middle of a match. It's not inspiring to the students, it's not fun, it may provide a challenge to the teams, but they shouldn't have to worry about these types of issues during a 2.5 minute span. I would say I have about a 90% success rate and determining and replicating the issue on the field.
2011 - 2013 - I remember hating the control system during these years, I just can't remember exactly why.
2014 - Horrible comm drops due to multiple issues and with no way to easily explain why. I remember issues with the radio and crio rebooting in the middle of a match being most common. This was caused primarily by brown outs to either, or loose wiring. The brown outs occured most frequently in the last 60 seconds of a match, especially when the robot started playing defense. Pushing matches stressed the control system beyond what it could handle. Additionally I remember parts of the control system having to be swapped out randomly due to shorts and failures (digital side car, radio power converter, etc.)
2015 - Hardly any comm drops compared to previous years. This was a refreshing turn after concerns of switching to a new control system. There were multiple factors that helped including the low stress on the robots, stronger connections (when wired properly), brow-out protection being built into the roborio, and the vrm keeping the radio alive during short voltage drops that were low enough to take out the rio. There were multiple instances of robots dropping below the rio's threshold, causing a reboot, but the radio stayed up and connected. This was the most solid I've ever felt our control system has been.
2016 - I feel like this year is some-what in between 2014 and 2016. Issues from wires coming loose or pulling out are back due to the intensity of the game, but we still have many of the protections in place that helped the control system in 2015 feel solid (vrm and brownout protection). The big issue I see this year is just how long it takes to get communication back with your robot. A team can die at the start of teleop, then start moving with 20 seconds left. One of the most annoying aspects of this year's control system is starting robot code after the DS has full communication back up. It can take the 50 seconds it takes the radio to boot up to nearly 70+ seconds just for the robot to be enabled again. In other words, if you drop, the chances of coming back in time are slim.
The Lucas
22-03-2016, 16:52
I initially had radio power problems during the build. I decided to strip the 24 AWG wire in the power twice as long as recommended (~3/4in), fold it in half, and point solder the end back the beginning of the strip (to make it more manageable during insertion). I know the VRM is rated for the 24 AWG as its minimum, but I feel that this loop provides a more reliable connection than just 24 AWG stranded. Just be careful not to force it down too much during install.
Also, if you mounted your radio with the barrel connector facing toward the floor, I suggest you rotate it so gravity is not working against you.
I continue to have major power problems using the old DLINKs and I am considering soldering directly to the PCB. So I think the problem is not specific to the new radio.
pfreivald
22-03-2016, 19:09
Out of curiosity, have you contacted CTR-E to see if they can help determine the root cause for the PDP and PCM? They are very easy to work with and very smart, I wouldn't be surprised if they end up helping find a root cause for both and then helping you get it fixed. Additionally how long did it take for you to replace what you assumed (and later verified) are bad components? I'd like to take this time to remind everyone that Spare Parts has multiple of these required components specifically for teams who are having issues. It's understood that you cannot open or modify them, so swap them out for an event and see if that helps.
I haven't contacted them as yet--too busy preparing for FLR this week!
It took maybe 20 minutes to replace the PCP, and ten to replace the PCM.
Our gear boxes were an accessibility issue that is, ultimately, our fault; the robot is packaged so tightly that we'd have to tear it apart to get the gearboxes off.
jojoguy10
23-03-2016, 00:58
We plug the port saver into the roboRIO, not the radio. The power wires only need to reach from near the roboRIO to the VRM. Usually the radio is mounted some distance from the rest of the control system components, and it's convenient to run a single cable from the port saver up to the radio.
The Ethernet port next to the coax power input is the only port on the radio that accepts this scheme.
The blue and brown pairs from the "plug" side of the port saver are indeed disconnected in the one we have installed on the robot, but that was mostly for ease of construction. I have used a different "passive injector" during testing and verified that having power going to those pins on the roboRIO as well does not break anything.
Hey Alan, I've been following this thread, and I wanted to make sure I understand everything correctly. Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?
Thanks,
Joe
cbale2000
23-03-2016, 02:14
I feel like we've had an absurd number of coms issues ever since we went to WiFi-based radios. I miss the old pre-2009 control system that would boot in 5 seconds and connect instantly. Basically the only coms issues I ever had with those systems was when something was physically damaged during a match. We should have stuck with something like that and just used WiFi for on-board video streaming, that way a connection failure doesn't shut down the whole machine for the entire match.
[/rant]
Alan Anderson
23-03-2016, 07:22
Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?
What you show will work, but it's backwards from how I did it. Wiring the power into the cable connected to the radio fails to realize one of the main advantages of Power Over Ethernet. If you wire it the other way around, with the power wires going into a short extension plugged into the roboRIO, you can put the radio at any distance from the power source by using a standard patch cable between it and the powered Ethernet jack.
olapmonkey
23-03-2016, 08:34
Hey Alan, I've been following this thread, and I wanted to make sure I understand everything correctly. Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?
Thanks,
Joe
This matches the PoE configuration we shop tested last night. Being paranoid now due to prior issues, we verified that we can provide power to the radio over both PoE and the standard barrel jack. We can unplug either individually and the radio won't lose power. So we now can have redundant sources of power to the radio from the VRM or pick and choose them individually. Overkill? Yes, but when your radio is intermittently rebooting during match play it's good to have a few extra tricks in the bag.
jojoguy10
23-03-2016, 09:05
What you show will work, but it's backwards from how I did it. Wiring the power into the cable connected to the radio fails to realize one of the main advantages of Power Over Ethernet. If you wire it the other way around, with the power wires going into a short extension plugged into the roboRIO, you can put the radio at any distance from the power source by using a standard patch cable between it and the powered Ethernet jack.
Oh, duh! It was late last night. Haha! Thanks Alan!
Joe Johnson
23-03-2016, 12:48
<snip>
Joe, I get you are trying to stay unbiased in this, but you are also trying to point fingers at FIRST for any dropped communication issues that appear on the field. That isn't a fair assessment. There's still a level of responsibility for the teams to take care of their own hardware and ensure that their robots are capable of playing the game at hand. Teams that take the time to make their wiring clean and ensure proper connections and relief and provided to their electronics rarely have issues. <snip>
I don't know what to say. I can't think of a single case in my engineering career where I got to say to management, "yes our customers are having a crummy experience but really, we just have dumb customers. If we had smarter customers we wouldn't have these problems."
I am trying not to flame out here but it is hard because I know how much pressure most teams are under and I sympathize with them when they have to watch their robots sit dead for so many matches. Nobody wins in this case.
Do I blame these teams? Yes. Yes, I do. But I blame FIRST too. And if my (anecdotal*) experiences at NEF-Reading and NEF-UMassD are any guide, this seems like a pretty big deal and I have not observed an appropriately big deal response from FIRST (e.g. a tech update insisting that no robot will pass inspection without X, Y, & Z features in place).
Dr. Joe J.
*which is why I was asking if there was anyone with data that could shed light on this. I am suspicious that I am just generalizing a few bad experiences to the general population.
Alan Anderson
23-03-2016, 15:35
Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?
I get the feeling that the problems are worse this year but I freely admit that I am a biased observer.
At present I have only my observations, with no solid data in hand to back up what I "feel" is the case. I will be asking around in the next few weeks to see if anyone I know has that data. But what I observe is that communication problems are very rare this year compared with the last two years.
I do not count cases where power was lost (loose connections, faulty crimps, a main breaker being accidentally hit by a stray robot part, etc.) or where an error in a team's programming caused the robot software to crash. Those can have similar symptoms to a "comm problem", but the underlying cause is unrelated to the field-to-robot communication method.
(Power issues this year are extremely prevalent, though. Teams simply aren't checking regularly to make sure their battery and breaker connections are tight, or that wires are properly inserted into the Weidmuller connectors.)
MrRoboSteve
23-03-2016, 15:40
Joe -- I helped a lot of teams with comm issues at week 1 and 2 events. In general, the robots I helped had a lack of care in wiring -- gobs of electrical tape, lack of strain relief, improperly stripped wire ends, loose connectors, etc. Electrical and programming on FRC robots the last thing on the task list for many teams, and gets shorted in the rush to get the robot into the bag.
If a team with this lack of care appears at the field at 7:50 PM on practice day, asking for their connection test, there's a decent chance you'll see those same numbers with a solid RSL in a match the next day.
In the CSA role, in the heat of the competition, it's hard to convince a team to make changes if you don't have a match failure to point to. Example: one team I worked with recently had a painfully bad radio wiring job. Yet it took me three failed matches to convince them to go to spare parts and replace the 12v pigtail.
It looks like there's been a lot of discussion on the radio, breaker, connectors, etc. being the issue. I think the likely culprit here isn't the PDP, but rather the VRM.
At the NC Wake County event, I helped out a team (5544 I believe) that had a failed VRM. The symptoms were:
Both VRM lights were solid.
The wires connecting the VRM to the PDP were solid and passed the tug test.
The VRM and PDP were properly mounted.
The Radio properly booted when the robot was idle.
The VRM was located on the opposite side of the robot to the Radio.
We discovered that whenever the team drove the robot in the pits, the radio would enter a boot loop (radio power light flickering at a high frequency, not the ~2 Hz blinking you see when programming the radio). The final point shown above is important because many teams don't realize that this is an issue.
The VRM is very noisy. Locating it anywhere near your radio will cause increased packet loss and trip time. Another team at the same event reported laggy controls and communication issues. They mounted their VRM under their radio and the logs showed a direct correlation between current consumption and packet loss. The VRM is also very low cost. The testing I described would have never been done had teams not been given a reason to suspect the VRM. I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.
I highly recommend that, once the debugging procedures outlined by many others in this thread has been performed, try swapping the VRM and see if that solves the issue. In subsequent events I attended (NC Guilford, NC Wake, NC Asheville) the rate of teams having radio issues has plummeted mostly due to awareness and meticulous checking on behalf of the robot inspectors, CSAs, FTAs, and FTAAs. While I do believe that steps could have been taken to decrease the number of radio issues seen on the field, I also believe that teams & mentors should learn from these issues and test their robots under competition conditions wherever possible.
Alan Anderson
24-03-2016, 19:11
I believe the root cause was that the team had an external circuit (LEDs) connected to the VRM for some time and the constant current draw damaged the VRM electronics.
That's a reasonable supposition. The VRM's "2 amp" output is actually likely to be damaged by anything higher than a continuous 1.5 amp draw. One of the Beta Test teams did mortally wound a VRM during testing by using it as labeled rather than as documented.
well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr
Would anyone be interested in doing a DFMEA style deep dive on this?
One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.
I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.
Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.
As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?
Would anyone be interested in doing a DFMEA style deep dive on this?
One might be that I have never used a torque wrench to tighten the terminals on the breaker. The spec sheet I have says up to 50 in*lbs. I am thinking that setting a wrench to 40 in*lbs and offering to snug down all terminals might help lot.
I am not sure what percentage of issues were main breakers, but I know a decent percentage. During inspection, we are asked to do a tug test, and at least a couple robots I have verified via tug test this year were loose later during the event.
Loose batteries terminals would be another item frequently causing voltage drop outs. We check the battery installed, and look at the others for insulation, but we have typically not tug testing every battery terminal.
As there are several other possible areas. Maybe it would be good to do the DFMEA and list of tips on how to stop the failure modes. Thoughts?
DFMEA????
DFMEA????
Design Failure Mode and Effect Analysis (http://www.isixsigma.com/dictionary/dfmea/). It's a Six-Sigma process that has you go through the system listing all the things that could go wrong, how bad it would be if they did, and how hard it is to detect them. After you do that, you take the ones that score the worst and develop plans to remove or mitigate the failure, then rinse and repeat.
AllenGregoryIV
27-03-2016, 17:58
Weidmuller connector length
Strip the wires that are inserted into the various Weidmuller connectors to the required length. Often teams make these too short and the result is the wire is not properly retained.
What is the required length?
The PDP (http://content.vexrobotics.com/vexpro/pdf/217-4244-PDP-Users-Guide-20150305.pdf), PCM (http://content.vexrobotics.com/vexpro/pdf/217-4243-PCM-Users-Guide-20150219.pdf), and VRM (http://content.vexrobotics.com/vexpro/pdf/217-4245-VRM-Users-Guide-20160117.pdf) documents list 3/8". The FIRST video on using Weidmuller connectors says 5/8" (https://www.youtube.com/watch?v=kCcDw3lDYis)
I believe the spec is actually 8mm (http://catalog.weidmueller.com/catalog/Start.do?localeId=en_DE&ObjectID=1825640000) but I just want to make sure since the official FIRST video says something different.
Radio PoE
Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set (https://www.adafruit.com/products/435) didn't work in my testing.
well, we found the problem mostly
one of the CSA's at Milwaukee(Pete Thomas) a 706 past mentor handed us a new 5.5x2.1 barrel jack for the radio power and that fixed the issue until the last match of seeding matches. Had NO issues from the time we put in that jack to that last seeding match. Then it started happening again and we lost comms 2 times in the elimination matches. an explanation for that has not been found.
Our VRM is under(on the opposite side of a piece of aluminum sheet) and about a foot to 18 inches away from the radio. Cannot tell you how frustrating this season has been. Absolutely no comm issues on our practice bot for the 3 weeks we were able to use that then comp bot lets us down. Wired by the same meticulous student using all new parts. FYI the barrel jacks that we initially used were(on the comp bot the one in the KOP) and on the practice bot the one from AM with the screw terminals. Both caused radio reboot issues. we also swapped out the comp bot VRM with the well tested practice bot VRM.
Suppose we will just have to go with POE as we did for the last half of the Milwaukee regional. When we had the comm issues in the elimination matches we had dual power to the radio. The new barrel jack and POE.
Still had comm losses. grrrrrr
Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?
Alan Anderson
28-03-2016, 00:09
Also has anyone found a COTS solution to doing PoE for the radio? The adafruit set (https://www.adafruit.com/products/435) didn't work in my testing.
How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.
AllenGregoryIV
28-03-2016, 02:27
How were you trying to use it? From what I can tell, each of the two parts in that pair is wired properly to work with the 18-24v POE port on the radio, as long as you get the polarity correct on the power wires. The only difference is what type of barrel connector it has, and you'd be cutting that off anyway in order to connect to the VRM.
Wow that would probably be my problem I just did a quick test using the barrel we already had connected, didn't realize the power and ground were reversed. I'll cut it tomorrow and make sure it works.
Can you post a picture of your robot? I'm thinking it may have something to do with radio placement. Another theory is the radio could be damaged. Have you tried swapping it out for a spare?
The radio is far from the VRM and as far away from any motors as we can get them. The closest one is a drive CIM and that is something like 12 inches away.
After we swapped out the barrel connector we had no issues the rest of Thursday and Friday all the way to the last seeding match on Saturday.
then 3 times in like 8 matches.
Had all the orange and yellow hats looking. Nobody could find an issue.
One thought I had was this. We are using these wide polyurethane beach tires.
We notices when we were on the practice field we sold get a strong static electricity shock when we touched to bot after driving a while and we could also hear a discharge when going over the defenses. Never seen this before with a robot.
We always lost comms not after a violent move but rather when beginning to cross a defense.
Now I am wondering if we have two problems. Radio power, fixed by POE and the new barrel connector and a static discharge causing another failure mode?
here is a pic
http://www.chiefdelphi.com/media/photos/42916
you can see the radio on the outside and away from most every thing electrical
http://www.chiefdelphi.com/media/photos/42916
http://www.chiefdelphi.com/media/img/0af/0af22aeb790e4746833d60010cf6c289_l.jpg
hope this works
RufflesRidge
28-03-2016, 10:15
The FIRST video on using Weidmuller connectors says 5/8" (https://www.youtube.com/watch?v=kCcDw3lDYis)
The page that video is embedded on (http://wpilib.screenstepslive.com/s/4485/m/13503/l/144971-wiring-the-2016-frc-control-system) contains this note:
The correct strip length is ~5/16" (8mm), not the 5/8" mentioned in the video.
From the VRM users guide
Strip wire back ~0.375” (3/8”)
evanperryg
28-03-2016, 12:59
we had several folks look at the fuses and they were deemed OK but I will pull them and insert new ones Thursday.
Thanks
Make sure you smash them in there. Excessive force is necessary for them. Also, make sure the power connections to and from the VRM are done properly. When they're secured the right way, you should be able to pull on them with all of your weight without them moving.
Ok, not on topic, but I have to ask what kind of tires are those?
And on topic, we too lost comms on our first match this past weekend. We were told that we should not have tinned the wiring going into the VRM....
Al Skierkiewicz
28-03-2016, 14:00
Bruce,
I know that this doesn't help you at this point but there doesn't seem to be any strain relief on the power plug for the radio. The barrel connector depends on the "spring force" of the outside contact to hold everything in place. I highly recommend using an adhesive backed wire tie point on the top of the radio. Loop the power wire up to the tie point and then secure with a a small wire tie. This small step keeps the connector from bouncing around and prevents pullout. Tape over the connector to the radio body also works. The antennas for this radio are on the sides of the radio, inside the vents. It doesn't help to have the stainless pneumatic tanks on that side of the radio but shouldn't be a big problem.
There is also the possibility that severe vibration has caused the power plug to pull out of the circuit board. This is easily repaired if the circuit board has not been damaged. The connector is a part of the circuit board that a good hit on the connector could cause a crack in the board. This has the same effect as broken wires. It may work most of the time, but vibration will make it lose the power.
Tinning wires is not recommended for Weidmuller terminals. They depend on the deformation of the strands to give maximum surface area for current to flow. With tinned wire, only the extreme outside surface of the wire touches the contact producing a connection equivalent to a #36 or smaller wire.
Team34Guy
28-03-2016, 14:11
We experienced the same thing at both the Arkansas and Rocket City Regionals. After speaking to other teams at Rocket City, the consensus was the power connection at the radio. It isn't as reliable as the D-Link. I found ours had backed out a little over an 1/8" and most likely lost connection briefly when we struck the wall in autonomous. Bomb Squad mentioned that they had experienced some issues prior to competition and had hot glued their connector, they didn't have any further issues with that connection.
It's extremely frustrating to say the least....Especially when Bob starts fussing! (cause usually he's right)
AllenGregoryIV
28-03-2016, 14:18
One of the other things that hasn't been metioned is the use of ferrules with the weildmuller connectors. We used them on nearly all of our competition robot connections and have had zero issues.
Here is the white paper from Weildmuller (http://www.e-sonic.com/productfiles/mf-WE1/Weidmuller_Ferrules_White_Paper.pdf) recommending their use with stranded wires.
We import ours from China using Aliexpress but you can also get them from US electronics vendors, ebay, or Amazon.
Make sure you are using the right crimp tool and gauge ferrules. Also get 8mm long ones.
AllenGregoryIV
28-03-2016, 14:37
This thread inspired me to write up this document. Anyone have any more suggestions to include.
https://docs.google.com/document/d/13xhikkb91_c2cIioHCmKU5IB0-lK4k20SGtXrCy1gzk/edit#
Ok, not on topic, but I have to ask what kind of tires are those?
And on topic, we too lost comms on our first match this past weekend. We were told that we should not have tinned the wiring going into the VRM....
they are 9.4 inch beach tires. 3 psi and they litterally float over the defenses
Bruce,
I know that this doesn't help you at this point but there doesn't seem to be any strain relief on the power plug for the radio. The barrel connector depends on the "spring force" of the outside contact to hold everything in place. I highly recommend using an adhesive backed wire tie point on the top of the radio. Loop the power wire up to the tie point and then secure with a a small wire tie. This small step keeps the connector from bouncing around and prevents pullout. Tape over the connector to the radio body also works. The antennas for this radio are on the sides of the radio, inside the vents. It doesn't help to have the stainless pneumatic tanks on that side of the radio but shouldn't be a big problem.
There is also the possibility that severe vibration has caused the power plug to pull out of the circuit board. This is easily repaired if the circuit board has not been damaged. The connector is a part of the circuit board that a good hit on the connector could cause a crack in the board. This has the same effect as broken wires. It may work most of the time, but vibration will make it lose the power.
Tinning wires is not recommended for Weidmuller terminals. They depend on the deformation of the strands to give maximum surface area for current to flow. With tinned wire, only the extreme outside surface of the wire touches the contact producing a connection equivalent to a #36 or smaller wire.
Al that was an old pic just to show the location of the radio. At the time of our latest comm losses during elims we had a new tighter barrel jack taped in place AND we were running POE. so we had two power sources.
this bot had been gone over by nearly every orange and yellow hat at both Cleveland and Milwaukee. After replacing the barrel jack and installing POE in Milwaukee we had no issues until the last match of quals then it started all over again. Had Pete Thomas doing everything he could think of during elims but he found nothing.
is it just the 706 curse? 6 time finalist in the last 4 years.
Thanks
one other thing
we NEVER lost comms during extreme contact with another robot or flying over defenses. It always happened when just starting to cross the defense and even happened when crossing the low bar once.
I am beginning to think a static discharge may be part of the problem as we noticed a static discharge when crossing defenses on the practice field.
AllenGregoryIV
28-03-2016, 17:38
From looking at logs and reported observations from the FTA's they see the radio rebooting. Immediately after they see the radio return they see the RIO so the belief is that the RADIO is the only think rebooting as the RIO takes longer than the radio and would not show up immediately after the radio if it was also booting.
Do you have the driver station logs from those matches that you can post?
There is also a system log file you can pull up using the RoboRIO webpage as well. We had a problem at Bayou were we were seeing the RoboRIO just crash for no reason (not our code but the whole RoboRIO, as in it stopped even showing up as a USB network adapter to computers once that crash happened, we never found the problem and just swapped RoboRIOs)
I will see if I can have our programmers post those from finals matches.
Monochron
31-03-2016, 12:19
One of the other things that hasn't been metioned is the use of ferrules with the weildmuller connectors. We used them on nearly all of our competition robot connections and have had zero issues.
...
Make sure you are using the right crimp tool and gauge ferrules. Also get 8mm long ones.
We used ferrules on almost all of our weidmuller connections and found that the crimps eventually work their way out after heavy competition. We are using the correct tool (ratcheting style) and gauge for our wires but getting consistent crimps never seemed possible. We are considering moving to mostly solidcore wire to avoid troubles in the future.
If you have gotten consistent crimps that hold up to this year's game, do you think that our crimping tool might be bad? I'm not sure where the failure is, but right now we aren't able to rely on ferrules.
one other thing
we NEVER lost comms during extreme contact with another robot or flying over defenses. It always happened when just starting to cross the defense and even happened when crossing the low bar once.
I am beginning to think a static discharge may be part of the problem as we noticed a static discharge when crossing defenses on the practice field.
I believe you are on to something. Static problems can be so elusive because changes in humidity and surface contact materials are so difficult to track.I can relate one situation where static buildup definitely was the culprit and why it may relate to your problems.
We have used dozens of UsDigital MA3 absolute encoders for steering feedback over years so we know their characteristics pretty well. We have also used aluminum AM Performance wheels that contact the carpet even with tread attached. One off season we decided to replace the performance wheels with AM Plaction wheels for a demonstration robot. Within a couple of hours of run time, we had two encoder failures. The outputs would start to jitter and eventually get too bad to use. After realizing the wheels were Polycarbonate and probably isolating the chassis to allow static buildup, I attached a chain to drag the ground and never had another problem.
Many people are running pneumatic wheels this year without problems but most tires have additives that are conductive. Looking at the beach tires, they don't appear to have additives that other tires have. Is it possible that they are insulating enough to allow a charge to build on your robot? We have also had more static problems when running belts. They tend to create static like a Van De Graaff generator.
I'm not sure if a chain is the best idea this year with the defenses but something to reduce the potential between the carpet and the frame might avoid the big discharge when you touch the metal defense. If you can find a static gun some of this can be more scientifically proven.
For those that might say that the control system is immune to static, after working with plastic extrusion for 30+ years I have seen enough electronic equipment affected and destroyed from static buildup to believe that nothing is immune. Some designs are more tolerant but at some point a large enough discharge will cause damage. Also, many devices will continue to operate after being zapped but are degraded and prone to erratic performance and failure until replaced.
The ESD theory is interesting. Our driver asked how he could keep from getting shocked when he was shutting off the robot at the Kansas City Regional.
protoserge
31-03-2016, 13:21
The ESD theory is interesting. Our driver asked how he could keep from getting shocked when he was shutting off the robot at the Kansas City Regional.
Were they shocked previously?
I am interested in this ESD theory. It sounds very plausible.
AllenGregoryIV
31-03-2016, 14:13
We used ferrules on almost all of our weidmuller connections and found that the crimps eventually work their way out after heavy competition. We are using the correct tool (ratcheting style) and gauge for our wires but getting consistent crimps never seemed possible. We are considering moving to mostly solidcore wire to avoid troubles in the future.
If you have gotten consistent crimps that hold up to this year's game, do you think that our crimping tool might be bad? I'm not sure where the failure is, but right now we aren't able to rely on ferrules.
We are currently using 20 AWG ferrules for everything (we haven't purchased a full set yet). We had to adjust the tension on our crimper to get a tight hold and we crimp the insulation in addition to the terminal. We tug test every wire and didn't have any failures at Bayou. We are using generic cheap crimps and a crimper from china.
Were they shocked previously?
I am interested in this ESD theory. It sounds very plausible.
I told him to have the sophomore turn off the robot that way he wouldn't get shocked. He did not say anything about getting shocked at the Colorado Regional. This may have been due to being on the floor with insulation over the ice arena.
We had one reboot during the Colorado Regional. It was in in our last qualification match. The problem was due to a loose battery connection.
I was the one of the CSAs in Colorado. Most of the issues with radios and roboRIOs were due to loose connections or components.
One robot had too much strain relief on the Ethernet cable and when the robot frame twisted, it pulled the RJ-45 all the way to one side and disconnected.
One robot had a battery cable move and unplug the Ethernet cable on the radio when going over a defense
Monochron
31-03-2016, 14:40
We are currently using 20 AWG ferrules for everything (we haven't purchased a full set yet). We had to adjust the tension on our crimper to get a tight hold and we crimp the insulation in addition to the terminal. We tug test every wire and didn't have any failures at Bayou. We are using generic cheap crimps and a crimper from china.
Hhm, adjusting the tension and crimping the insulation as well could make a difference for us. Thanks for the tips, I'll look into it.
I guess I need to complete my theory on how static might create radio problems. The static voltage buildup doesn't cause the problem, it is the sudden discharge of that voltage (ESD) as Greg stated that can create problems.
Years before RFID was mainstream technology I bought and installed a series of antennas to read tags tied to material conveying hoses. They were used to identify where the hoses were plugged. Almost immediately I had lockups and failures. I finally determined that static discharges were being received by the antennas and were actually damaging the receivers. They wouldn't fail immediately but damage the inputs progressively. I was able to tell the damage by measuring input resistance and predict the failures. The problem was totally eliminated by creating paths for the charges to dissipate farther away from the antennas.
We are dealing with a router with antennas here also. The higher the gain the more possibility that the input can be overdriven by large magnetic pulses. The technology has advanced drastically over recent years but theoretically the radio would be the most vulnerable device unless the discharge were to the actual control system. Since it is supposed to be isolated from the frame that is less likely.
This is all just speculation and not based on anything tested with this equipment. Consumer electronic devices are tested for these kinds of events but at some reasonable level. I have seen many fail with the levels of discharge easily produced in plastics handling. The trick is to reduce the potential to an acceptable level.
We need to watch for signs of affects caused by discharges.
Ah, yes.... like the old lightning straps we used to have on our cars.... We need to find those old rubber straps. lol
What's next? Rockers? :yikes:
wish I had another regional to go to to test this theory.
I can see where these beach tires may have uses in other games and I sure dont want these issues again.
We are running a belt drive on AM plastic belt sprockets. Is that what is creating the charge? And the polyurethane wheels are insulating it so it can build up?
What would the best way to prevent the discharge be?
olapmonkey
31-03-2016, 16:27
Maybe one of these...
Gates 90330 Static Strap https://www.amazon.com/dp/B000CRBR8U/ref=cm_sw_r_other_awd_iJy.wb6T3YNQP
And maybe have drive team next year wear tin foil hats?
billbo911
31-03-2016, 16:43
Static was not an issue for us this year. Both events we attended had relatively high (for our area) humidity.
In the past we would attach a short piece of chain to the frame and let it drag on the carpet. Only a couple links on the floor were enough.
You should see the looks we would get from inspectors and judges. But, once we explained the logic, they never questioned our approach.
This year we felt that dragging chain was a much higher risk than static build up might cause.
After being FTAA at two separate events, one thing I can say with 100% certainty. The field this year will find and expose any fault your robot has.
Road Rash
31-03-2016, 21:42
Years before RFID was mainstream technology I bought and installed a series of antennas to read tags tied to material conveying hoses. They were used to identify where the hoses were plugged. Almost immediately I had lockups and failures. I finally determined that static discharges were being received by the antennas and were actually damaging the receivers.
I'm not an electrical engineer but I have hands on experience with switching material lines in the dry air of a Wisconsin winter at the sheet extrusion plant I'm employed at. I have felt those static discharges and I can see where it could affect sensitive electronic equipment.
Our forklifts all have short chains making a connection to the ground. Some of our product is corona treated (https://en.wikipedia.org/wiki/Corona_treatment) to improve ink adhesion for our customers in the graphics arts industry. Naturally, this makes our finished product contain a charge in spite of our efforts to reduce this with static bar systems further downstream. The chains are there so our forklift operators aren't the ones completing the circuit when exiting the vehicle after lifting the finished product off the end of the sheet stacker.
This is a great thread with a lot of good information to absorb. Our team is weak on electrical and I'm just trying to help them out any way I can.
EDIT: Those of you looking to reduce ESD using a small chain dragging on the ground may want to look into using Static String (http://stopstatic.com/) instead. We also utilize this string to reduce static buildup on our extrusion lines. Copper tinsel (http://www.amazon.com/Amstat-Copper_Tinsel_36foot_single_roll-Anti-Static-Copper-Tinsel/dp/B00C0VN9GY) works as well and both items should have fewer issues with snagging on obstacles than a chain would.
Al Skierkiewicz
04-04-2016, 07:20
We are considering moving to mostly solidcore wire to avoid troubles in the future.
Please don't do this. Solid wire is not for use on moving objects. It has a real tendency to fatigue and break just when you need it the most. (Al's Murphy's law corollary...If it can go wrong, it will on Einstein.) Most of the contacts on the components are designed for stranded wire so that the wire can conform to the internal shape of the contact. The ferrules are specifically designed to be used with a certain product so be sure to buy the type for your application. The ferrules for use with the PDP are not round.
Be careful what you use for static discharge, any damage to the field (carpet or other) will irk the field supervisor who will then tell the Head Ref. More importantly for those who suspect static issues is to look where your wiring is running through the robot. If it is near the wheels, or other moving parts, what static might build up will rapidly jump to that wiring. While most of the components are designed to survive that discharge, that doesn't mean they will continue to function during the match. It is entirely possible that a power reboot is the only way to correct the condition. Power wiring that receives the static discharge will pass that along to other wiring that might run in parallel. Most wiring teams use only has a 300 volt rating for the insulation.
Road Rash
04-04-2016, 10:12
Be careful what you use for static discharge, any damage to the field (carpet or other) will irk the field supervisor who will then tell the Head Ref.
Good advice. The tinsel breaks easily leaving pieces of copper behind and is the reason we don't use it anymore where I work. I envision snagging issues with chains. Static string is the best bet. It's expensive, but it's lightweight, very durable and works as intended in the environment we use it in. However, even with the cost, a 72' roll of it will last for years considering you only need to use a few inches at a time. It's basically some stranded copper wire with a durable conductive sheathing.
I also agree, it's best to engineer the static issues out of the build altogether. If you cannot do this, then use the string.
One of the other things that hasn't been metioned is the use of ferrules with the weildmuller connectors. We used them on nearly all of our competition robot connections and have had zero issues.
Here is the white paper from Weildmuller (http://www.e-sonic.com/productfiles/mf-WE1/Weidmuller_Ferrules_White_Paper.pdf) recommending their use with stranded wires.
We import ours from China using Aliexpress but you can also get them from US electronics vendors, ebay, or Amazon.
Make sure you are using the right crimp tool and gauge ferrules. Also get 8mm long ones.
We got our ferrules from www.ferrulesdirect.com (http://www.ferrulesdirect.com/) and our crimper from amazon (http://www.amazon.com/Signstek-Self-adjustable-Crimping-0-25-6-0mm2-End-sleeves/dp/B018X8S7UA). We found that installing ferrules on our 16AWG wires (i.e. from the compressor) made them too large to work with the Weidmuller connectors.
The cable that came in our KOP had a right-angle plug on it. Once it was plugged in, wrapping a long cable-tie around the radio and the plug worked well to keep it plugged in very securely. We were able to pass the cable-tie under the radio, between it's "feet", so it could not slip off.
waialua359
04-04-2016, 13:52
I believe you are on to something. Static problems can be so elusive because changes in humidity and surface contact materials are so difficult to track.I can relate one situation where static buildup definitely was the culprit and why it may relate to your problems.
We have used dozens of UsDigital MA3 absolute encoders for steering feedback over years so we know their characteristics pretty well. We have also used aluminum AM Performance wheels that contact the carpet even with tread attached. One off season we decided to replace the performance wheels with AM Plaction wheels for a demonstration robot. Within a couple of hours of run time, we had two encoder failures. The outputs would start to jitter and eventually get too bad to use. After realizing the wheels were Polycarbonate and probably isolating the chassis to allow static buildup, I attached a chain to drag the ground and never had another problem.
Many people are running pneumatic wheels this year without problems but most tires have additives that are conductive. Looking at the beach tires, they don't appear to have additives that other tires have. Is it possible that they are insulating enough to allow a charge to build on your robot? We have also had more static problems when running belts. They tend to create static like a Van De Graaff generator.
I'm not sure if a chain is the best idea this year with the defenses but something to reduce the potential between the carpet and the frame might avoid the big discharge when you touch the metal defense. If you can find a static gun some of this can be more scientifically proven.
For those that might say that the control system is immune to static, after working with plastic extrusion for 30+ years I have seen enough electronic equipment affected and destroyed from static buildup to believe that nothing is immune. Some designs are more tolerant but at some point a large enough discharge will cause damage. Also, many devices will continue to operate after being zapped but are degraded and prone to erratic performance and failure until replaced.
Mr. Novak,
thank you for your expert opinion.
I think we definitely need to look into this.
We lost COM way too many times at only our Hawaii regional.
It never happened after a hard bump or hard defense where we would suspect cables being loose. We are quite certain it is not our connections.
Hawaii has VERY high humidity.
The thing that caught my attention is getting shocked by static electricity when touching the robot this past weekend.
Hawaii has VERY high humidity.
The thing that caught my attention is getting shocked by static electricity when touching the robot this past weekend.
High humidity and static electricity are usually mutually exclusive. One method used in industry to control static electricity is to humidify the air. I have not had static problems at work since moving from Eastern Canada to Houston but then our equipment does not have so many rotating parts. It is possible that the pneumatic tires used by many teams this year may generate much more static charge than other types of wheels used more commonly in previous years.
Try wiping some fabric softener sheets all over the tires. They leave a mildly conductive film that will help dissipate the static charge that accumulates on the tires. You would want to wipe all the way to the shafts or metal hubs that are in contact with the shafts. Hopefully, this will dissipate and equalize the charge over the whole robot so there cannot be an electro-static discharge from one part of the robot to another. It may be necessary to re-apply the fabric softener periodically since it may rub off. Alternatively, there are industrial anti-static coatings and sprays such as Staticide (http://www.aclstaticide.com/antistatic_coatings.html), but they are probably harder to get than the fabric softener. I also just found these instructions on how to make your own anti-static spray (http://www.instructables.com/id/Creating-Anti-Static-Spray/).
Perhaps CTRE can add some clamping devices such as TVS' across the inputs and outputs of the PDP and VRM. They would add only a dollar, or less, to the part costs though they would need some board space. The wiring in our robots is short enough that a TVS in the PDP would probably limit the voltages in the whole circuit running to the radio quite effectively.
waialua359
04-04-2016, 14:26
High humidity and static electricity are usually mutually exclusive. One method used in industry to control static electricity is to humidify the air. I have not had static problems at work since moving from Eastern Canada to Houston but then our equipment does not have so many rotating parts. It is possible that the pneumatic tires used by many teams this year may generate much more static charge than other types of wheels used more commonly in previous years.
Try wiping some fabric softener sheets all over the tires. They leave a mildly conductive film that will help dissipate the static charge that accumulates on the tires. You would want to wipe all the way to the shafts or metal hubs that are in contact with the shafts. Hopefully, this will dissipate and equalize the charge over the whole robot so there cannot be an electro-static discharge from one part of the robot to another. It may be necessary to re-apply the fabric softener periodically since it may rub off. Alternatively, there are industrial anti-static coatings and sprays such as Staticide (http://www.aclstaticide.com/antistatic_coatings.html), but they are probably harder to get than the fabric softener. I also just found these instructions on how to make your own anti-static spray (http://www.instructables.com/id/Creating-Anti-Static-Spray/).
Perhaps CTRE can add some clamping devices such as TVS' across the inputs and outputs of the PDP and VRM. They would add only a dollar, or less, to the part costs though they would need some board space. The wiring in our robots is short enough that a TVS in the PDP would probably limit the voltages in the whole circuit running to the radio quite effectively.
Thanks for the insight and links. We will definitely look into this for Championships.
spacepenguine
05-04-2016, 14:40
This is a great discussion on static electricity. Supposedly the robot frame is isolated from the electronics, but that doesn't mean it can't cause problems.
I only saw a small mention of VRM wiring, but suspect some teams may be hurting themselves by pairing up the radio and LEDs/lights on the 12v/2a bus of the VRM. There is a rule and checkbox on the inspection form to prevent this, but I'm not sure it was consistently applied across districts and regionals. We're all volunteers, after all.
From non-FIRST experience I would hot glue or permanently fix in anything with a barrel connector. They like to work their way out with vibration. We've seen this a lot with CMU's RoboBuggy.
KrazyCarl92
05-04-2016, 18:58
Team 5811 experienced issues with both radio and RoboRIO reboots at the Greater Pittsburgh Regional in Week 2. In about a third to half of our matches, we would be dead on the field for some period of time. We made an action plan for our Week 5 event at QCR to mitigate these issues, assuming that power connections to the concerned components was the root cause.
1. We had been using ferrule crimps on our RoboRIO power connection. After being eliminated in Pittsburgh, I had the opportunity to thoroughly check Team 3138's beautiful wiring where I noticed that they were using ferrules on every single connection EXCEPT the RoboRIO power connection. They stated that the clasp that screws in from the side only gets a single point of contact with a ferrule connector (due to the way the ferrule connector does not seat nicely within the RoboRIO), but that bare stranded wire actually provides a superior connection in this instance. Easy enough: we would trim our ferrule crimp off those wires, trim, then strip to establish a better RoboRIO power connection.
2. We noticed at Pittsburgh that the Radio barrel connector for power was at best a "janky" connection, causing us to lose radio power every so often during normal play. To mitigate this failure, we decided to try Power Over Ethernet (PoE) as supported by the Radio. We prepared a wire with the appropriate pinouts labeled prior to the event. Unfortunately, the wiring diagram from Team 3015 that we were using was vague, and used both solid and dashed lines to indicate which pins should be powered on the Ethernet cable. We decided to try the solid lines only first for fear of blowing out the radio (12V power to pin 4, Ground to pin 7 from the VRM 2A circuit). Voila! it worked in the pits and the radio was turning on! Unfortunately, we went to the field for the very first practice match and the robot would not connect. We then decided to use the dashed lines in the wiring diagram as well to test out how that would work (12V power to pins 4 & 5, Ground to pins 7 & 8 from the VRM 2A circuit). This did not lead to better results, and upon testing it turned out that the pins on the Ethernet connector and the wire itself for pin 5 did not have continuity (not sure why, faulty cable I suppose). At this point, we didn't have extra Ethernet cables to spare and didn't want to waste other teams' cables testing, so we decided to take another approach...
3. At Greater Pittsburgh we had attempted hot gluing the barrel connector to the radio on Saturday. This seemed to work! We had no further failures of radio connections on the field (all other failures were RIO failures, per the driver station logs). However, when we unbagged the robot at QCR, the hot glue had worn out and the barrel connector was loose again. After we did not achieve success with the PoE solution (lack of resources, not to say the solution itself doesn't work), we decided to go back to the barrel connector for radio power and attempt using a rubber band instead of hot glue to hold in the barrel connector securely. This rubber band wrapped around the back of the barrel connector and the opposite side of the radio to hold the barrel connector in place. I was at first nervous about the extra forces and potential bending loads on the connector, but this seemed to do the trick.
We are happy to report that after successful implementation of solutions 1 & 3, our robot had zero connection failures at QCR! This allowed us to successfully breach the outerworks in every single match, and increase our boulder scoring output.
We've learned a lot through this experience that we will hopefully be able to put to good use to help other teams keep their robots alive during their matches. I would almost even suggest implementing these as inspection standards so teams can only pass inspection with these solutions implemented.
TL;DR: We love ferrule connectors, and they belong on just about every other possible connection except the power on the RoboRIO; bare wire is best there. The barrel connectors that come in the KOP for radio power are ill-suited to the rigors of FRC (especially this year's game); use PoE, hot glue, or a rubber band to mitigate the risk of your radio losing power. It is important that teams be aware of solutions that may help their robots stay alive.
phargo#1018
05-04-2016, 21:29
For what it's worth, I did some checking.
The VRM is specified to operate down to 5 Volts. It is also specified to provide between 11.93 and 12.49 Volts, reference CTR's web site.
I thought it might be wise to contact the radio supplier, Open-Mesh. First off, a giant thank-you to Open-Mesh for talking. OM's standard Voltage range is 12 to 24 Volts at 1 Amp. The contact I had at OM indicated that he believes that the radio will operate properly down to 11.8 Volts. He also indicated that the radio should operate properly down to 11 or 11.5 Volts if the output power were reduced.
I will confess, I do not know what setting FRC is using for these devices; I am assuming that they are using the default power. Does anyone know?
I agree with the posts about the barrel connectors, ferrules too!!
If you find your radio is re-setting, you might check the output Voltage on your VRM. Is it less than 12 Volts? If so your device meets the specification requirements, but you might want to use some caution. You should be okay down to 11.8 Volts.
The 11.8 Volts applies at the receiving end after all IR drops are considered.
What wire are you using between the VRM and the radio, 20 AWG, 22 AWG? How long are your wires?
Suppose that you are producing 11.93 Volts at the VRM and you have 3 feet of 22 AWG wire (18 inches out and 18 inches back) and the radio is drawing 1 Amp. 22 AWG wire will account for an 0.050 Volt drop (0.030 @ 20 AWG, 0.077 @ 24 AWG). Any additional resistance due to wire termination at either end will increase this drop.
Bottom line, two suggestions:
1) Know what Voltage your VRM is providing, and
2) Keep the VRM and the radio close to each other!
Hope this helps, I am interested in follow up information!
Thanks for reading!
TL;DR: We love ferrule connectors, and they belong on just about every other possible connection except the power on the RoboRIO; bare wire is best there. The barrel connectors that come in the KOP for radio power are ill-suited to the rigors of FRC (especially this year's game); use PoE, hot glue, or a rubber band to mitigate the risk of your radio losing power. It is important that teams be aware of solutions that may help their robots stay alive.
I would strongly suggest that you leave the wires a bit long if you are not using ferrules for connections where you may want to re-do the connection at some point such as the RoboRio power. The strands of the wires tend to fray and you risk shorting to adjacent wires at the Wiedmuller connectors. The wires are only 3mm apart, center-to-center. The best way to fix wire strands that have frayed is to cut the exposed conductor off and re-strip to the appropriate length. To be able to do this, you will need some extra wire length.
I think it would be easier for people to understand how you use a rubber band to hold the radio power plug on the radio if you post a picture. It is a great idea if am reading you right but it is tricky to envision how the rubber band holds the plug. I am assuming that you have a straight plug.
Thanks for the insight and links. We will definitely look into this for Championships.
I hope it helps. I will be cheering for you and your team at Championship. It would be great to see you all back in Texas again.
Supposedly the robot frame is isolated from the electronics, but that doesn't mean it can't cause problems.
You are correct. The isolation of the electronics from the frame is checked with a hand-held meter set to measure Ohms. The highest voltage that such a test would apply between the electronics and the chassis would be 9V or less since these meters are typically powered with 9V batteries or some number of AA's (I have never seen more than 6). The insulation or air gap needed to prevent 9V from arcing across is pretty small (a fraction of a mm). Electronic components rated to resist static-discharges typically have ratings of at least 1500V, often 2500V. Any half decent static charge would have no trouble arcing over the small air gap (or punching through the thin insulation) needed to satisfy the robot isolation test.
Al Skierkiewicz
06-04-2016, 07:34
1. We had been using ferrule crimps on our RoboRIO power connection. After being eliminated in Pittsburgh, I had the opportunity to thoroughly check Team 3138's beautiful wiring where I noticed that they were using ferrules on every single connection EXCEPT the RoboRIO power connection. They stated that the clasp that screws in from the side only gets a single point of contact with a ferrule connector (due to the way the ferrule connector does not seat nicely within the RoboRIO), but that bare stranded wire actually provides a superior connection in this instance. Easy enough: we would trim our ferrule crimp off those wires, trim, then strip to establish a better RoboRIO power connection.
Carl,
Can you explain this a little better? What clasp/screws are you referring to? It sounds like the brown connector on the RoboRio. If that is the case, I believe that connector does not have the screw contacting the wire directly. Instead it compresses an open contact inside the connector.
As I stated above, please use ferrules that are designed for the termination you are using. The PDP ferrules should be square in cross section to maximize contact and minimize resistance. If you use round (or solid wire) then WAGO states you must derate the contact by at least two wire sizes. A #12 will actually perform as a #16 due to the lack of contact area.
31Josh61
06-04-2016, 07:44
Hi,
Team 3161 had the same problem at the waterloo regional. we thought it was about shock and moving the router worked but it think this is becoming a problem. In the finals of waterloo. 1114 lost there comms 2 times.
Why is everyone missing the obvious solution? Use passive Power Over Ethernet (POE) on the port that supports it. The Ethernet connector snaps firmly into place. I've never seen one come loose due to bounces or vibration this year.
...snip....
I have now seen a handful of Ethernet cable failures this season. Several at the laptop (maybe 8 failures of 5 events x 40 robots). About 50% of those were classmates. I have also observed a few "bad" Ethernet cables between the robot and rio (approximately 4-6).
One item that tends to mask the poor connection issue is using a "printer cable" to directly connect into the rio during inspection. While handy, I will be even stronger in my wording with teams that they likely have a cable/connector issue they should look into in those cases.
The majority of failures have been the other "power" related failures noted by many others.
Alan Anderson
06-04-2016, 11:41
I have now seen a handful of Ethernet cable failures this season.
I saw a few bad Ethernet cables myself. But in no case did the connector ever come out on its own. Unlike the ill-fitting Jaguar CAN connectors, the OM5P-AN's RJ-45 receptacle seems quite reliable. If the cable is good, there shouldn't be a problem.
I saw a few bad Ethernet cables myself. But in no case did the connector ever come out on its own. Unlike the ill-fitting Jaguar CAN connectors, the OM5P-AN's RJ-45 receptacle seems quite reliable. If the cable is good, there shouldn't be a problem.
What was scarier for me was cable staying connected in many cases, but causing intermittent connectivity. Intermittent are often the hardest to diagnose/detect and provide correction for.
sgeckler
06-04-2016, 12:00
The ferrules for use with the PDP are not round.
Al, I don't see any mention of ferrules in the 2016 control system wiring documentation I can find. Not the basic and not in the PDP user guide from CTR. Is there something more specific for the PDP that describes the "proper" shape of ferrules you could use? I assume all Wiedmuller would take round? - Sam
Ferrules not mentioned here:
http://www.ctr-electronics.com/PDP%20User%27s%20Guide.pdf
Or here:
https://wpilib.screenstepslive.com/s/4485/m/24166/l/144971-wiring-the-2016-frc-control-system
Alan Anderson
06-04-2016, 16:36
Al, I don't see any mention of ferrules in the 2016 control system wiring documentation I can find. Not the basic and not in the PDP user guide from CTR. Is there something more specific for the PDP that describes the "proper" shape of ferrules you could use? I assume all Wiedmuller would take round? - Sam
Ferrules are not recommended. It's expected that teams will use the Wago connectors as designed, with stranded wire.
Ferrules are not actually discouraged, but you should do your research to see what Wago and Weidmuller have to say about them. The "proper shape" of a ferrule almost always includes flat sides. You must use an appropriate ferrule crimper to make them. Square shapes are probably going to work best for the connectors we use, because they provide a large connection surface and you won't have to worry about accidentally getting them in the wrong orientation.
Greg McKaskle
07-04-2016, 02:41
As a CSA, the most common category of comms failures are power related. It stinks for a robot to lose power or reboot in a match, but if I'm able to show them the cause of the failure and discuss solutions, I consider that a good outcome. My favorite is when I narrow it down and the students actually find the issue and identify a solution. Many of these are simply loose nuts on the battery or breaker, -- sometimes the PDP. Sometimes it is a bad crimp on the same connections. Loose fuses on the PDP are another source of failure.
I have less familiarity with it, but I have occasionally recommended replacing the circuit breaker/switch because it seemed like the softest touch would cause a loss of power.
I have not seen that many issues with radio power, but I'm certainly not a fan of the barrel connectors and generally encourage some form of positive retention, at least a strip of electrical tape stretched front to back across the jack. In previous years, it was quite common to see a bad DLink at an event. Whether this was damaged through mechanical shock or vibration, static discharge, or was bad from the factory wasn't ever clear, but some DLinks simply had high loss that went away with a radio change. I have yet to see that with OpenMesh.
I would really like to see the OP logs. The cursors, messages, and timings give very good evidence of what happened on the robot. When the disconnect happens, several things take place. The DS starts to ping each element in the chain to identify what it can still see, and the roboRIO, if it is still up, will also ping the radio from its side and post-inject messages into the log.
Key things to look for:
If the comms disconnect is only a few seconds long, I generally find that the ethernet cable was not snapped in or wiggling/tugging will cause link lights to go out.
If the roboRIO reboots but the radio is still up, the FTA can easily see this and will often tell the CSA or team. The second tab of the DS also shows the ping status of each device. Also, the results of the pings are printed to the log file as messages. As the robot comes back up you will also see a code start message.
If the radio reboots and not the roboRIO, there will be a green vertical cursor between the orange and yellow ones. Hovering over pretty much anything on the graph displays a descriptor in the lower left. The green cursors are the roboRIO telling the DS when it saw what radio go away and return. This is also in the event messages.
If the radio went out and there is no green cursor and no code start message, both devices lost power.
If the code restarts and the outage is five to ten seconds, it is more likely a code crash and restart rather than a power issue. This also looks different because the radio and roboRIO are still present and the CPU trace will sometimes help indicate the type of crash.
I hope to be able to make the log or even DS make a diagnosis to categorize the issues, but it just hasn't happened yet. As for logging while the robot sits still. Some teams do this, and it is a fine thing to do with your team. I'm pretty comfortable with the diagnostic capabilities of the existing log files and I'm not sure that RIO logs will actually be that much more effective at diagnosing problems.
Several times as CSA, I've intended to keep details about each failure and keep stats of what was found. But it quickly becomes more important to help teams than to keep count. And some teams don't ask for help and you don't find out until second-day afternoon that they have had multiple issues and you were not at the field to observe them.
Greg McKaskle
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.