COMM lost way to many times

[pfreivald]

We had this exact problem at TVR in our first four matches.

It wasn't the fuses.
It wasn't the wire connections.
After isolating and ruling out all other issues (and reproducing the fault by giving the PCP a shake), we swapped the PCP and the problem didn't repeat. [Side note: the PCP, barely used since coming out of the kit, rattled like a maraca when shaken. Back home, the simplest wiring setup possible, very secure, would lose voltage when shaken.]

In our second-to-last qualifying match, half of our pneumatic system failed to actuate. We checked our wiring and all seemed fine, and it worked in the pit (even after some jostling)--we couldn't reproduce the failure. It failed the next game as well, and we couldn't reproduce the failure. After QF 1 we swapped the PCM for a new one and it worked fine through our last two matches. We have not at this time tried the PCM in other situations.

It's aggravating to have failures in KOP devices that we're required to use on our robot, but cannot repair or upgrade to something more reliable. FRC requires robust components, and expecting teams to deal with internal failures of KOP parts used in a manner consistent with their intended purpose as "part of the game" is not acceptable--quality control on FRC KOP components should be top notch, and it simply isn't.

(The "tested" sticker on the back of the PCP was a bit of an insult-to-injury chunk of irony. They said it was tested...but never said whether or not it worked.)

[Alan Anderson]

Originally Posted by Bruceb

Ok I read that link and I want to make sure I understand how you are using this.
You take a port saver cord, open the cable somewhere and solder a red wire to the blue and white with blue stripe wires and a black wire to the brown and white with brown stripe wires then you stick those wires into the VRM, plug the port saver into the POE port on the radio and the ethernet cable from the RIO into the end of the port saver cable you just modified so you are using only ONE of the ethernet ports on the radio, correct?

We plug the port saver into the roboRIO, not the radio. The power wires only need to reach from near the roboRIO to the VRM. Usually the radio is mounted some distance from the rest of the control system components, and it's convenient to run a single cable from the port saver up to the radio.

The Ethernet port next to the coax power input is the only port on the radio that accepts this scheme.

Do you cut the wires in the port saver so power only goes to the radio or do you just solder to those wires so power goes to both the radio and the RIO?

The blue and brown pairs from the "plug" side of the port saver are indeed disconnected in the one we have installed on the robot, but that was mostly for ease of construction. I have used a different "passive injector" during testing and verified that having power going to those pins on the roboRIO as well does not break anything.

[Bruceb]

Originally Posted by Bruceb

is it possible that the main breaker problem you reference can cause the radio to reboot without the RIO rebooting?
All indications are that our RIO is not rebooting.

bump

[RufflesRidge]

Originally Posted by Bruceb

bump

Extremely unlikely. Can you elaborate on the indications? The implication you were provided by the FTA\CSA earlier in the thread that the roboRIO takes longer to boot than the radio is false.

[scca229]

Originally Posted by RufflesRidge

Extremely unlikely. Can you elaborate on the indications? The implication you were provided by the FTA\CSA earlier in the thread that the roboRIO takes longer to boot than the radio is false.

Correct that the roboRIO boots quicker, but if both of them went down, then communications between the roboRIO and DS aren't going to come back until 10-15 seconds after the radio is fully back up (50 seconds or so itself). Net is 60-65 seconds before the robot moves again. That may be what the FTA was trying to say.

[Joe Johnson]

Originally Posted by scca229

Correct that the roboRIO boots quicker, but if both of them went down, then communications between the roboRIO and DS aren't going to come back until 10-15 seconds after the radio is fully back up (50 seconds or so itself). Net is 60-65 seconds before the robot moves again. That may be what the FTA was trying to say.

Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?

I get the feeling that the problems are worse this year but I freely admit that I am a biased observer. If FIRST has data that shows that this problem is small and getting smaller then, well, I won't say I'll be happy, but I will at least feel better knowing that the ball is moving toward the right end zone.

Dr. Joe J.

[mdballard]

We had issues with our development robot. It would drop the radio almost any time we went over a defense. All of our other components stayed powered. The RIO never rebooted. We did several things to harden the control system:
1) made an improvised shock plate to mount the radio to. This dampens the shock transferred to the radio itself. There seems to be a strong indication that high G shock, as opposed to vibration, is an issue for the new radio.

2) We use a plug with screw terminals for the radio power. Once the wires were screwed into the terminals on the plug adapter, we soldered the wires in and insulated the leads. We verified isolation between the leads. Then zip tied the plug adaptor to the shock plate such that: 1) the tension relief moved as a system with the shock plate and radio, 2) would not move relative to the radio itself. We also zip tied the Ethernet cables to the shock plate.

3) For all wiring connections, we made tension reliefs, out of semi-rigid lexan and non-conductive foam tape, that could be zip tied to the component with the connections. This greatly reduced the movement of the wires in the connectors relative to their connected components.

4) We made sure the radio had physical separation from other CAN bus connected components. We found, as some other teams, that if the radio was placed too close to the PCM it appeared to interfere with the CAN transceiver in the device. At least that is how the issue was manifesting from observation without opening the PCM itself. When we relocated the radio, the issues went away. This was not a problem with last years radio.

5) We use 45amp power pole connectors from our TalonSRX controllers to the motors.

6) We use star rings on the main breaker and ensure the connections are tight. We fortunately did not have a faulty main breaker, but that is something that can be checked as described by others in this thread.

7) We perform a check on the robot before it leaves the pit prior to every match.

In the STL regional we didn't have any loss of communication with the robot during any of our matches.

We did have a problem before our first qualification match where the Ethernet port on our driverstation laptop was accidentally broken. But that had nothing to do with the robot. We fixed that issue with a new laptop that uses a full size port as opposed to the low profile. We also incorporate an Ethernet dongle that is fastened to the carrier.

All that said, the game has been fairly brutal this year. Definitely need to keep inspecting and repairing after every match to avoid failures due to damage.

Matt

[Schnabel]

Originally Posted by pfreivald

.....
It's aggravating to have failures in KOP devices that we're required to use on our robot, but cannot repair or upgrade to something more reliable. FRC requires robust components, and expecting teams to deal with internal failures of KOP parts used in a manner consistent with their intended purpose as "part of the game" is not acceptable--quality control on FRC KOP components should be top notch, and it simply isn't.

(The "tested" sticker on the back of the PCP was a bit of an insult-to-injury chunk of irony. They said it was tested...but never said whether or not it worked.)

Out of curiosity, have you contacted CTR-E to see if they can help determine the root cause for the PDP and PCM? They are very easy to work with and very smart, I wouldn't be surprised if they end up helping find a root cause for both and then helping you get it fixed. Additionally how long did it take for you to replace what you assumed (and later verified) are bad components? I'd like to take this time to remind everyone that Spare Parts has multiple of these required components specifically for teams who are having issues. It's understood that you cannot open or modify them, so swap them out for an event and see if that helps.

Originally Posted by Joe Johnson

Does anyone have data on the frequency, number, and duration of COMM problems over time (meaning year over year)?...

Joe, I get you are trying to stay unbiased in this, but you are also trying to point fingers at FIRST for any dropped communication issues that appear on the field. That isn't a fair assessment. There's still a level of responsibility for the teams to take care of their own hardware and ensure that their robots are capable of playing the game at hand. Teams that take the time to make their wiring clean and ensure proper connections and relief and provided to their electronics rarely have issues. For example, one of the teams who dropped comms constantly at a previous event I attended had a rats nest of wires and relied on gravity and double sided tape to hold pieces together. Every time they lost comm we were able to determine what had come loose. Sometimes comms drop and a cause cannot be determined, that's just a fact of life when you are on the field. Even on my high school team this happened, eventually we found the issue or found a work around to prevent it (looking at you, 2009 bot who wouldn't boot properly every time). My point is, there are a lot of troubleshooting steps teams can and should take prior to entering the field to play. These include tug tests, checking for loose connectors, etc. You wouldn't believe the number of teams I've told to do that very thing (even at the drivers meeting) and they don't.

I don't have any data to back me up, but here are my general impressions of the control system year over year. Note that when a robot drops I attempt to do everything in my power (and allotted time) to find the issue on the field itself. I hate phantom issues showing up in the middle of a match. It's not inspiring to the students, it's not fun, it may provide a challenge to the teams, but they shouldn't have to worry about these types of issues during a 2.5 minute span. I would say I have about a 90% success rate and determining and replicating the issue on the field.

2011 - 2013 - I remember hating the control system during these years, I just can't remember exactly why.

2014 - Horrible comm drops due to multiple issues and with no way to easily explain why. I remember issues with the radio and crio rebooting in the middle of a match being most common. This was caused primarily by brown outs to either, or loose wiring. The brown outs occured most frequently in the last 60 seconds of a match, especially when the robot started playing defense. Pushing matches stressed the control system beyond what it could handle. Additionally I remember parts of the control system having to be swapped out randomly due to shorts and failures (digital side car, radio power converter, etc.)

2015 - Hardly any comm drops compared to previous years. This was a refreshing turn after concerns of switching to a new control system. There were multiple factors that helped including the low stress on the robots, stronger connections (when wired properly), brow-out protection being built into the roborio, and the vrm keeping the radio alive during short voltage drops that were low enough to take out the rio. There were multiple instances of robots dropping below the rio's threshold, causing a reboot, but the radio stayed up and connected. This was the most solid I've ever felt our control system has been.

2016 - I feel like this year is some-what in between 2014 and 2016. Issues from wires coming loose or pulling out are back due to the intensity of the game, but we still have many of the protections in place that helped the control system in 2015 feel solid (vrm and brownout protection). The big issue I see this year is just how long it takes to get communication back with your robot. A team can die at the start of teleop, then start moving with 20 seconds left. One of the most annoying aspects of this year's control system is starting robot code after the DS has full communication back up. It can take the 50 seconds it takes the radio to boot up to nearly 70+ seconds just for the robot to be enabled again. In other words, if you drop, the chances of coming back in time are slim.

[The Lucas]

I initially had radio power problems during the build. I decided to strip the 24 AWG wire in the power twice as long as recommended (~3/4in), fold it in half, and point solder the end back the beginning of the strip (to make it more manageable during insertion). I know the VRM is rated for the 24 AWG as its minimum, but I feel that this loop provides a more reliable connection than just 24 AWG stranded. Just be careful not to force it down too much during install.

Also, if you mounted your radio with the barrel connector facing toward the floor, I suggest you rotate it so gravity is not working against you.

I continue to have major power problems using the old DLINKs and I am considering soldering directly to the PCB. So I think the problem is not specific to the new radio.

[pfreivald]

Originally Posted by Schnabel

Out of curiosity, have you contacted CTR-E to see if they can help determine the root cause for the PDP and PCM? They are very easy to work with and very smart, I wouldn't be surprised if they end up helping find a root cause for both and then helping you get it fixed. Additionally how long did it take for you to replace what you assumed (and later verified) are bad components? I'd like to take this time to remind everyone that Spare Parts has multiple of these required components specifically for teams who are having issues. It's understood that you cannot open or modify them, so swap them out for an event and see if that helps.

I haven't contacted them as yet--too busy preparing for FLR this week!

It took maybe 20 minutes to replace the PCP, and ten to replace the PCM.

Our gear boxes were an accessibility issue that is, ultimately, our fault; the robot is packaged so tightly that we'd have to tear it apart to get the gearboxes off.

[jojoguy10]

Originally Posted by Alan Anderson

We plug the port saver into the roboRIO, not the radio. The power wires only need to reach from near the roboRIO to the VRM. Usually the radio is mounted some distance from the rest of the control system components, and it's convenient to run a single cable from the port saver up to the radio.

The Ethernet port next to the coax power input is the only port on the radio that accepts this scheme.



The blue and brown pairs from the "plug" side of the port saver are indeed disconnected in the one we have installed on the robot, but that was mostly for ease of construction. I have used a different "passive injector" during testing and verified that having power going to those pins on the roboRIO as well does not break anything.

Hey Alan, I've been following this thread, and I wanted to make sure I understand everything correctly. Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?

Thanks,
Joe

[cbale2000]

I feel like we've had an absurd number of coms issues ever since we went to WiFi-based radios. I miss the old pre-2009 control system that would boot in 5 seconds and connect instantly. Basically the only coms issues I ever had with those systems was when something was physically damaged during a match. We should have stuck with something like that and just used WiFi for on-board video streaming, that way a connection failure doesn't shut down the whole machine for the entire match.

[/rant]

[Alan Anderson]

Originally Posted by jojoguy10

Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?

What you show will work, but it's backwards from how I did it. Wiring the power into the cable connected to the radio fails to realize one of the main advantages of Power Over Ethernet. If you wire it the other way around, with the power wires going into a short extension plugged into the roboRIO, you can put the radio at any distance from the power source by using a standard patch cable between it and the powered Ethernet jack.

[olapmonkey]

Originally Posted by jojoguy10

Hey Alan, I've been following this thread, and I wanted to make sure I understand everything correctly. Attached are a couple of pictures I took of the setup that I think you're describing. Could you double-check my work to make sure I'm understanding you correctly?

Thanks,
Joe

This matches the PoE configuration we shop tested last night. Being paranoid now due to prior issues, we verified that we can provide power to the radio over both PoE and the standard barrel jack. We can unplug either individually and the radio won't lose power. So we now can have redundant sources of power to the radio from the VRM or pick and choose them individually. Overkill? Yes, but when your radio is intermittently rebooting during match play it's good to have a few extra tricks in the bag.

[jojoguy10]

Originally Posted by Alan Anderson

What you show will work, but it's backwards from how I did it. Wiring the power into the cable connected to the radio fails to realize one of the main advantages of Power Over Ethernet. If you wire it the other way around, with the power wires going into a short extension plugged into the roboRIO, you can put the radio at any distance from the power source by using a standard patch cable between it and the powered Ethernet jack.

Oh, duh! It was late last night. Haha! Thanks Alan!