Carver Connection Issues?

[mototom]

We have started to notice during the competition that there seems to be numerous connections issues on the Carver field centered mostly around the Blue Alliance Station, positions 1 and 3. Numerous times laptops need to be replaced with field laptops, robots lose connection during the match, USB joysticks re-calibrate due to disconnects causing erratic robot operation, and even the TI launchpad not operating properly.

Has anyone else noticed these connection issues on Carver?

[scca229]

I'm not there but every one of those symptoms is normally an issue on the driver station laptop itself and not the field/FMS. Are these "connection issues" being reported by a first-hand, was actually wearing a driver/HP/Coach button person? I only ask because from my FTAA experience, what is reported is not necessarily what the real issue was with no willful misrepresentation, but not having full grasp of the issue or how to describe it correctly. It is akin to Windows throwing an "out of memory" error (there are literally thousands of reasons that error is thrown, many not having anything to do with memory).

*Going off the top of my head from getting problem DS Laptops attached at 3 Regionals this year*
Has the laptop been confirmed to not have AV running? Auto-Updates of any kind turned off (Windows, AV, NI, Java,...)? Power saving features all disabled (including USB power saving)? Known working RJ-45 jack where jiggling the cable does not cause a port reset? Custom Dashboard turned off? Cameras turned off? Plugged into the provided power in the station? Set to High Performance mode so that the CPU is not throttled at any time? Wireless/Bluetooth disabled (not just disconnected but fully disabled), really any port that isn't the one connected to the field ethernet cable? Physical network connection set to DHCP? In the right station (you'd be surprised, or not, at how often this happens but YAY, the DS software tells you this year)?

If the Loaner Classmate worked for the match, then I would be looking at the DS Laptop and making sure everything is in order on it first. The FMS has *nothing* to do with what is running on the DS laptop, it only tells the DS software what mode it is in. Controllers disconnecting is ALWAYS a DS Laptop or controller issue and if you move the controller to another port and rescan (right in the DS is the rescan button), they typically start working again and then verify that power saving is OFF for USB ports when you get back to the pit.

[protoserge]

I'd venture to say the driver stations are the issue. The FTAs at championships are veterans and are very good at what they do. If it was a field issue, plugging in the Classmate would most likely not solve the issue.

My guess is that to keep schedule, the go-to fix for these issues is to substitute the Classmate and follow up with the team/give guidance on what to fix on their driver station before their next match.

Robots losing connection during match is not a field fault. It is a driver station or robot issue. FMS does not connect to robots, only driver stations. A loose Ethernet cable, loose robot radio power wire, unseated breakers, or other robot electrical issue is more likely the cause.

[mototom]

Originally Posted by scca229

*Going off the top of my head from getting problem DS Laptops attached at 3 Regionals this year*
Has the laptop been confirmed to not have AV running? Auto-Updates of any kind turned off (Windows, AV, NI, Java,...)? Power saving features all disabled (including USB power saving)? Known working RJ-45 jack where jiggling the cable does not cause a port reset? Custom Dashboard turned off? Cameras turned off? Plugged into the provided power in the station? Set to High Performance mode so that the CPU is not throttled at any time? Wireless/Bluetooth disabled (not just disconnected but fully disabled), really any port that isn't the one connected to the field ethernet cable? Physical network connection set to DHCP? In the right station (you'd be surprised, or not, at how often this happens but YAY, the DS software tells you this year)?

If the Loaner Classmate worked for the match, then I would be looking at the DS Laptop and making sure everything is in order on it first. The FMS has *nothing* to do with what is running on the DS laptop, it only tells the DS software what mode it is in. Controllers disconnecting is ALWAYS a DS Laptop or controller issue and if you move the controller to another port and rescan (right in the DS is the rescan button), they typically start working again and then verify that power saving is OFF for USB ports when you get back to the pit.

Thanks for taking the time it respond in such detail and with all these suggestions. No AV running. All updates turned off.All power saving features turned off, including USB port/hub power saving features. The ethernet cable never causes an issue in the pit or in prior matches/competitions, network set to DHCP. In the correct station. Moving the connector does cause it to reconnect properly, but it is time consuming, and has not happened before. Additionally, when the joysticks reset, the recallibrate with sometimes disastrous results. The only thing not tried was using driver station power. We will try that next. Thanks!

[protoserge]

Make sure IPv6 is disabled under your network adapter settings, as well as Windows Firewall and WiFi (shouldn't matter, but more of a "just in case").

[dodar]

Just from watching the stream, blue on Carver looks to be having constant connection issues.

[scca229]

Originally Posted by mototom

Moving the connector does cause it to reconnect properly, but it is time consuming, and has not happened before. Additionally, when the joysticks reset, the recallibrate with sometimes disastrous results.

Focusing on this part because this is solely a laptop issue. Once the controller is moved to another port and rescanned, before leaving the USB tab in the DS, press a button on each controller to make sure they are where they are supposed to be. If they were location locked before, at least one is very likely in the wrong place now and can cause some very interesting unintended operation. We had one multiple offender of this same thing at Arizona West that turned out to be the power saving on the USB ports...why I asked to check that item specifically. The team was pretty much ready to go with the laptop well before plugging into the ethernet cable and it turned out that the USB ports were just shutting off because the timer was expiring since no keys were touched on the keyboard. Activity on the joysticks didn't reset the timer.

Just watched match 103 which 329 was in and didn't see anything out of the ordinary other than the dance in auto, which I'm assuming is intentional.

[Landonh12]

I'm volunteering on Carver. Usually when we have a connection problem the FTAs go out and fix the issue almost immediately. I don't think we've seen one disconnect during a match so far.

There have been a couple of times that the connections have started to freak out, but not during a match. We're sorting it out.

[dodar]

Originally Posted by Landonh12

I'm volunteering on Carver. Usually when we have a connection problem the FTAs go out and fix the issue almost immediately. I don't think we've seen one disconnect during a match so far.

There have been a couple of times that the connections have started to freak out, but not during a match. We're sorting it out.

The Pink team's last match they died and then came back.

[scca229]

Originally Posted by dodar

The Pink team's last match they died and then came back.

What is "died and then came back"?

Possibilities I can come up with for just on the robot itself:

• Battery cable loose at Anderson briefly disconnecting causing full robot reboot?
• Battery cable loose at main breaker briefly disconnecting causing full robot reboot?
• Battery cable loose at battery briefly disconnecting causing full robot reboot?
• Battery cable loose at PDB briefly disconnecting causing full robot reboot?
• roboRIO power cable loose at PDB causing brief disconnect rebooting roboRIO?
• roboRIO power cable loose at roboRIO end causing brief disconnect resulting in roboRIO reboot?
• Red fuse not fully seated in PDB causing brief disconnect and roboRIO reboot?
• Yellow fuse not fully seated in PDB causing brief disconnect and radio reboot?
• Radio power cable loose/stray strands at VRM causing brief disconnect and radio reboot?
• Radio power cable at radio causing brief disconnect and radio reboot?
• Ethernet cable not fully inserted in roboRIO causing brief disconnect and reattach?
• Ethernet cable not fully inserted in radio causing brief disconnect and reattach?

The above is only considering components that the robot HAS to have to pass inspection (and I'm probably missing some). I have no clue of any additional stuff on that particular robot to troubleshoot since I don't know what other components are on it. Each of the above issues can be narrowed down based on how long it was out of commission. 10 seconds or so is probably a radio power issue. 40 seconds is a typical roboRIO reboot. 3 or 4 seconds might be the ethernet cable.

As an FTAA, when I see a bot stop moving or the connection monitor shows a red box somewhere on their line, I make a quick assessment of what I can see via status lights (THANK YOU teams that mount the roboRIO & radio in easily visible locations) and what body motions the team is making behind the glass (do they look confused?). I look at the radio and see if the status light is red, indicative of a reboot and power issue and I look for the roboRIO to see what it shows. When the radio goes to 2 blue lights, if the bot doesn't move fairly quickly, then the lights on the roboRIO will tell if it is booting as well. That narrows down whether the whole bot lost power, or only a part of it so that I can notify the team and/or RI/CSA to help them out.

Possible causes on the DS Laptop are spelled out earlier in the thread along with quite a few more that I'm not able articulate in the few minutes I took to type this out.

Did the RSL stop blinking when the "died and then came back" occurred? How long was the "non-motion" period? Was the "non-motion" intentional? On bots that the teams have mounted the RSL is a location visible from all around (yes, I know the Robot Manual only says from the front...wish they'd change that), I can easier tell if the non-motion is due to the robot not receiving from the DS. If it is flashing, I'll stay put but keep a closer eye on it until they move again.

I'm not saying that the FMS can't ever be the cause of an issue, but the vast majority of the time, the cause is able to be traced to something on the robot or on the DS laptop. Normally when the FMS is involved in the problem, an entire alliance or both alliances experience the problem as they are sharing the same Layer 1 resource, albeit in their own logical sandbox.


Any additional info on what the "died and then came back" issue was?

[protoserge]

"Died and came back" sounds like one of the following:

1. Ethernet port issue from DS to FMS - unlikely if nothing was touched, but possible.
   
2. RoboRIO power
   
3. Radio power

Check the battery log. Did you brown out? What was the maximum and minimum voltage? What was the latency?

If the duration of the "died and came back" was on the order of 20-30 seconds, I would suspect a radio reboot. Look at the radio power supply from the radio all the way to the battery. TUG on wires. If they come out of a connector or unplug you found the problem. A solid tug on any of the power wires should never pull the wire out of a WAGO or crimp.

[mototom]

First, thanks everyone for helping out with this issue, it has been pretty frustrating for us and we just want to make sure everything is working ok.

Second, forgive me if I don't quote or post properly formatted responses as I am not a CD maven.

So, just to update what happened next.

Quote:
Originally Posted by Landonh12 View Post
I'm volunteering on Carver. Usually when we have a connection problem the FTAs go out and fix the issue almost immediately. I don't think we've seen one disconnect during a match so far.

There have been a couple of times that the connections have started to freak out, but not during a match. We're sorting it out.
The Pink team's last match they died and then came back.

I saw that match and went and checked their driver station log and they did indeed disconnect during that match. They did not experience a brownout, and I did not troubleshoot anything on their robot, nor do I know if they experienced any further issues.

Possibilities I can come up with for just on the robot itself:

I cut off the list to save space. We did indeed check all these things. We have a pretty solid electrical engineering mentor who is fanatical about his team's wiring and they had gone over everything. We detect brownouts in our code because our encoders might reset, and we checked the DS logs and saw no evidence of that. The ethernet port has a pigtail on it, but again, we have not seen this issue in the pits, any of our prior regionals, or in any of our prior matches. Our RSL stayed solid the entire time.

The bottom line is that our last three matches of the day were all on the red side and we experienced NO ISSUES during that time after having made no changes to our robot or driver station.

Just watched match 103 which 329 was in and didn't see anything out of the ordinary other than the dance in auto, which I'm assuming is intentional.

Yes, we were on the red side, and our robot was the last to connect, and our sendable chooser that picks our autonomous mode did not come up until the instant before the match started and our drivers were unable to pick the correct autonomous mode. We have since changed the default autonomous mode to be the one we intended on using from now on. So, the dance, while not intentional and perhaps amusing, was not related to this issue.

Robots losing connection during match is not a field fault. It is a driver station or robot issue.

While I agree that this is usually the case with one robot having connection issues at multiple drivers station. When many robots seem to have it at one driver station or one alliance side, then it makes me question whether this is isolated to a single robot, especially this late in the season when all of these robots have been connected to the field dozens of times before.

A big shout out to the FTA who was there for us during our last match. (Sadly, I do not know his name, or I would mention it here) He made sure we had the time to get our sendable chooser and select our autonomous (although it was already fixed). He also stayed right by us the entire time with his tablet making sure everything was running smoothly for us. That was terrific and we really appreciate that! He reported no issues to us regarding any of this.

[lriddick]

We had a similar situation in DC, where our robot went so called dead during quarter finals, not only did it happen to us but it happen to our alliance team mates at the same time. When the FTS came by he said that the USB connection was lost. We did a little research after the fact at received the log files from the other team that this happened to as well. Come to find out Both robots lost there USB connection at the same time, coincidence.. The other teams robot recovered automatically we had to pushed F1 and was able to recover but it was to late, could there be a bug somewhere??

[protoserge]

Originally Posted by lriddick

We had a similar situation in DC, where our robot went so called dead during quarter finals, not only did it happen to us but it happen to our alliance team mates at the same time. When the FTS came by he said that the USB connection was lost. We did a little research after the fact at received the log files from the other team that this happened to as well. Come to find out Both robots lost there USB connection at the same time, coincidence.. The other teams robot recovered automatically we had to pushed F1 and was able to recover but it was to late, could there be a bug somewhere??

Did both laptops have the same USB power saving function on?