Driver-station Redundancy and Backup

Hi CD Community,

While competing in EF-6 at Battlecry @ WPI we experienced a new failure for us. We lost communication between our control system laptop and the FMS. We regained connection for a short period of time before it dropped again. It was the first time we had experienced any glitchy behavior from that laptop and the only failure we had at the event. We were told by the event staff that the issue was with our Ethernet port. Seeing as no other robots had issues with that particular driver station, I think their diagnosis is correct. We also saw no issues after the match when we performed our systems check and saw no issues prior to the failure. For the subsequent match we borrowed a USB to Ethernet adapter from our alliance captain 4564, Orange Chaos and had no issues.

We have a few Ethernet extensions and a USB to Ethernet adapter on their way but this seems more like a band-aid than a real solution.

This series of events has created a few questions:

  1. Is there a good way to test the mechanical and software connection between the laptop and the FMS during match setup?

  2. Is there any way we or FIRST could implement redundancy in that communication pathway?

  3. What actions does your team take to avoid situations like this and have you ever detected a fault before it occurred on the field?

Thanks,
CHAOS 131

1 Like

Having a USB to Ethernet adapter is not a bad idea. We’ve definitely run into issues with worn out Ethernet cables or ports not staying properly connected. (To the point where our operator would hold the Ethernet cable into the port for the whole match to prevent us disconnecting.)

I remember in 2017 at Buckeye having to tape our ethernet cable into the port to keep it in place. We had this happen a few times before an FTA said it was likely a bad ethernet port on the laptop. After that event we got a new laptop and never had the issue again.

We’ve had similar issues with ethernet ports getting worn out and loose. We’ve seem similar issues with USB ports for controllers too. I’d recommend you get yourself a port saver that you keep plugged into your laptop throughout the season. They are cheap (<$10) and avoid having to retire an entire laptop because the port is worn out. Here is one for sale on Amazon as a reference.

3 Likes

Oh yes, lovely memories of 2018 Miami Valley. Our old trusty reliable Classmate PC decided it didn’t like the FMS (or vice versa), so whenever we plugged it in to the network, the Ethernet driver shut down. That was fun. The Classmate worked fine on tether and elsewhere, so there was no way to predict this failure mode before connecting to the field.

A new laptop fixed the issue. We needed a newer, bigger one anyway for more camera and shuffleboard display real estate.

We keep at least one other laptop (usually a programming laptop) in the pit area that is capable of being a backup DS.

We’ve used an Ethernet pigtail port saver on our laptops to preserve the mechanical integrity of the Ethernet port. Rarely do we experience physical connectivity issues with this setup.

Before each event, I recommend to go to up to the people dealing with the FMS and ask them if you can connect your laptop to FMS to test the connection.

I’m really glad we did that at the last event because they were able to make sure we had everything set up on our computer correctly. They helped us with some simple stuff but I’m sure if you run into a problem more interesting they’ll be able to help as well.

I guess what I am really concerned about is detecting the Fault before it causes an issue. The Ethernet saver or USB adapter helps reduce the rate of occurrence, but does nothing to help detect the issue before it becomes a real problem. I’m looking for the action we should have taken after completing the systems check and before the match to ensure this isn’t an issue.

For context, we have had connection issues in the past and have dealt with them appropriately but this one came out of nowhere. We ran many matches at this event without a single glitch before this occurred.

  1. From my experience the best luck is proactively avoiding issues up front.

First, make sure you don’t have other apps on your driver station. Apps trying to check for updates on the network (now your robot) are known issues.

We had a similar issue this season where the laptop would lose the connection; we’d see the indicator on the driver station software blink red for an instance. The robot would stutter but the Comm indicator on the Roborio never blinked. It seemed to fix itself after we reentered the robot IP address in the driver station software. Theory is an IP address conflict or a messed up cache in MDNS.

Watch for broken latches on the Ethernet cable. Bringing spares (cables and adapters) is good practice.

Thanks, we will be implementing most of those activities and a few more. I just wish there was a device or piece of software that could test the health of the network connection and verify the connection was good and had plenty of margin.

The field connection has to be good for a match to even start.
There are already tests for that on your Driver Station app and the Field Monitor display.
So what you are looking for is something that will predict future mid-match failures?

Mark,

Coming from a Medical Devices background and not having too much knowledge of the FMS and networking, I am looking for something that can be predictive of failure or have a “Fail-operative” condition.

If I were designing the system from the ground up it might look like two independent and parallel paths of communication between the DS <-> FMS <-> Robot. Seeing as that is a pipe-dream, I was hoping for something that could show that a mid-match failure was more likely to occur as well as good steps to take to resolve the issue/check.

Teams have significant control over their robot configuration and can design out faults in their systems when they deem the issue to be significant. What we don’t have much control over is how our data gets to the robot.

The typical (although rare) failure of this nature is due to a bad PC Ethernet port, NIC failure, or other PC hardware or OS/software.
A redundant path would seem to require the use of a backup full second driver station PC.

I had a feeling the answer was going to be “not possible without a major overhaul of the FMS” :slightly_frowning_face:

We will do the best we can with what we have.

The simplest solution (from an FMS perspective) to your particular issue would be to take the PC out of the hands of the teams that are damaging them and bury them away from humans in a case like the SCC under the middle alliance station.
However, even that still leaves game controller connections (and hand-held controllers) to fail during a match. That’s actually a much more common failure condition for today’s field system.

1 Like

That’s an interesting idea. For a Pie-in-the-sky level approach, I was thinking about the possibility of using both Ethernet and USB for FMS/DS communications. Send the packet to the robot over both channels. The robot keeps a queue of incoming packets and only reacts to the most futuristic packet.

Too many other failure conditions on a PC, and we’d run out of available USB ports even sooner.
One of the more common failures during a match is Windows starting to perform an OS update :slight_smile:

The greater complexity of dual transmission paths from the same problematic source (PC) would, I think, introduce more potential error conditions.

For your dream scenario there are Ethernet protocols designed with physical layer redundancy for edge devices in mind. PRP uses parallel networks between all devices, or DLR allows for a ring topology with less cabling than full redundancy of the network. These require special hardware or drivers to be used though so you are looking at a rebuild of the FMS electronics and special drivers or cards on every team laptop.

I like the idea, But not the implementation. I think FIRST should look into putting USB hookups out with the Ethernet connections. More and more manufacturers are removing their Ethernet ports or are going to those fold-able hinges that don’t hold up to being Plugged In/Unplugged over 150+ times a season. Also USB is just more durable to being Plugged in/Unplugged, and can breakaway when you forget to unplug the cable so you dont slack-line your driver station or destroy a consumable Ethernet cable for a ridiculous reason.