How to determine root cause of robot dropping from Teleop to Disabled during match.

[tr6scott]

At West Michigan.
Match SemiFinal 2
Video of match: https://www.youtube.com/watch?v=ib3zz6QXHbc
Robot in question is 2137 TORC, bottom right hand corner.

We experienced two "field disconnects" in this match, the first one lasted about a second, and the second one lasted about 3/10ths of a second.

This data comes from our logging on the roborio on the bot. Our logging completed on the bot, is only done during autonomous and teleop, so when we dropped to the disabled state, we didn't log data, but I can tell this based on the gaps in the data time stamp. These events are highlighted yellow in the attached robot log file.

I am the lead electrical and programming mentor, but was not at the event. I am doing a postmortem trying to determine the root cause. The team was told by the FTA that the issue was not with the field, and there no basis for a replay. TORC bounced back from this anomaly, and the alliance was able to capture the blue banner.

Based on our log data, I know the roboRio did not lose power or reset. From the length of time of the glitch, only being a second, that the robot radio probably was not the cause, as that is a typical 30 second reboot. Also, the robot worked fine after the comms were re-established, but our elevator carriage had been sprung from the frame, rendering it useless. In the video, you can see the drivers and FTA moving the driver station around, and the robot springing back to life. You can also see the stack light go from solid red, to flashing red, back to solid red.

We should have driver station logs available on Tuesday, when we unpack, so one of the questions is what should I be looking for in the driver station logs for this match? I have not done much with the driver station logs, as we typically try to log what is important to us on the bot itself. Can you point me to a doc that describes driver station log files, and how to read the data?

A couple other questions for my general knowledge, what determines if who is at fault on a FRC field, when a glitch occurs?
What data determines this, and do we have access to this?

Just a general comment about the field at West Michigan. There seemed to be many delays, much more than any of the other fields in Michigan, along with more replays on the West Michigan field than the other two. Just an observation as I watched the other two webcasts play finals, while we were trying to finish quarter finals.

Thanks for any help.

[plnyyanks]

You are correct - connectivity drops that short are not indicative of a robot power issue. Here are the things I would look for if I was FTAAing and you came to me with this.

- Error logs stored on the DS (there's a tab in the log viewer to show these). It's possible that some of your code is taking too long to execute and tripping a watchdog, disabling you briefly. This should be logged for you to go back and look at. This is more likely to explain the second, shorter disconnect.
- It's hard to tell from the video where/how your bridge is mounted. The antennas run the long ways down the bridge, and that should not be parallel to metal and the bridge shouldn't be near any forms of RF interference (power wires, motors, etc.). This could lots of dropped packets depending on your robot location/proximity to the field AP/current wifi situation. The DS logs should show packet loss, and if this is the problem, you'll probably see it throughout the match, just maybe not as bad.
- How's the ethernet port on your DS laptop? The ports tend to not be very resilient to the constant plugging/unplugging of field connections, especially if it's an older laptop. Ethernet pigtails are a good investment. If something is stuck in the port, or if it's worn out, it's possible you could momentarily lose connection.
- What other software is running on the DS? Is there a firewall? If so, that's most likely the cause - they like to block DS traffic and cause disconnects. My recommendation is to totally disable it (that way you know it's not interfering with anything). If you don't want to do that, make sure that the Driver Station program and the NI mDNS Responder are whitelisted.
- Is your DS laptop used for anything else? I've seen instances where other programs that require an internet connection (Dropbox, the Autodesk updater thing, other update services) interfere with DS communication in just the way you described - momentary losses while the ping for updates. Killing their processes in task manager is the quick fix, uninstalling the thorough fix. If you can, have a laptop totally dedicated to being a DS, with as few other things running as possible, so there are minimal causes for niggling issues like these.

That's about all I can suggest without seeing more info. Post the logs when you get a change, and I'll be happy to take a look.

Originally Posted by tr6scott

I have not done much with the driver station logs, as we typically try to log what is important to us on the bot itself. Can you point me to a doc that describes driver station log files, and how to read the data?

Here's an overview for the log viewer:
http://wpilib.screenstepslive.com/s/...og-file-viewer
It has some examples of logs indicating common failures, which would be good to compare yours too.

Originally Posted by tr6scott

A couple other questions for my general knowledge, what determines if who is at fault on a FRC field, when a glitch occurs?
What data determines this, and do we have access to this?

This is a decision made by the FTA. Sometimes it's obvious, sometimes it's more of a grey area. Replays are typically granted if the most likely cause for a connectivity issues was field-related (this is a judgment call; again, some are easier than others). There are a lot of factors that to into making the call, and it's very dependent on the specific situation. Since I'm not an FTA, I'll let those who are fill in the details.

The main data points we use on the field are:
- Watching the Field Monitor and area lights for indications of robot issues. Read the section entitled "What Information Does FMS Log" in the White Paper. It's a little dated now, but the list of items logged is basically the same. The Field Monitor looks like a table, with one row for each of the 6 robots and has the following columns: ethernet links (is something plugged into the player station), DS connected (is the DS communicating with FMS), radio connected, robotRIO connected, battery, bandwidth usage, and packet trip time. You've probably seen this up on a monitor at the scoring table. If you ask one of the field staff to take a look with something specific to check, they'll usually show you what's logged.
- Status lights on your robot, both radio and roboRIO. Make them visible from across the field, it's unbelievably helpful. If we can't see the status lights, it's much harder to diagnose issues, which means you'll be less likely to be granted a replay.
- DS logs. Data there is logged more frequently than on FMS, so often those are the most helpful. Check for logged errors, battery drops, packet loss, things I mentioned above.
- Looking at your physical robot. Is everything plugged in properly? Are any wires loose? Are the PDP fuses in all the way? Are you sure they are?

[118418]

Originally Posted by plnyyanks

You are correct - connectivity drops that short are not indicative of a robot power issue. Here are the things I would look for if I was FTAAing and you came to me with this.

I'd second these suggestions.

Looking at the log, there are two periods, ~23s-34s, and ~35s-39s into the match, where your LoopTime (Column E) stays flat at 20ms. They correspond to your video when the the bot was staying still. This and all other log data during these same periods indicate that the bot was running fine but was idle. So, it must be the communication or the DS not sending commands over.

[ozrien]

Hey Scott, can you zip up all the logs from that day (yes all of them)? If you can get them up before wed, I can take a quick look (Wed is travel day for next event).

Default folder location:
C:\Users\Public\Documents\FRC\Log Files

[tr6scott]

Attached are all of the log files from last Saturday.

I believe the match in question is 2015-03_21 16_24_57

Thanks for all the help guys!

P.S. There appears to be no additional software made it onto this classmate.

[tr6scott]

Picture of the electronics board. Radio is mounted on 1/4" Baltic birch plywood, and the cover, that removes is also 1/4" Baltic birch. They are separated on the sides by 1"x 2" Al. tube.

[Greg McKaskle]

The logs have info for the following matches, and I put some comments beside those worth commenting on.

All of the matches have an error near the beginning of auto having to do with WAIT_FOR_PID. I don't know what impact this would have on auto. Many of them also have joystick errors that seem to occur in Begin.

Qual 3, 13, 20, 26, 29, 37, 52, 55, 66 -- clean
Qual 47 -- two second glitch during auto after your robot stopped moving, another that was one second long.
Qual 71 -- lots of stuff happening before the match started, but clean for the match.
Qual 79 -- pretty clean, but some packet loss a third through tele, no disable.
Elim 4, 8, 12, 15, 16

Elim 10 -- Comms are very clean, but robot drops at 2:24:49.250. It came back for just a few tele packets about 11 seconds later, dropped again a few times over the next 5.5 seconds, and then continued with very clean comms. At 2:24:59, there is a message that says no code is running, and at 2:25:50, there are joystick output errors that occur in most of your log files when your code starts up. Possibly in Begin. So it looks kinda like the code restarted, but I can't say why it happened, and I can't say for sure that this even happens in Begin. I also see the FMS times out and disappears about the same time as the robot. So I tend to think the ethernet cable came loose from the laptop and that was the main issue. It would be good to know if the FTA or students think that was what happened. As for the error regarding joystick outputs, it would be good to know when the code does this to know if this is really Begin or somewhere else.


Elim 14 -- clean for the match, but noisy after the match

I wouldn't worry about the radio or radio placement. I'd check the laptop ethernet retention and speak with the drive team to see whether they think the cable could have been the issue. I'd look at the code a bit for the joystick output error, and perhaps even pull the cable to see if the error shows up in the log when the DS and robot reconnect.

Hope this helps.
Greg McKaskle

[plnyyanks]

Greg hit all the important things in there, and I agree the next step is getting some input .

Next time you get some hands on time with your robot, go check the PDP fuses. Nothing I've seen indicates its an issue, but if not pushed in all the way they slowly wiggle out over time and can cause some strange issues (not always full reboots, either). Here's a post showing what a completely pushed in fuse looks like.

[tr6scott]

The WAIT_FOR_PID is a command we use in our beescript autonomous. When auto starts, we do an elevator reset, which homes the elevator, and resets the encoder, then puts the elevator in pid control. This command just waits until the elevator is under the pid control, before sending it a target height. Obviously, they have a typo in the script or the command name and they don't match up. Not an issue for this event, as we were just sitting and not moving in auto, but something on the list to fix.

The joystick error has been happening all season, but has not been an issue, except for throwing the error. The code runs in periodic tasks, and is setting the joystick rumble for the driver and operator. The operators joystick will rumble if he hits an overtravel on the elevator, and the drivers joystick will rumble when the there is a tote in position to stack.

Greg, if you want to send me you github username I will add you to the repository, if you want to check out the joystick error.

I will try to get additional information from the drivers at tonight's meeting.

We also checked all of the fuses after this match, and nothing was found to be loose.

[Alan Anderson]

Originally Posted by tr6scott

We also checked all of the fuses after this match, and nothing was found to be loose.

The way you worded that makes me wonder if you are talking about the right thing. The 10- and 20-amp PDP fuses are the important ones here. They're the red and yellow miniature automotive fuses on either side of the Weidmuller terminals for the roboRIO, PCM, and VRM power. If one of them has an intermittent contact for a moment, the roboRIO loses power. If the other one fails briefly, the VRM (and thus the D-Link) loses power.

Even if not fully seated, they usually won't feel loose. But not feeling loose doesn't mean they are making proper contact. If you can pull them out by hand, they weren't in all the way. It takes some serious pushing to get them in correctly, and once that's done, they definitely won't come out on their own.

[tr6scott]

Alan,

Yes, those fuses were included in the "All fuses" were checked. We experienced a disconnect at our Waterford event (Week2), and when we were alerted of the problem, we found one of the ATM fuses to not be completely seated. Having already been through this scenario once this season, that was the first thing I had instructed the pit crew to confirm as they were fixing the elevator after the match.

But I am now going to go back to those logs and review that match too.

[Monochron]

We had a very similar situation in our quarter finals match. The first one we exhibited similar behavior all throughout teleop (autonomous went fine). The second match we only dropped out for about 30 seconds or so. The DS logs showed that we lost communication with the FMS for the duration of those times. CSA's suggested that this was caused by our radio losing power, but it looked like they couldn't investigate the issue further than that.

I'm eager to see if you can resolve this, and I'll see if I can make any progress on evidence for what caused our problem.

[billbo911]

Originally Posted by Greg McKaskle

....

I wouldn't worry about the radio or radio placement. I'd check the laptop ethernet retention and speak with the drive team to see whether they think the cable could have been the issue. I'd look at the code a bit for the joystick output error, and perhaps even pull the cable to see if the error shows up in the log when the DS and robot reconnect.

Hope this helps.
Greg McKaskle

Greg has a very strong point here. RJ45 connectors, while normally reliable, can become fairly unreliable. Many factors can contribute to this issue. This is a big issue on my team. We have managed to damage beyond repair a few laptop's Ethernet ports.
The repeated connecting and disconnecting that takes place with most Driver's Station laptops can easily contribute to their early demise. There are a couple things that can be done to help reduce this wear and tear. One of the simplest and easiest ways to do this is to use an Ethernet Port Saver. Install it and tie it down so the end attached to the laptop doesn't move WRT the laptop.

Another issue I have personally seen giving lots of teams grief, and may apply to this thread directly, has to do with the design of the Ethernet port on the laptop used for the Driver's Station. This design can easily cause intermittent connection failures as well as the cable completely coming out of the laptop.
Here are a couple pictures of my personal laptop that has been used as a Driver's Station and has also completely lost the cable when bumped just a slight bit. This is an Acer laptop, but I have seen this same configuration on several different manufacturer's laptops.




In the second picture I am pulling down on the small cover plate that opens up when the cable is inserted. This cover rests right on the latch clip for the cable. Any pressure on the cover causes the cable latch to release. Believe me, it happens A LOT!!

The resolution to this is to make a relief notch in the Driver's Console assembly so that no contact with the cover plate can be made unintentionally.

[Tom Line]

Originally Posted by billbo911

Greg has a very strong point here. RJ45 connectors, while normally reliable, can become fairly unreliable. Many factors can contribute to this issue. This is a big issue on my team. We have managed to damage beyond repair a few laptop's Ethernet ports.
The repeated connecting and disconnecting that takes place with most Driver's Station laptops can easily contribute to their early demise. There are a couple things that can be done to help reduce this wear and tear. One of the simplest and easiest ways to do this is to use an Ethernet Port Saver. Install it and tie it down so the end attached to the laptop doesn't move WRT the laptop.

Another issue I have personally seen giving lots of teams grief, and may apply to this thread directly, has to do with the design of the Ethernet port on the laptop used for the Driver's Station. This design can easily cause intermittent connection failures as well as the cable completely coming out of the laptop.
Here are a couple pictures of my personal laptop that has been used as a Driver's Station and has also completely lost the cable when bumped just a slight bit. This is an Acer laptop, but I have seen this same configuration on several different manufacturer's laptops.




In the second picture I am pulling down on the small cover plate that opens up when the cable is inserted. This cover rests right on the latch clip for the cable. Any pressure on the cover causes the cable latch to release. Believe me, it happens A LOT!!

The resolution to this is to make a relief notch in the Driver's Console assembly so that no contact with the cover plate can be made unintentionally.

I have an acer laptop as well, and I'm convinced whoever designed that ethernet port was paid to find the most contrived, poorly engineered way possible to design it. Of course, I didn't realize it until FIRST season started, because how many other times do you plug your laptop into ethernet except at competitions?

Bleah.

[MrRoboSteve]

Other options:

Buy an old Thinkpad like this, and put a small SSD in it.

Use a USB2 to Ethernet adapter.