How to determine root cause of robot dropping from Teleop to Disabled during match.

[tr6scott]

Attached are all of the log files from last Saturday.

I believe the match in question is 2015-03_21 16_24_57

Thanks for all the help guys!

P.S. There appears to be no additional software made it onto this classmate.

[tr6scott]

Picture of the electronics board. Radio is mounted on 1/4" Baltic birch plywood, and the cover, that removes is also 1/4" Baltic birch. They are separated on the sides by 1"x 2" Al. tube.

[Greg McKaskle]

The logs have info for the following matches, and I put some comments beside those worth commenting on.

All of the matches have an error near the beginning of auto having to do with WAIT_FOR_PID. I don't know what impact this would have on auto. Many of them also have joystick errors that seem to occur in Begin.

Qual 3, 13, 20, 26, 29, 37, 52, 55, 66 -- clean
Qual 47 -- two second glitch during auto after your robot stopped moving, another that was one second long.
Qual 71 -- lots of stuff happening before the match started, but clean for the match.
Qual 79 -- pretty clean, but some packet loss a third through tele, no disable.
Elim 4, 8, 12, 15, 16

Elim 10 -- Comms are very clean, but robot drops at 2:24:49.250. It came back for just a few tele packets about 11 seconds later, dropped again a few times over the next 5.5 seconds, and then continued with very clean comms. At 2:24:59, there is a message that says no code is running, and at 2:25:50, there are joystick output errors that occur in most of your log files when your code starts up. Possibly in Begin. So it looks kinda like the code restarted, but I can't say why it happened, and I can't say for sure that this even happens in Begin. I also see the FMS times out and disappears about the same time as the robot. So I tend to think the ethernet cable came loose from the laptop and that was the main issue. It would be good to know if the FTA or students think that was what happened. As for the error regarding joystick outputs, it would be good to know when the code does this to know if this is really Begin or somewhere else.


Elim 14 -- clean for the match, but noisy after the match

I wouldn't worry about the radio or radio placement. I'd check the laptop ethernet retention and speak with the drive team to see whether they think the cable could have been the issue. I'd look at the code a bit for the joystick output error, and perhaps even pull the cable to see if the error shows up in the log when the DS and robot reconnect.

Hope this helps.
Greg McKaskle

[plnyyanks]

Greg hit all the important things in there, and I agree the next step is getting some input .

Next time you get some hands on time with your robot, go check the PDP fuses. Nothing I've seen indicates its an issue, but if not pushed in all the way they slowly wiggle out over time and can cause some strange issues (not always full reboots, either). Here's a post showing what a completely pushed in fuse looks like.

[tr6scott]

The WAIT_FOR_PID is a command we use in our beescript autonomous. When auto starts, we do an elevator reset, which homes the elevator, and resets the encoder, then puts the elevator in pid control. This command just waits until the elevator is under the pid control, before sending it a target height. Obviously, they have a typo in the script or the command name and they don't match up. Not an issue for this event, as we were just sitting and not moving in auto, but something on the list to fix.

The joystick error has been happening all season, but has not been an issue, except for throwing the error. The code runs in periodic tasks, and is setting the joystick rumble for the driver and operator. The operators joystick will rumble if he hits an overtravel on the elevator, and the drivers joystick will rumble when the there is a tote in position to stack.

Greg, if you want to send me you github username I will add you to the repository, if you want to check out the joystick error.

I will try to get additional information from the drivers at tonight's meeting.

We also checked all of the fuses after this match, and nothing was found to be loose.

[Alan Anderson]

Originally Posted by tr6scott

We also checked all of the fuses after this match, and nothing was found to be loose.

The way you worded that makes me wonder if you are talking about the right thing. The 10- and 20-amp PDP fuses are the important ones here. They're the red and yellow miniature automotive fuses on either side of the Weidmuller terminals for the roboRIO, PCM, and VRM power. If one of them has an intermittent contact for a moment, the roboRIO loses power. If the other one fails briefly, the VRM (and thus the D-Link) loses power.

Even if not fully seated, they usually won't feel loose. But not feeling loose doesn't mean they are making proper contact. If you can pull them out by hand, they weren't in all the way. It takes some serious pushing to get them in correctly, and once that's done, they definitely won't come out on their own.

[tr6scott]

Alan,

Yes, those fuses were included in the "All fuses" were checked. We experienced a disconnect at our Waterford event (Week2), and when we were alerted of the problem, we found one of the ATM fuses to not be completely seated. Having already been through this scenario once this season, that was the first thing I had instructed the pit crew to confirm as they were fixing the elevator after the match.

But I am now going to go back to those logs and review that match too.

[notmattlythgoe]

We actually had a match at VA that our robot code just stopped running. FTA said the rio never disconnected from the field. No errors on the DS stating any code errors that crashed it. It just stopped. Never figured out what the issue was and it didn't do it again.

[tr6scott]

Alan,

Follow up on Waterford event.
Video: https://youtu.be/Ai4AiWz7680

at 57 second on video you see the stacklight flash.

You also see our drivers struggle with that stack after restored.

FTA came to us with the issue, and mentor said one of the ATM fuses was not seated completely, he felt the fuse move, and seat when he pushed on it.

As this was Finals Match 2, not much done after this to diagnose the issue.

[Monochron]

We had a very similar situation in our quarter finals match. The first one we exhibited similar behavior all throughout teleop (autonomous went fine). The second match we only dropped out for about 30 seconds or so. The DS logs showed that we lost communication with the FMS for the duration of those times. CSA's suggested that this was caused by our radio losing power, but it looked like they couldn't investigate the issue further than that.

I'm eager to see if you can resolve this, and I'll see if I can make any progress on evidence for what caused our problem.

[plnyyanks]

Originally Posted by Monochron

We had a very similar situation in our quarter finals match. The first one we exhibited similar behavior all throughout teleop (autonomous went fine). The second match we only dropped out for about 30 seconds or so. The DS logs showed that we lost communication with the FMS for the duration of those times. CSA's suggested that this was caused by our radio losing power, but it looked like they couldn't investigate the issue further than that.

It sounds like the CSA was on to something - a radio reboot typically takes between 30 and 40 seconds to come back (the DS logs tell you exactly how long your robot was down). In contrast, a roboRIO will take around 25 seconds to reboot.

Common causes of radio reboots:
- Brownouts due to your battery being low. These events correlate to large current draws
- PDP Fuses. Push them in all the way
- Multiple things plugged into the radio source on the VRM
- The barrel connector not being plugging in all the way, especially if it's an older one

[Monochron]

Originally Posted by plnyyanks

It sounds like the CSA was on to something - a radio reboot typically takes between 30 and 40 seconds to come back (the DS logs tell you exactly how long your robot was down). In contrast, a roboRIO will take around 25 seconds to reboot.

Common symptoms of radio reboots:
- Brownouts due to your battery being low. These events correlate to large current draws
- PDP Fuses. Push them in all the way
- Multiple things plugged into the radio source on the VRM
- The barrel connector not being plugging in all the way, especially if it's an older one

Yeah we didn't notice any unusually large voltage spikes in the logs but I'm still checking different possible causes.

[MrRoboSteve]

Also make sure you are using the appropriately sized barrel connector. If the outer diameter is too small, intermittent power is possible.

[billbo911]

Originally Posted by Greg McKaskle

....

I wouldn't worry about the radio or radio placement. I'd check the laptop ethernet retention and speak with the drive team to see whether they think the cable could have been the issue. I'd look at the code a bit for the joystick output error, and perhaps even pull the cable to see if the error shows up in the log when the DS and robot reconnect.

Hope this helps.
Greg McKaskle

Greg has a very strong point here. RJ45 connectors, while normally reliable, can become fairly unreliable. Many factors can contribute to this issue. This is a big issue on my team. We have managed to damage beyond repair a few laptop's Ethernet ports.
The repeated connecting and disconnecting that takes place with most Driver's Station laptops can easily contribute to their early demise. There are a couple things that can be done to help reduce this wear and tear. One of the simplest and easiest ways to do this is to use an Ethernet Port Saver. Install it and tie it down so the end attached to the laptop doesn't move WRT the laptop.

Another issue I have personally seen giving lots of teams grief, and may apply to this thread directly, has to do with the design of the Ethernet port on the laptop used for the Driver's Station. This design can easily cause intermittent connection failures as well as the cable completely coming out of the laptop.
Here are a couple pictures of my personal laptop that has been used as a Driver's Station and has also completely lost the cable when bumped just a slight bit. This is an Acer laptop, but I have seen this same configuration on several different manufacturer's laptops.




In the second picture I am pulling down on the small cover plate that opens up when the cable is inserted. This cover rests right on the latch clip for the cable. Any pressure on the cover causes the cable latch to release. Believe me, it happens A LOT!!

The resolution to this is to make a relief notch in the Driver's Console assembly so that no contact with the cover plate can be made unintentionally.

[Tom Line]

Originally Posted by billbo911

Greg has a very strong point here. RJ45 connectors, while normally reliable, can become fairly unreliable. Many factors can contribute to this issue. This is a big issue on my team. We have managed to damage beyond repair a few laptop's Ethernet ports.
The repeated connecting and disconnecting that takes place with most Driver's Station laptops can easily contribute to their early demise. There are a couple things that can be done to help reduce this wear and tear. One of the simplest and easiest ways to do this is to use an Ethernet Port Saver. Install it and tie it down so the end attached to the laptop doesn't move WRT the laptop.

Another issue I have personally seen giving lots of teams grief, and may apply to this thread directly, has to do with the design of the Ethernet port on the laptop used for the Driver's Station. This design can easily cause intermittent connection failures as well as the cable completely coming out of the laptop.
Here are a couple pictures of my personal laptop that has been used as a Driver's Station and has also completely lost the cable when bumped just a slight bit. This is an Acer laptop, but I have seen this same configuration on several different manufacturer's laptops.




In the second picture I am pulling down on the small cover plate that opens up when the cable is inserted. This cover rests right on the latch clip for the cable. Any pressure on the cover causes the cable latch to release. Believe me, it happens A LOT!!

The resolution to this is to make a relief notch in the Driver's Console assembly so that no contact with the cover plate can be made unintentionally.

I have an acer laptop as well, and I'm convinced whoever designed that ethernet port was paid to find the most contrived, poorly engineered way possible to design it. Of course, I didn't realize it until FIRST season started, because how many other times do you plug your laptop into ethernet except at competitions?

Bleah.