ARENA Fault

[Alan Anderson]

Originally Posted by kylelanman

A common response that I have heard is "After reviewing the log of the entire match nothing seems out the ordinary so the match will not be replayed". How is 1 to 4 robots loosing coms not something wrong? Something is clearly wrong. Thousands of fans can tell with out looking at a log that something went wrong.

Something obviously went wrong. It is not obvious that what went wrong has anything to do with the ARENA. If it was really an issue of losing communication, the FMS logs would confirm that to be the case.

More than one team has replaced their WGA and seen their troubles go away. That's not an ARENA fault. More than one team has confirmed problems with reading joysticks and/or the Cypress board after letting their Classmate go to sleep or when running low on battery power. That's not an ARENA fault either.

Team 45 lost a qualification match when the robot appeared to disable itself midway through autonomous, showed that it was re-enabled at teleop but with the drive motors nonresponsive, and sat there not moving for the rest of the match. Obviously a communication failure, right? Wrong. The diagnostic lights were green the whole time. The real fault was the battery we used for that match, which decided to die sometime between having been charged and the start of the match. (In that case, the FMS did pick up on the low voltage issue, and so did the drive team who knew what to look for on the Driver Station.)

[Vikesrock]

Originally Posted by Alan Anderson

The diagnostic lights were green the whole time.
...
(In that case, the FMS did pick up on the low voltage issue, and so did the drive team who knew what to look for on the Driver Station.)

I've pulled two parts out of Alan's response here to highlight something that I have found extremely helpful in diagnosing our robot issues in the past.

The Driver's Station and Dashboard provide a large amount of feedback about what is going on with your robot.

At our event last year our robot stopped moving in three separate matches. By observing our Dashboard during these faults I was well aware that our DS was still connected to the robot and in 2 of the 3 matches to the cRIO. After thorough checks of the electrical system a few loose wires/chassis shorts/other issues were found and the problems never returned.

Knowing what information is provided on the Driver Station and Dashboard and how to use it to diagnose your robot is the best way to help track down these mysterious faults.

[Dave Flowerday]

Originally Posted by Alan Anderson

If it was really an issue of losing communication, the FMS logs would confirm that to be the case.

I keep seeing this idea presented in all these threads about suspected field faults. I disagree with it, and I'm surprised more people don't feel the same way.

1. Most of us have no idea what the FMS even logs in the first place, and therefore what kind of failures could be detected with these logs. For all I know, the log may consist of a timer that fires every 100ms that just writes "Everything is fine" into the logfile. I'm sure that the logs contain lots of data, and maybe it's enough that it really can rule out anything but a team problem, but how do we know there isn't a bug in the logging mechanism? Or some other gremlin? Surely anyone who's been involved with software for more than a few weeks can admit that bugs can end up anywhere and can easily trick you into thinking something is working when it isn't.

2. Somewhat of a rehashing of #1, can the field system confirm that packets from the OI are getting to the robot? Or does it just verify that it can "see" the robot and OI from it's vantage point? Is it possible that FMS can be successfully connected to the team's OI and successfully pinging (or whatever) the robot, but that the OI and the robot aren't talking to each other?

3. Dozens of teams are reporting problems here. Ultimately, it may not be a problem with the field hardware, but it seems pretty clear that there are defects with the system as a whole (the system being everything from the Classmate and Rio to the arena controls, the WiFI adapters, joysticks, etc). Who's responsible for the overall operation of the entire controls package? Who's looking into these reports of issues? Surely all the technical folks at FIRST aren't just writing this off as a team issue, are they? Even if they believe it is the fault of the teams, I would hope that they would want to get to the bottom of it anyway, just to be sure. With so many different vendors involved in the system, though, I'm not sure who takes ultimate ownership of a potential system-level issue.

I've been involved in enough FIRST competitions to know that when the people working at the event say, "there were no errors with the field", what they usually really mean is, "We couldn't find any specific problem, and therefore we're going to blame you (the team) so we don't have to replay the match. If we admit that we aren't 100% sure what happened, we'll be here for weeks as we replay almost every single match." I've worked at plenty of events and I've been part of these conversations several times.

[kylelanman]

Originally Posted by Dave Flowerday

I've been involved in enough FIRST competitions to know that when the people working at the event say, "there were no errors with the field", what they usually really mean is, "We couldn't find any specific problem, and therefore we're going to blame you (the team) so we don't have to replay the match. If we admit that we aren't 100% sure what happened, we'll be here for weeks as we replay almost every single match." I've worked at plenty of events and I've been part of these conversations several times.

That is my teams exact opinion in the matter.

[Mark McLeod]

Certainly the control/field system interaction has grown complex, and there are far too many places and ways gremlins can interrupt normal robot operation. It's not like the IFI days when an expert on the full system was delivered with each field. Even in the IFI days, although they look rosey two years later, not every robot death on the field could be or was explained.

Nowadays, without a single source supplier, it's mostly luck if a Regional ends up with a volunteer or two with the necessary skills and experience to chase down and troubleshoot team issues with the field. With technical volunteers scarce in any case, to find one free to roam between the field and the pits to follow up with team troubleshooting is indeed rare.

I worked the NJ Regional running between the field and the teams to track down obscure problems that teams were having and do the same thing at the Long Island Regional. Part of the reason teams tend to blame the field when their robot stops is the black box nature of what FMS supplies. It's common for teams to blame the field, because one of their motors didn't work (but it worked in the pit!), or if their robot takes off on it's own.

Almost without fail teams who blamed the field did not look at their own multitude of status lights and couldn't give me any useful information. They couldn't even tell me what the RSL was doing when their robot failed. I had to be there on the field to analyse the problem and provide a correct diagnosis (correct diagnosis being one that finds a problem that would cause the symptoms).

The art of troubleshooting is something we mentors need to teach more of, but I've come to realize that many mentors don't have good troubleshooting skills either.

[Dave Flowerday]

Originally Posted by Mark McLeod

It's common for teams to blame the field, because one of their motors didn't work (but it worked in the pit!), or if their robot takes off on it's own.

I agree, and I will add that the opposite is also true: it's common for event staff and volunteers to blame the teams' equipment, because the field worked fine in the previous match. I think both parties assume their side is correct and therefore the other must be broken. The reality is there have been enough root-caused issues with both robots and the field that both should be suspected when an issue occurs. There's plenty of threads in the scorekeeper's forum on usfirst.org that talk about having FMS crash multiple times and having to "reboot the field", and obviously there's been enough issues found with wiring etc. on robots over the years too.

[Mark McLeod]

Originally Posted by Dave Flowerday

The reality is there have been enough root-caused issues with both robots and the field that both should be suspected when an issue occurs.

Any good troubleshooter will suspect everything. I witnessed some of those FMS reset issues, so I know it's not faultless.

The problem is of course that there are far too few troubleshooters who straddle the fence and can unemotionally examine both sides, and aren't consumed by official duties. I've always been able to find a problem with the robot that solved the issue (last year and this year), as long as I'm there to examine the status lights and diagnostic messages. Even with wireless bridge failures the statistics page on the bridge usually tells that story, as long as I can prevent the team from turning their robot off after failing in a match. That doesn't mean I don't list FMS as a potential culprit every time. In fact that's always the first consideration, because it needs to be checked immediately while the problem is occurring. FMS failures tend to take the whole field down, or one entire end, or a single driver station eStop might fail, but it's pretty obvious to the field crew when that happens.

You can't actually solve a problem unless you can divorce yourself from taking sides. If you find yourself blaming something you know nothing about, then that's not troubleshooting and it won't solve the problem.

One of the issues I've noticed this year and last is that teams look on the NI reps as experts in the whole system, whereas, they typically are not. They are all experts in LabVIEW, many in cRIO operation, some work with teams and know the whole FRC control scheme, but few of them know anything about FMS.

[Bruceb]

706 was with 2481 in the finals at Milwaukee. We had the camera loss at the same time as they lost all control. It was imediatly after the auto mode ended and there had been no robot to robot interactions and no collisions with the field. After the match we logged out of driver mode and back in and regained the camera. Sure was hard trying to win the match with 2481 down again and a sub playing defense. Nothing against the sub, they did a great job and kept the score down to 7-3 but with noone feeding us balls from middle we were pretty helpless. As coach I am hitting myself now for not telling our third match defence partner(sorry I don't remember the team number) to abandon defence and feed us some balls. Grrr Hind sight.
Anyway, I have to ask why is it that we are using a control system that is so complex and error prone? I can go to a model airplane field and have 15 planes in the air at the same time and none of them will have com problems.
Are there some things that every team should do between matches other than look for hardware issues like rebbot the classmate or log out and back in? I know that some computer systems need to be restarted regularily to keep from getting flaky errors. Also, having power at the drivers stations could be helpfull here. Is it legal to bring a battery and an inverter to the drivers station?
Bruce

[Vikesrock]

Originally Posted by Bruceb

Is it legal to bring a battery and an inverter to the drivers station?
Bruce

Yes.

http://forums.usfirst.org/showthread.php?t=15017

[Mark McLeod]

Originally Posted by Dave Flowerday

I will add that the opposite is also true: it's common for event staff and volunteers to blame the teams' equipment, because the field worked fine in the previous match.

I agree too.

That's the first thing I put a stop to when I was asked to Tech the SBPLI field last year-empty speculation by field staff. If the field staff didn't KNOW what the problem was, they were to refer it to me and not start guessing. It's actually harmful in that it usually focuses troubleshooting in exactly the wrong place. I worked the WPA desk one regional where if they had any field connection problems they sent it back to us to be reset, but I never found even one that was set incorrectly. Of course, the robot that caught on fire was plenty obvious...

It worked out well for Long Island last year, but I still had an IT professional telling me it was a field fault that shutdown only his team's drive motors. His explanation was that it couldn't be his code, and to be fair it actually wasn't, but it also wasn't a field failure. It could have been prevented by adding an error check in his code.