Re: ARENA Fault
 

Everyone needs to remember that with every year that FIRST changes something with the control system, there will be a huge learning curve for all to understand how it all works. Also, it takes time to really fine tune the system, yeh NI and FIRST may be able to debug and get the system working to the best it can, but the true test really won't come until the Regionals when we bombard the system with all we got.

I know that in DC there were many teams that were having comm issues, but I assure you (99.9% at least) that these problems had nothing to do with the FMS faulting. Many teams were forgetting to plug their raido's back into the e-port on the cRio, or were forgetting to put a fresh battery in their robot, or hadn't restarted the DS program since their last match to name some of the major issues. We had one team that barely even moved the whole competition because they were having some sort of comm issue after another, but they were gracious and went back to the pit and worked hard to figure out solutions. We had a NI rep at the regional and he worked very close with that team to help them through these problems. It is issues and interactions like that that will help everyone learn more about the system and where it's weaknesses are and how we can improve on them in the future.

I know that it gets frustrating to be out there and all of a sudden not be able to compete, but remember your gracious professionalism and trust that the technical people behind the scenes are working hard to solve any and all problems and will make sure that there is a fair solution that comes out of it.

Re: ARENA Fault
 

Any good troubleshooter will suspect everything. I witnessed some of those FMS reset issues, so I know it's not faultless.

The problem is of course that there are far too few troubleshooters who straddle the fence and can unemotionally examine both sides, and aren't consumed by official duties. I've always been able to find a problem with the robot that solved the issue (last year and this year), as long as I'm there to examine the status lights and diagnostic messages. Even with wireless bridge failures the statistics page on the bridge usually tells that story, as long as I can prevent the team from turning their robot off after failing in a match. That doesn't mean I don't list FMS as a potential culprit every time. In fact that's always the first consideration, because it needs to be checked immediately while the problem is occurring. FMS failures tend to take the whole field down, or one entire end, or a single driver station eStop might fail, but it's pretty obvious to the field crew when that happens.

You can't actually solve a problem unless you can divorce yourself from taking sides. If you find yourself blaming something you know nothing about, then that's not troubleshooting and it won't solve the problem.

One of the issues I've noticed this year and last is that teams look on the NI reps as experts in the whole system, whereas, they typically are not. They are all experts in LabVIEW, many in cRIO operation, some work with teams and know the whole FRC control scheme, but few of them know anything about FMS.

Re: ARENA Fault
 

706 was with 2481 in the finals at Milwaukee. We had the camera loss at the same time as they lost all control. It was imediatly after the auto mode ended and there had been no robot to robot interactions and no collisions with the field. After the match we logged out of driver mode and back in and regained the camera. Sure was hard trying to win the match with 2481 down again and a sub playing defense. Nothing against the sub, they did a great job and kept the score down to 7-3 but with noone feeding us balls from middle we were pretty helpless. As coach I am hitting myself now for not telling our third match defence partner(sorry I don't remember the team number) to abandon defence and feed us some balls. Grrr Hind sight.
Anyway, I have to ask why is it that we are using a control system that is so complex and error prone? I can go to a model airplane field and have 15 planes in the air at the same time and none of them will have com problems.
Are there some things that every team should do between matches other than look for hardware issues like rebbot the classmate or log out and back in? I know that some computer systems need to be restarted regularily to keep from getting flaky errors. Also, having power at the drivers stations could be helpfull here. Is it legal to bring a battery and an inverter to the drivers station?
Bruce

Re: ARENA Fault
 

Yes.

http://forums.usfirst.org/showthread.php?t=15017

Re: ARENA Fault
 

I agree too.

That's the first thing I put a stop to when I was asked to Tech the SBPLI field last year-empty speculation by field staff. If the field staff didn't KNOW what the problem was, they were to refer it to me and not start guessing. It's actually harmful in that it usually focuses troubleshooting in exactly the wrong place. I worked the WPA desk one regional where if they had any field connection problems they sent it back to us to be reset, but I never found even one that was set incorrectly. Of course, the robot that caught on fire was plenty obvious...

It worked out well for Long Island last year, but I still had an IT professional telling me it was a field fault that shutdown only his team's drive motors. His explanation was that it couldn't be his code, and to be fair it actually wasn't, but it also wasn't a field failure. It could have been prevented by adding an error check in his code.

Re: ARENA Fault
 

While our robot never lost communication with the field, I did notice some problems:
1) We took FOREVER to establish the initial communication with our robot sometimes. While we were waited on to establish communication before starting the match, it was very frustrating to have to cycle power on the bot and cause delays for everybody.
2) Sometimes in the middle of the match I noticed some lag on the connection between the robot and my joysticks. It is pretty intense during a match so I am not sure if the robot was losing comm for a second or two every now and then or if it was just a bad wifi connection.

Re: ARENA Fault
 

I've found laggy controls to be a robot problem, not field. Most common for me have been the camera and CAN bus. No camera comms = error messages every 5 ms = no CPU for robot operation. Same applies to CAN. Who knows what else could do it

Re: ARENA Fault
 

We also had the problem of the robot losing connection with the driver's station before the match started. They let us stay out and reboot for the first 5 matches but for 6 and 7 they made us leave the field before we had time to reboot so the robot was dead. We showed the field people that if we waited until the team number was changed from the previous match our robot worked fine. We tried that for match 8 and it didn't work when the two alliance partners were on the field. For match 9 we waited until the other two alliance partners were fully connected and that worked. Were were able to run this way all the way to through the finals. We still don't know what the problem is but we'll try to solve it at 10,000 Lakes. We're a bit concerned that if we don't get it solved we won't be able to run in Atlanta.

Re: ARENA Fault
 

1. Everyone deals with field faults at times. Dean and Woody like to point out that this isn't a "fair" game. You're playing against better funded, better engineered, better mentored teams and there isn't much you can do about it. It sucks, but it's part of the game. I know it sounds glib, but these are facts we end up having to deal with.

2. FIRST does the best they can. They have some top notch folks working on these issues. We were lucky - we got to work with some of them during beta. Their turn-around time on issues that are found is spectacular - in many instances they are issuing fixes in as little as a day. But in the end they are a limited resource working with limited time and limited money. The result is an imperfect product. (See Toyota's accelerator pedals for how that works).

3. A great majority of the issues that are occuring /right now/ on the field are being caused by the teams. In nearly every match we played during our first regional, we had to wait for one and sometimes two teams that hadn't bothered to charge their laptop, or make sure it was up and running before the match. None had paid any attention to the multiple warnings during the prior week. Shame on them.

I know - none of these will make you feel any better. We've been in your shoes: last year our driverstation fried (static) in the elims during a week 1 regional and they gave us absolutely no special dispensation to fix it. We spent multiple elim matches with a dead / half dead robot. It wasn't our fault. It stunk. In the end we cursed those little blue bricks, the regolith, and the officials (who were simply following the rules) before moving on and getting over it.

First will do the best they can, and so will we. The only thing I can say is to try to stay gracious and professional, and teach your kids on your team how to do the same in the face of an unfair result. Continue to try to get into the beta tests, present papers at nationals, and help local teams to improve the end product so that we can all have fewer issues.

Re: ARENA Fault
 

I mentor a team that has had similar issues, and I volunteer as a scorekeeper, so I have a foot in both graves so to speak.

Most of the information we have available to us from the FMS display is the same information you have showing on your classmate. We do have the luxury of seeing it for all six robots at once. Much of what is written into the logs is the info each team's classmate reports to the FMS. So at the moment your classmate looses contact with your robot, that would be logged. Of course, the FMS doesn't know why it happened.

When I'm doing the scorekeeper job, I'm constantly monitoring both the action on the field and the FMS display. When I see that a robot's battery voltage is unusually low, I alert the FTS by radio to alert the team. They might have noticed it on their classmate display, but also might not. Likewise for missed packets and drops in robot link. Dpending on whether experience leads us to think it is most likely centered around the robot or the driver station, I'll ask the FTA to investigate while the match is still being played. I make my own written notes and cross check that with event history up to that point, to look for issues that follow teams or driver stations.

During the events, scorekeepers and FTA's have comm links back to FIRST HQ when things seem to be out of whack. In my opinion, you are making a mistake if you think that these people do not care deeply and take seriously every issue and possible bug that comes up. But FIRST is not an immense army; It would surprise most of you how few paid staffers actually make FIRST happen. Give them some credit for working many long hours. Same witht he event volunteers. I've been a part of many reviews and discussions long after the pits close about what transpired during the day. It's not uncommon for us to make a decision to reverse a DQ/red card/yellow card or replay a match the next day if we feel it is the fair thing to do.

There seems to be an assumption that the field is more complex the last two years than it was for IFI run events. I'm not sure that's true. One example is the radios. Most don't remember that IFI placed a tree of individual radios to facilitate robot communication. It was possible that one of them could fail (or the individual wires that led to them), leading to one robot losing communication while the others ran fine. Today's field has a single radio in the form of a Cisco access point which communicates with all robots simultaneously. A failure for one is likely a failure for all and thus a replayed match.

Each classmate is connected by ethernet cable to a switch in the station cabinet in each alliance station. This in turn is wired to the access point which communicates to the robots. There aren't a lot of points of failure here that could affect one team and not an entire alliance or all robots. Still, there are some and we watch those both during the match and from match to match. If I saw evidence of an intermittent error at one driver station, I'd alert the FTA and head ref and recommend replaying some matches if I thought there was any sort of field problem. A replay does not have to be done immediately.

Electrical problems can be much harder to sort out than mechanical ones. We do not replay matches when your chain breaks or your motor burns up or you forget to plug something back in. We do not have the luxury of replaying matches until the outcome is unaffected by robot design and construction issues. We also cannot do so when your programming fails or a breaker trips or something shorts out temporarily. But for some reason, maybe because those failures are sometimes not obvious (or even leave no evidence), a claim of field fault almost inevitably follows.

I share your frustration when you don't know how to fix your problems, but sometimes we don't get the information back to help you. Once the teams leave the field and begin the troubleshooting process, often they find the problem was a simple one of their own making, but don't let us know. If I did get that information, I could pass it on to other teams as best practices to keep their robots in top form. Unfortunately I usually only get feedback from immediately after the match as the FTA makes an immediate diagnosis. That said, reread this and similar threads where you'll find the variety of ways a robot can fail during an active FIRST match and try to avoid these things happening to you.

Re: ARENA Fault
 

@rspurlin

Perhaps you could provide me with a little more insight. What you just provided was great.

The light on the alliance station, what does this indicate? Our RSL light was indicating normal operation while our alliance light was flashing. A few seconds later the RSL started flashing too. Our assumption is that the classmate lost comms first and then when the robot noticed the classmate was gone it disabled itself and the RSL updated to indicate it was disabled.

After this happened logging out of the Driver profile, into Developer, running arp -d, logging out of developer, and back into driver gave us comms for a few seconds. I'm hoping you can answer this question. Does the system allow a team to logout in the middle of a match if everything is connected correctly? I was not on the field but was told that they tried this in an earlier match and it did not let them logout.

I have 1 final question as to how the FMS is setup. I kinda assumed from day one that the FMS controls the robot. When allowed the classmates send signals to the FMS that get forwarded to the robot. Or is the FMS setup so that the classmates control the robots directly and the FMS send signals to the classmate to change the mode (teleop, auto, disabled)? I understand physically how the network is setup I asking from a logical stand point.

Here is scenario 1.
Classmate ---- FMS ---- AP ---- Robot
FMS ---- Classmate ---- AP ---- Robot



A bad cable between the driver station and the switch or the alliance switch and the main switch is my theory. We only had comm problems on the red alliance. I have come across plenty of network cables that provide partial connectivity.

Re: ARENA Fault
 

Alliance station status indicators:

Green--E-stop.
Flashing red or blue--no communication, hunting for communication, that sort of thing.
Solid red or blue--match running, communication.
All off--match waiting to start, communication. Either that or the entire system lost power...

Re: ARENA Fault
 

I'm aware of that I more specifically interested in how communication or lack there of is determined?
No communication with the Robot or
No communication with the Classmate or
Classmate can't communicate with robot?

Re: ARENA Fault
 

I don't know how they check robot comms, other than the RSL. Classmate-FMS is viewed in the Dashboard. I'd guess you can check the dashboard (some of the indicators would tell you).

Re: ARENA Fault
 

Eric and Kyle,
I believe I was the inspector. The team (and many other teams) brought me an adapter that they couldn't ping. Neither could I, even after two manufacturer resets. I could not connect, I could not see the web interface although it appeared to be handshaking to the computer. A power reset at one point, sent the device into never never land and it never completed. The team took it from me when it would not boot. I merely confirmed what they already thought.
In the final match in Wisconsin, three robots failed. I was informed that two of them lost the Classmate due to dead batteries and the third was a shutdown of the USB hub powered from the Classmate, also a battery issue.
I wish I had a nickel for every match I have missed because I was helping a team. I could buy a nice lunch.
I believe the alliance station also flashes during autonomous if a robot is not running autonomous code.