ARENA Fault

[kylelanman]

ARENA as it is defined in the rules.

The ARENA includes all elements of the game infrastructure that are required to play Breakaway: the FIELD, the ALLIANCE STATIONS, the GOALS, the BALLS, and all supporting communications, arena control, and scorekeeping equipment.

The only rule I see regarding ARENA faults.

<T18> If, in the judgment of the Head Referee, an “ARENA fault” occurs that affects either the play or the outcome of the MATCH, the MATCH will be replayed. Example ARENA faults include broken field elements, power failure to a portion of the field, improper activation of the field control system, errors by field personnel, etc.

A common response that I have heard is "After reviewing the log of the entire match nothing seems out the ordinary so the match will not be replayed". How is 1 to 4 robots loosing coms not something wrong? Something is clearly wrong. Thousands of fans can tell with out looking at a log that something went wrong.

Outside of FIRST when someone has an IT related problem and it can not be identified from the log we don't tell the person(s) they are out of luck and we aren't going to help them. The IT department looks into the problem and rectifies the situation as soon as possible.

The least they could do is have a professional on site at each regional to inspect the robot after the match back in the pit and help figure out what is wrong. How can a bunch of high-schoolers be expected to find the problem when the inspectors at the event can't. From the looks and sounds of it the only reason that help is not offered is that no one actually knows what causes all the comm problems.

If FIRST can not help us troubleshoot the mysterious comm problems that never happen at the shop but randomly happen on the field then teams should be given the benefit of the doubt and the match should be replayed when they occurr.

[EricH]

If the field is communicating correctly, then the problem lies with either the teams or Murphy. Quite often, robots lose communication if a cable gets knocked loose--always check your radio-cRIO cable before and after the match. This is quite difficult to pinpoint using field equipment--everything will look normal, but the robot won't do a thing.

I know that at Arizona, we did have two experts (not necessarily professionals) who could find errors. One designed the FMS system and was judging the event; the other was a beta tester (2 years) who got a volunteer pass so that he wouldn't keep getting sent out of the field area because the FTA was always asking him to look at something. Sometimes, even they couldn't figure out why the comm link was lost.

As for replaying such matches when they occur, there is a really really good chance that that will cause the event to run very much over the time allowed. Not just a couple of minutes, either. We're talking running over the way the Championship always does--a couple of hours.

[kylelanman]

Originally Posted by EricH

I know that at Arizona, we did have two experts (not necessarily professionals) who could find errors. One designed the FMS system and was judging the event; the other was a beta tester (2 years) who got a volunteer pass so that he wouldn't keep getting sent out of the field area because the FTA was always asking him to look at something. Sometimes, even they couldn't figure out why the comm link was lost.

If the best of the best can't figure it out then there is clearly something wrong here. FIRST should not provide us with a control system that is so advance that no one is able to troubleshoot. Then tell us we're out of luck when a team that has a 15-0 record mysteriously looses comms in the Finals and they loose it all with no explanation at all.

Originally Posted by Alan Anderson

Something obviously went wrong. It is not obvious that what went wrong has anything to do with the ARENA. If it was really an issue of losing communication, the FMS logs would confirm that to be the case.

I'd have to argue that when one robot loses comms and another video feed at the exact same time that there something wrong between the alliance station and the robot.....The only thing that falls in to that category that is between the alliance station and the robot that I am aware of is the Arena/FMS.

Originally Posted by Alan Anderson

More than one team has replaced their WGA and seen their troubles go away. That's not an ARENA fault. More than one team has confirmed problems with reading joysticks and/or the Cypress board after letting their Classmate go to sleep or when running low on battery power. That's not an ARENA fault either.

We tried to do that. We had the newer bridge and were trying to configure it during matches. We asked a question regarding the WPA or WPA2 setting to the WPA inspector. He did not answer the question instead took the radio set it to factory settings plugged it in to his computer and tried to use his script/program to configure it. He got confused as to why it didn't work and wanted to go watch his own team compete. When we tried to call a timeout to configure it and swap radios the head ref told us no.

We disabled the classmate from going to sleep early in the season. During the final matches used another team's classmate battery because ours was dead.

[EricH]

Originally Posted by kylelanman

If the best of the best can't figure it out then there is clearly something wrong here. FIRST should not provide us with a control system that is so advance that no one is able to troubleshoot.

Most of the time, the best of the best did figure it out. The one they couldn't figure out was why a particular combination of new/old adapters would trigger a field-wide comm loss for no apparent reason. No other combination did that. Try troubleshooting that one...

Originally Posted by kylelanman

We tried to do that. We had the newer bridge and were trying to configure it during matches. We asked a question regarding the WPA or WPA2 setting to the WPA inspector. He did not answer the question instead took the radio set it to factory settings plugged it in to his computer and tried to use his script/program to configure it. He got confused as to why it didn't work and wanted to go watch his own team compete. When we tried to call a timeout to configure it and swap radios the head ref told us no.

The WPA inspector apparently didn't read the update that informed everyone of the program. You can't use it on the new bridge--it has to be done manually. His wanting to go see his own team compete, however--while that's understandable, there's a time and place to do it, like when there's not a team that needs your help right now to be able to compete.

As for the timeout, there are specific rules regarding when you can/can't use one. Quals, you're out of luck. Elims, there's a specific window to call it--if you don't call it in that window, you're out of luck.

Regarding the loss of comms and the loss of video feed at the same time on two different teams: It may be that there were separate failures that just happened to happen at the same time--say, alliance partners hit each other by accident and both cables come loose simultaneously. That's not a field issue, it's a robot issue.

[Mark McLeod]

We had the entire Blue side dead for Teleop in an early Quals match.
I'm sure everyone in the stands "knows" it was a field fault.

Of course one robot had a 5v battery, another had a 0 volt Classmate, and the third was pinned under a partner and didn't want to rip it's camera off trying to get out from under.

[Dave Flowerday]

Originally Posted by Mark McLeod

Of course one robot had a 5v battery, another had a 0 volt Classmate, and the third was pinned under a partner and didn't want to rip it's camera off trying to get out from under.

It seems that Classmate battery issues are definitely a contributing factor to the feeling that the field doesn't work very well. I wish someone could explain to me why FIRST is torturing themselves by not providing a $5 power strip at each alliance station and in turn suffering from the massive number of reported issues that are related to Classmate batteries. It seems like that rule isn't doing anyone any good and is just making all the perceived problems appear worse than they are. I guess I don't have much sympathy for FIRST if they're getting blamed for field faults when the issue is really a low battery when they could so easily solve that particular problem.

[Mike o.]

Everyone needs to remember that with every year that FIRST changes something with the control system, there will be a huge learning curve for all to understand how it all works. Also, it takes time to really fine tune the system, yeh NI and FIRST may be able to debug and get the system working to the best it can, but the true test really won't come until the Regionals when we bombard the system with all we got.

I know that in DC there were many teams that were having comm issues, but I assure you (99.9% at least) that these problems had nothing to do with the FMS faulting. Many teams were forgetting to plug their raido's back into the e-port on the cRio, or were forgetting to put a fresh battery in their robot, or hadn't restarted the DS program since their last match to name some of the major issues. We had one team that barely even moved the whole competition because they were having some sort of comm issue after another, but they were gracious and went back to the pit and worked hard to figure out solutions. We had a NI rep at the regional and he worked very close with that team to help them through these problems. It is issues and interactions like that that will help everyone learn more about the system and where it's weaknesses are and how we can improve on them in the future.

I know that it gets frustrating to be out there and all of a sudden not be able to compete, but remember your gracious professionalism and trust that the technical people behind the scenes are working hard to solve any and all problems and will make sure that there is a fair solution that comes out of it.

[pathew100]

Originally Posted by EricH

If the field is communicating correctly, then the problem lies with either the teams or Murphy...

Be careful there. I'm a scorekeeper. But yes, sometimes my "law" does come into play unfortunately.

[Chris86]

While our robot never lost communication with the field, I did notice some problems:
1) We took FOREVER to establish the initial communication with our robot sometimes. While we were waited on to establish communication before starting the match, it was very frustrating to have to cycle power on the bot and cause delays for everybody.
2) Sometimes in the middle of the match I noticed some lag on the connection between the robot and my joysticks. It is pretty intense during a match so I am not sure if the robot was losing comm for a second or two every now and then or if it was just a bad wifi connection.

[Radical Pi]

Originally Posted by Chris86

While our robot never lost communication with the field, I did notice some problems:
1) We took FOREVER to establish the initial communication with our robot sometimes. While we were waited on to establish communication before starting the match, it was very frustrating to have to cycle power on the bot and cause delays for everybody.
2) Sometimes in the middle of the match I noticed some lag on the connection between the robot and my joysticks. It is pretty intense during a match so I am not sure if the robot was losing comm for a second or two every now and then or if it was just a bad wifi connection.

I've found laggy controls to be a robot problem, not field. Most common for me have been the camera and CAN bus. No camera comms = error messages every 5 ms = no CPU for robot operation. Same applies to CAN. Who knows what else could do it

[Tom Ore]

We also had the problem of the robot losing connection with the driver's station before the match started. They let us stay out and reboot for the first 5 matches but for 6 and 7 they made us leave the field before we had time to reboot so the robot was dead. We showed the field people that if we waited until the team number was changed from the previous match our robot worked fine. We tried that for match 8 and it didn't work when the two alliance partners were on the field. For match 9 we waited until the other two alliance partners were fully connected and that worked. Were were able to run this way all the way to through the finals. We still don't know what the problem is but we'll try to solve it at 10,000 Lakes. We're a bit concerned that if we don't get it solved we won't be able to run in Atlanta.

[Tom Line]

1. Everyone deals with field faults at times. Dean and Woody like to point out that this isn't a "fair" game. You're playing against better funded, better engineered, better mentored teams and there isn't much you can do about it. It sucks, but it's part of the game. I know it sounds glib, but these are facts we end up having to deal with.

2. FIRST does the best they can. They have some top notch folks working on these issues. We were lucky - we got to work with some of them during beta. Their turn-around time on issues that are found is spectacular - in many instances they are issuing fixes in as little as a day. But in the end they are a limited resource working with limited time and limited money. The result is an imperfect product. (See Toyota's accelerator pedals for how that works).

3. A great majority of the issues that are occuring /right now/ on the field are being caused by the teams. In nearly every match we played during our first regional, we had to wait for one and sometimes two teams that hadn't bothered to charge their laptop, or make sure it was up and running before the match. None had paid any attention to the multiple warnings during the prior week. Shame on them.

I know - none of these will make you feel any better. We've been in your shoes: last year our driverstation fried (static) in the elims during a week 1 regional and they gave us absolutely no special dispensation to fix it. We spent multiple elim matches with a dead / half dead robot. It wasn't our fault. It stunk. In the end we cursed those little blue bricks, the regolith, and the officials (who were simply following the rules) before moving on and getting over it.

First will do the best they can, and so will we. The only thing I can say is to try to stay gracious and professional, and teach your kids on your team how to do the same in the face of an unfair result. Continue to try to get into the beta tests, present papers at nationals, and help local teams to improve the end product so that we can all have fewer issues.

[rspurlin]

I mentor a team that has had similar issues, and I volunteer as a scorekeeper, so I have a foot in both graves so to speak.

Most of the information we have available to us from the FMS display is the same information you have showing on your classmate. We do have the luxury of seeing it for all six robots at once. Much of what is written into the logs is the info each team's classmate reports to the FMS. So at the moment your classmate looses contact with your robot, that would be logged. Of course, the FMS doesn't know why it happened.

When I'm doing the scorekeeper job, I'm constantly monitoring both the action on the field and the FMS display. When I see that a robot's battery voltage is unusually low, I alert the FTS by radio to alert the team. They might have noticed it on their classmate display, but also might not. Likewise for missed packets and drops in robot link. Dpending on whether experience leads us to think it is most likely centered around the robot or the driver station, I'll ask the FTA to investigate while the match is still being played. I make my own written notes and cross check that with event history up to that point, to look for issues that follow teams or driver stations.

During the events, scorekeepers and FTA's have comm links back to FIRST HQ when things seem to be out of whack. In my opinion, you are making a mistake if you think that these people do not care deeply and take seriously every issue and possible bug that comes up. But FIRST is not an immense army; It would surprise most of you how few paid staffers actually make FIRST happen. Give them some credit for working many long hours. Same witht he event volunteers. I've been a part of many reviews and discussions long after the pits close about what transpired during the day. It's not uncommon for us to make a decision to reverse a DQ/red card/yellow card or replay a match the next day if we feel it is the fair thing to do.

There seems to be an assumption that the field is more complex the last two years than it was for IFI run events. I'm not sure that's true. One example is the radios. Most don't remember that IFI placed a tree of individual radios to facilitate robot communication. It was possible that one of them could fail (or the individual wires that led to them), leading to one robot losing communication while the others ran fine. Today's field has a single radio in the form of a Cisco access point which communicates with all robots simultaneously. A failure for one is likely a failure for all and thus a replayed match.

Each classmate is connected by ethernet cable to a switch in the station cabinet in each alliance station. This in turn is wired to the access point which communicates to the robots. There aren't a lot of points of failure here that could affect one team and not an entire alliance or all robots. Still, there are some and we watch those both during the match and from match to match. If I saw evidence of an intermittent error at one driver station, I'd alert the FTA and head ref and recommend replaying some matches if I thought there was any sort of field problem. A replay does not have to be done immediately.

Electrical problems can be much harder to sort out than mechanical ones. We do not replay matches when your chain breaks or your motor burns up or you forget to plug something back in. We do not have the luxury of replaying matches until the outcome is unaffected by robot design and construction issues. We also cannot do so when your programming fails or a breaker trips or something shorts out temporarily. But for some reason, maybe because those failures are sometimes not obvious (or even leave no evidence), a claim of field fault almost inevitably follows.

I share your frustration when you don't know how to fix your problems, but sometimes we don't get the information back to help you. Once the teams leave the field and begin the troubleshooting process, often they find the problem was a simple one of their own making, but don't let us know. If I did get that information, I could pass it on to other teams as best practices to keep their robots in top form. Unfortunately I usually only get feedback from immediately after the match as the FTA makes an immediate diagnosis. That said, reread this and similar threads where you'll find the variety of ways a robot can fail during an active FIRST match and try to avoid these things happening to you.

[kylelanman]

@rspurlin

Perhaps you could provide me with a little more insight. What you just provided was great.

The light on the alliance station, what does this indicate? Our RSL light was indicating normal operation while our alliance light was flashing. A few seconds later the RSL started flashing too. Our assumption is that the classmate lost comms first and then when the robot noticed the classmate was gone it disabled itself and the RSL updated to indicate it was disabled.

After this happened logging out of the Driver profile, into Developer, running arp -d, logging out of developer, and back into driver gave us comms for a few seconds. I'm hoping you can answer this question. Does the system allow a team to logout in the middle of a match if everything is connected correctly? I was not on the field but was told that they tried this in an earlier match and it did not let them logout.

I have 1 final question as to how the FMS is setup. I kinda assumed from day one that the FMS controls the robot. When allowed the classmates send signals to the FMS that get forwarded to the robot. Or is the FMS setup so that the classmates control the robots directly and the FMS send signals to the classmate to change the mode (teleop, auto, disabled)? I understand physically how the network is setup I asking from a logical stand point.

Here is scenario 1.
Classmate ---- FMS ---- AP ---- Robot
FMS ---- Classmate ---- AP ---- Robot



A bad cable between the driver station and the switch or the alliance switch and the main switch is my theory. We only had comm problems on the red alliance. I have come across plenty of network cables that provide partial connectivity.

[EricH]

Alliance station status indicators:

Green--E-stop.
Flashing red or blue--no communication, hunting for communication, that sort of thing.
Solid red or blue--match running, communication.
All off--match waiting to start, communication. Either that or the entire system lost power...