ARENA Fault
 

ARENA as it is defined in the rules.

The ARENA includes all elements of the game infrastructure that are required to play Breakaway: the FIELD, the ALLIANCE STATIONS, the GOALS, the BALLS, and all supporting communications, arena control, and scorekeeping equipment.

The only rule I see regarding ARENA faults.

<T18> If, in the judgment of the Head Referee, an “ARENA fault” occurs that affects either the play or the outcome of the MATCH, the MATCH will be replayed. Example ARENA faults include broken field elements, power failure to a portion of the field, improper activation of the field control system, errors by field personnel, etc.

A common response that I have heard is "After reviewing the log of the entire match nothing seems out the ordinary so the match will not be replayed". How is 1 to 4 robots loosing coms not something wrong? Something is clearly wrong. Thousands of fans can tell with out looking at a log that something went wrong.

Outside of FIRST when someone has an IT related problem and it can not be identified from the log we don't tell the person(s) they are out of luck and we aren't going to help them. The IT department looks into the problem and rectifies the situation as soon as possible.

The least they could do is have a professional on site at each regional to inspect the robot after the match back in the pit and help figure out what is wrong. How can a bunch of high-schoolers be expected to find the problem when the inspectors at the event can't. From the looks and sounds of it the only reason that help is not offered is that no one actually knows what causes all the comm problems.

If FIRST can not help us troubleshoot the mysterious comm problems that never happen at the shop but randomly happen on the field then teams should be given the benefit of the doubt and the match should be replayed when they occurr.

Re: ARENA Fault
 

If the field is communicating correctly, then the problem lies with either the teams or Murphy. Quite often, robots lose communication if a cable gets knocked loose--always check your radio-cRIO cable before and after the match. This is quite difficult to pinpoint using field equipment--everything will look normal, but the robot won't do a thing.

I know that at Arizona, we did have two experts (not necessarily professionals) who could find errors. One designed the FMS system and was judging the event; the other was a beta tester (2 years) who got a volunteer pass so that he wouldn't keep getting sent out of the field area because the FTA was always asking him to look at something. Sometimes, even they couldn't figure out why the comm link was lost.

As for replaying such matches when they occur, there is a really really good chance that that will cause the event to run very much over the time allowed. Not just a couple of minutes, either. We're talking running over the way the Championship always does--a couple of hours.

Re: ARENA Fault
 

Something obviously went wrong. It is not obvious that what went wrong has anything to do with the ARENA. If it was really an issue of losing communication, the FMS logs would confirm that to be the case.

More than one team has replaced their WGA and seen their troubles go away. That's not an ARENA fault. More than one team has confirmed problems with reading joysticks and/or the Cypress board after letting their Classmate go to sleep or when running low on battery power. That's not an ARENA fault either.

Team 45 lost a qualification match when the robot appeared to disable itself midway through autonomous, showed that it was re-enabled at teleop but with the drive motors nonresponsive, and sat there not moving for the rest of the match. Obviously a communication failure, right? Wrong. The diagnostic lights were green the whole time. The real fault was the battery we used for that match, which decided to die sometime between having been charged and the start of the match. (In that case, the FMS did pick up on the low voltage issue, and so did the drive team who knew what to look for on the Driver Station.)

Re: ARENA Fault
 

I've pulled two parts out of Alan's response here to highlight something that I have found extremely helpful in diagnosing our robot issues in the past.

The Driver's Station and Dashboard provide a large amount of feedback about what is going on with your robot.

At our event last year our robot stopped moving in three separate matches. By observing our Dashboard during these faults I was well aware that our DS was still connected to the robot and in 2 of the 3 matches to the cRIO. After thorough checks of the electrical system a few loose wires/chassis shorts/other issues were found and the problems never returned.

Knowing what information is provided on the Driver Station and Dashboard and how to use it to diagnose your robot is the best way to help track down these mysterious faults.

Re: ARENA Fault
 

I understand your frustration with regards to robot(s) losing communication with FMS during matches, but you need to understand a few things.

1) Sometimes, it is genuinely the fault of the robot (or team that fielded the robot). Robots can lose communication with FMS for a variety of reasons, a large number of which result from how the team hooked up their robot. A bad ribbon cable, poor electrical connection, etc. could all cause this. FMS logs will just show a large number (100%) dropped packets for the duration of such events. However, a large number of other events can cause FMS to show 100% dropped packets, hence it is (near) impossible to arrive at a sound conclusion. As much as any of us who've been hit by comm issues doesn't want to admit it, it is very like that the fault isn't in FMS, but its in some aspect of our robot. In the rare situations where FMS did fail, I wholeheartedly trust the people behind the scoring table to diagnose the issue accordingly and act appropriately, but I'm also willing to accept that they can't possibly figure everything out. Remember, our control system consists of a ultraportable PC, a wireless gaming adapter, and an industrial grade controller. Were any 2 of those components really intended to work in unison? No. This leaves plenty of room for small mistakes to cause catastrophic results (consider all the possible issues with the radio's reset button, a plug for the radio that's probably not designed to be shaken as robots collide/cross bumps/etc, and all other potential failure points, these wouldn't be on a system who's sole purpose was to robustly control FRC Robots. They also would probably make the system cost prohibitive).

2) The field crew does its very best to give every team the chance to play; regardless of whether their comm issues are robot-sided or field-sided. Genuine mistakes happen, and the FTA and co. would never intentionally ignore situations in which it is clear that a FMS bug caused robot(s) to lose communication. However, I have yet to see a situation in which FMS causes less than all of the robots on the field to lose communication.

3) The FTAs do their best to diagnose the issue, but understand that they're people too. I know first-hand that the FTA for the TC and Detroit FiM District events will stop at nothing to pinpoint the cause of robots not communicating with the field right. The FTA at Cass Tech was also obviously working extremely hard, as the back of his shirt was clearly covered in sweat. Unfortunately, there were still robots that had comm issues at both events. Nevertheless, it is important to credit the hard work that the FTAs (and FTAAs) do at all the events, since they're the ones working (largely) behind the scenes to make the whole event run smoothly.

4) All that being said, it is crucial that each and every team take all necessary measures to ensure that their robot doesn't become the cause of comm issues. Ensure that all electrical connections (especially those to the radio, cRIO, etc) are secure, especially if you traverse bumps. Ensure your code is robust, as a watchdog bug can (and will) cause a loss of comm, either momentarily or permanently.

5) Remember that FRC competitions are volunteer run. The FTAs are volunteers, the refs are volunteers, the querers are volunteers. If you know a cRIO/controls/something expert who's willing to volunteer at the events to look at robots who've had communication issues, then invite them to offer their help. Otherwise, other teams are more than willing to provide any assistance they can. If you're continuously having comm problems, ask the FTA if you can try to sync to the field after the end of matches on Friday (or maybe even at lunch).

I guess that's all I have to contribute...

Re: ARENA Fault
 

To take a shot in the dark here, there has been a major problem on the Driver Stations this year. There has been problems of the Joysticks saying that they are connected but not responding, most times the I/O board will be attached to the hub and be draining too much current for everything connected to the hub to remain operational. There have been some cases where it has happened with just joysticks connected to the hub, there's been multiple solutions thrown out there to be tested to solve the problem. Now, relating to your particular situation there is a chase that it was a field problem, it plausible, but by some chance it is also probable that this lack of Joystick response to the Classmate is the culprit, and this problem does not appear on the log.

Re: ARENA Fault
 

I keep seeing this idea presented in all these threads about suspected field faults. I disagree with it, and I'm surprised more people don't feel the same way.

1. Most of us have no idea what the FMS even logs in the first place, and therefore what kind of failures could be detected with these logs. For all I know, the log may consist of a timer that fires every 100ms that just writes "Everything is fine" into the logfile. I'm sure that the logs contain lots of data, and maybe it's enough that it really can rule out anything but a team problem, but how do we know there isn't a bug in the logging mechanism? Or some other gremlin? Surely anyone who's been involved with software for more than a few weeks can admit that bugs can end up anywhere and can easily trick you into thinking something is working when it isn't.

2. Somewhat of a rehashing of #1, can the field system confirm that packets from the OI are getting to the robot? Or does it just verify that it can "see" the robot and OI from it's vantage point? Is it possible that FMS can be successfully connected to the team's OI and successfully pinging (or whatever) the robot, but that the OI and the robot aren't talking to each other?

3. Dozens of teams are reporting problems here. Ultimately, it may not be a problem with the field hardware, but it seems pretty clear that there are defects with the system as a whole (the system being everything from the Classmate and Rio to the arena controls, the WiFI adapters, joysticks, etc). Who's responsible for the overall operation of the entire controls package? Who's looking into these reports of issues? Surely all the technical folks at FIRST aren't just writing this off as a team issue, are they? Even if they believe it is the fault of the teams, I would hope that they would want to get to the bottom of it anyway, just to be sure. With so many different vendors involved in the system, though, I'm not sure who takes ultimate ownership of a potential system-level issue.

I've been involved in enough FIRST competitions to know that when the people working at the event say, "there were no errors with the field", what they usually really mean is, "We couldn't find any specific problem, and therefore we're going to blame you (the team) so we don't have to replay the match. If we admit that we aren't 100% sure what happened, we'll be here for weeks as we replay almost every single match." I've worked at plenty of events and I've been part of these conversations several times.

Re: ARENA Fault
 

If the best of the best can't figure it out then there is clearly something wrong here. FIRST should not provide us with a control system that is so advance that no one is able to troubleshoot. Then tell us we're out of luck when a team that has a 15-0 record mysteriously looses comms in the Finals and they loose it all with no explanation at all.

I'd have to argue that when one robot loses comms and another video feed at the exact same time that there something wrong between the alliance station and the robot.....The only thing that falls in to that category that is between the alliance station and the robot that I am aware of is the Arena/FMS.

We tried to do that. We had the newer bridge and were trying to configure it during matches. We asked a question regarding the WPA or WPA2 setting to the WPA inspector. He did not answer the question instead took the radio set it to factory settings plugged it in to his computer and tried to use his script/program to configure it. He got confused as to why it didn't work and wanted to go watch his own team compete. When we tried to call a timeout to configure it and swap radios the head ref told us no.

We disabled the classmate from going to sleep early in the season. During the final matches used another team's classmate battery because ours was dead.

Re: ARENA Fault
 

That is my teams exact opinion in the matter.

Re: ARENA Fault
 

Certainly the control/field system interaction has grown complex, and there are far too many places and ways gremlins can interrupt normal robot operation. It's not like the IFI days when an expert on the full system was delivered with each field. Even in the IFI days, although they look rosey two years later, not every robot death on the field could be or was explained.

Nowadays, without a single source supplier, it's mostly luck if a Regional ends up with a volunteer or two with the necessary skills and experience to chase down and troubleshoot team issues with the field. With technical volunteers scarce in any case, to find one free to roam between the field and the pits to follow up with team troubleshooting is indeed rare.

I worked the NJ Regional running between the field and the teams to track down obscure problems that teams were having and do the same thing at the Long Island Regional. Part of the reason teams tend to blame the field when their robot stops is the black box nature of what FMS supplies. It's common for teams to blame the field, because one of their motors didn't work (but it worked in the pit!), or if their robot takes off on it's own.

Almost without fail teams who blamed the field did not look at their own multitude of status lights and couldn't give me any useful information. They couldn't even tell me what the RSL was doing when their robot failed. I had to be there on the field to analyse the problem and provide a correct diagnosis (correct diagnosis being one that finds a problem that would cause the symptoms).

The art of troubleshooting is something we mentors need to teach more of, but I've come to realize that many mentors don't have good troubleshooting skills either.

Re: ARENA Fault
 

Most of the time, the best of the best did figure it out. The one they couldn't figure out was why a particular combination of new/old adapters would trigger a field-wide comm loss for no apparent reason. No other combination did that. Try troubleshooting that one...

The WPA inspector apparently didn't read the update that informed everyone of the program. You can't use it on the new bridge--it has to be done manually. His wanting to go see his own team compete, however--while that's understandable, there's a time and place to do it, like when there's not a team that needs your help right now to be able to compete.

As for the timeout, there are specific rules regarding when you can/can't use one. Quals, you're out of luck. Elims, there's a specific window to call it--if you don't call it in that window, you're out of luck.

Regarding the loss of comms and the loss of video feed at the same time on two different teams: It may be that there were separate failures that just happened to happen at the same time--say, alliance partners hit each other by accident and both cables come loose simultaneously. That's not a field issue, it's a robot issue.

Re: ARENA Fault
 

I agree, and I will add that the opposite is also true: it's common for event staff and volunteers to blame the teams' equipment, because the field worked fine in the previous match. I think both parties assume their side is correct and therefore the other must be broken. The reality is there have been enough root-caused issues with both robots and the field that both should be suspected when an issue occurs. There's plenty of threads in the scorekeeper's forum on usfirst.org that talk about having FMS crash multiple times and having to "reboot the field", and obviously there's been enough issues found with wiring etc. on robots over the years too.

Re: ARENA Fault
 

We had the entire Blue side dead for Teleop in an early Quals match.
I'm sure everyone in the stands "knows" it was a field fault.

Of course one robot had a 5v battery, another had a 0 volt Classmate, and the third was pinned under a partner and didn't want to rip it's camera off trying to get out from under.

Re: ARENA Fault
 

It seems that Classmate battery issues are definitely a contributing factor to the feeling that the field doesn't work very well. I wish someone could explain to me why FIRST is torturing themselves by not providing a $5 power strip at each alliance station and in turn suffering from the massive number of reported issues that are related to Classmate batteries. It seems like that rule isn't doing anyone any good and is just making all the perceived problems appear worse than they are. I guess I don't have much sympathy for FIRST if they're getting blamed for field faults when the issue is really a low battery when they could so easily solve that particular problem.

Re: ARENA Fault
 

Be careful there. I'm a scorekeeper. :D But yes, sometimes my "law" does come into play unfortunately.

Re: ARENA Fault
 

Everyone needs to remember that with every year that FIRST changes something with the control system, there will be a huge learning curve for all to understand how it all works. Also, it takes time to really fine tune the system, yeh NI and FIRST may be able to debug and get the system working to the best it can, but the true test really won't come until the Regionals when we bombard the system with all we got.

I know that in DC there were many teams that were having comm issues, but I assure you (99.9% at least) that these problems had nothing to do with the FMS faulting. Many teams were forgetting to plug their raido's back into the e-port on the cRio, or were forgetting to put a fresh battery in their robot, or hadn't restarted the DS program since their last match to name some of the major issues. We had one team that barely even moved the whole competition because they were having some sort of comm issue after another, but they were gracious and went back to the pit and worked hard to figure out solutions. We had a NI rep at the regional and he worked very close with that team to help them through these problems. It is issues and interactions like that that will help everyone learn more about the system and where it's weaknesses are and how we can improve on them in the future.

I know that it gets frustrating to be out there and all of a sudden not be able to compete, but remember your gracious professionalism and trust that the technical people behind the scenes are working hard to solve any and all problems and will make sure that there is a fair solution that comes out of it.

Re: ARENA Fault
 

Any good troubleshooter will suspect everything. I witnessed some of those FMS reset issues, so I know it's not faultless.

The problem is of course that there are far too few troubleshooters who straddle the fence and can unemotionally examine both sides, and aren't consumed by official duties. I've always been able to find a problem with the robot that solved the issue (last year and this year), as long as I'm there to examine the status lights and diagnostic messages. Even with wireless bridge failures the statistics page on the bridge usually tells that story, as long as I can prevent the team from turning their robot off after failing in a match. That doesn't mean I don't list FMS as a potential culprit every time. In fact that's always the first consideration, because it needs to be checked immediately while the problem is occurring. FMS failures tend to take the whole field down, or one entire end, or a single driver station eStop might fail, but it's pretty obvious to the field crew when that happens.

You can't actually solve a problem unless you can divorce yourself from taking sides. If you find yourself blaming something you know nothing about, then that's not troubleshooting and it won't solve the problem.

One of the issues I've noticed this year and last is that teams look on the NI reps as experts in the whole system, whereas, they typically are not. They are all experts in LabVIEW, many in cRIO operation, some work with teams and know the whole FRC control scheme, but few of them know anything about FMS.

Re: ARENA Fault
 

706 was with 2481 in the finals at Milwaukee. We had the camera loss at the same time as they lost all control. It was imediatly after the auto mode ended and there had been no robot to robot interactions and no collisions with the field. After the match we logged out of driver mode and back in and regained the camera. Sure was hard trying to win the match with 2481 down again and a sub playing defense. Nothing against the sub, they did a great job and kept the score down to 7-3 but with noone feeding us balls from middle we were pretty helpless. As coach I am hitting myself now for not telling our third match defence partner(sorry I don't remember the team number) to abandon defence and feed us some balls. Grrr Hind sight.
Anyway, I have to ask why is it that we are using a control system that is so complex and error prone? I can go to a model airplane field and have 15 planes in the air at the same time and none of them will have com problems.
Are there some things that every team should do between matches other than look for hardware issues like rebbot the classmate or log out and back in? I know that some computer systems need to be restarted regularily to keep from getting flaky errors. Also, having power at the drivers stations could be helpfull here. Is it legal to bring a battery and an inverter to the drivers station?
Bruce

Re: ARENA Fault
 

Yes.

http://forums.usfirst.org/showthread.php?t=15017

Re: ARENA Fault
 

I agree too.

That's the first thing I put a stop to when I was asked to Tech the SBPLI field last year-empty speculation by field staff. If the field staff didn't KNOW what the problem was, they were to refer it to me and not start guessing. It's actually harmful in that it usually focuses troubleshooting in exactly the wrong place. I worked the WPA desk one regional where if they had any field connection problems they sent it back to us to be reset, but I never found even one that was set incorrectly. Of course, the robot that caught on fire was plenty obvious...

It worked out well for Long Island last year, but I still had an IT professional telling me it was a field fault that shutdown only his team's drive motors. His explanation was that it couldn't be his code, and to be fair it actually wasn't, but it also wasn't a field failure. It could have been prevented by adding an error check in his code.

Re: ARENA Fault
 

While our robot never lost communication with the field, I did notice some problems:
1) We took FOREVER to establish the initial communication with our robot sometimes. While we were waited on to establish communication before starting the match, it was very frustrating to have to cycle power on the bot and cause delays for everybody.
2) Sometimes in the middle of the match I noticed some lag on the connection between the robot and my joysticks. It is pretty intense during a match so I am not sure if the robot was losing comm for a second or two every now and then or if it was just a bad wifi connection.

Re: ARENA Fault
 

I've found laggy controls to be a robot problem, not field. Most common for me have been the camera and CAN bus. No camera comms = error messages every 5 ms = no CPU for robot operation. Same applies to CAN. Who knows what else could do it

Re: ARENA Fault
 

We also had the problem of the robot losing connection with the driver's station before the match started. They let us stay out and reboot for the first 5 matches but for 6 and 7 they made us leave the field before we had time to reboot so the robot was dead. We showed the field people that if we waited until the team number was changed from the previous match our robot worked fine. We tried that for match 8 and it didn't work when the two alliance partners were on the field. For match 9 we waited until the other two alliance partners were fully connected and that worked. Were were able to run this way all the way to through the finals. We still don't know what the problem is but we'll try to solve it at 10,000 Lakes. We're a bit concerned that if we don't get it solved we won't be able to run in Atlanta.

Re: ARENA Fault
 

1. Everyone deals with field faults at times. Dean and Woody like to point out that this isn't a "fair" game. You're playing against better funded, better engineered, better mentored teams and there isn't much you can do about it. It sucks, but it's part of the game. I know it sounds glib, but these are facts we end up having to deal with.

2. FIRST does the best they can. They have some top notch folks working on these issues. We were lucky - we got to work with some of them during beta. Their turn-around time on issues that are found is spectacular - in many instances they are issuing fixes in as little as a day. But in the end they are a limited resource working with limited time and limited money. The result is an imperfect product. (See Toyota's accelerator pedals for how that works).

3. A great majority of the issues that are occuring /right now/ on the field are being caused by the teams. In nearly every match we played during our first regional, we had to wait for one and sometimes two teams that hadn't bothered to charge their laptop, or make sure it was up and running before the match. None had paid any attention to the multiple warnings during the prior week. Shame on them.

I know - none of these will make you feel any better. We've been in your shoes: last year our driverstation fried (static) in the elims during a week 1 regional and they gave us absolutely no special dispensation to fix it. We spent multiple elim matches with a dead / half dead robot. It wasn't our fault. It stunk. In the end we cursed those little blue bricks, the regolith, and the officials (who were simply following the rules) before moving on and getting over it.

First will do the best they can, and so will we. The only thing I can say is to try to stay gracious and professional, and teach your kids on your team how to do the same in the face of an unfair result. Continue to try to get into the beta tests, present papers at nationals, and help local teams to improve the end product so that we can all have fewer issues.

Re: ARENA Fault
 

I mentor a team that has had similar issues, and I volunteer as a scorekeeper, so I have a foot in both graves so to speak.

Most of the information we have available to us from the FMS display is the same information you have showing on your classmate. We do have the luxury of seeing it for all six robots at once. Much of what is written into the logs is the info each team's classmate reports to the FMS. So at the moment your classmate looses contact with your robot, that would be logged. Of course, the FMS doesn't know why it happened.

When I'm doing the scorekeeper job, I'm constantly monitoring both the action on the field and the FMS display. When I see that a robot's battery voltage is unusually low, I alert the FTS by radio to alert the team. They might have noticed it on their classmate display, but also might not. Likewise for missed packets and drops in robot link. Dpending on whether experience leads us to think it is most likely centered around the robot or the driver station, I'll ask the FTA to investigate while the match is still being played. I make my own written notes and cross check that with event history up to that point, to look for issues that follow teams or driver stations.

During the events, scorekeepers and FTA's have comm links back to FIRST HQ when things seem to be out of whack. In my opinion, you are making a mistake if you think that these people do not care deeply and take seriously every issue and possible bug that comes up. But FIRST is not an immense army; It would surprise most of you how few paid staffers actually make FIRST happen. Give them some credit for working many long hours. Same witht he event volunteers. I've been a part of many reviews and discussions long after the pits close about what transpired during the day. It's not uncommon for us to make a decision to reverse a DQ/red card/yellow card or replay a match the next day if we feel it is the fair thing to do.

There seems to be an assumption that the field is more complex the last two years than it was for IFI run events. I'm not sure that's true. One example is the radios. Most don't remember that IFI placed a tree of individual radios to facilitate robot communication. It was possible that one of them could fail (or the individual wires that led to them), leading to one robot losing communication while the others ran fine. Today's field has a single radio in the form of a Cisco access point which communicates with all robots simultaneously. A failure for one is likely a failure for all and thus a replayed match.

Each classmate is connected by ethernet cable to a switch in the station cabinet in each alliance station. This in turn is wired to the access point which communicates to the robots. There aren't a lot of points of failure here that could affect one team and not an entire alliance or all robots. Still, there are some and we watch those both during the match and from match to match. If I saw evidence of an intermittent error at one driver station, I'd alert the FTA and head ref and recommend replaying some matches if I thought there was any sort of field problem. A replay does not have to be done immediately.

Electrical problems can be much harder to sort out than mechanical ones. We do not replay matches when your chain breaks or your motor burns up or you forget to plug something back in. We do not have the luxury of replaying matches until the outcome is unaffected by robot design and construction issues. We also cannot do so when your programming fails or a breaker trips or something shorts out temporarily. But for some reason, maybe because those failures are sometimes not obvious (or even leave no evidence), a claim of field fault almost inevitably follows.

I share your frustration when you don't know how to fix your problems, but sometimes we don't get the information back to help you. Once the teams leave the field and begin the troubleshooting process, often they find the problem was a simple one of their own making, but don't let us know. If I did get that information, I could pass it on to other teams as best practices to keep their robots in top form. Unfortunately I usually only get feedback from immediately after the match as the FTA makes an immediate diagnosis. That said, reread this and similar threads where you'll find the variety of ways a robot can fail during an active FIRST match and try to avoid these things happening to you.

Re: ARENA Fault
 

@rspurlin

Perhaps you could provide me with a little more insight. What you just provided was great.

The light on the alliance station, what does this indicate? Our RSL light was indicating normal operation while our alliance light was flashing. A few seconds later the RSL started flashing too. Our assumption is that the classmate lost comms first and then when the robot noticed the classmate was gone it disabled itself and the RSL updated to indicate it was disabled.

After this happened logging out of the Driver profile, into Developer, running arp -d, logging out of developer, and back into driver gave us comms for a few seconds. I'm hoping you can answer this question. Does the system allow a team to logout in the middle of a match if everything is connected correctly? I was not on the field but was told that they tried this in an earlier match and it did not let them logout.

I have 1 final question as to how the FMS is setup. I kinda assumed from day one that the FMS controls the robot. When allowed the classmates send signals to the FMS that get forwarded to the robot. Or is the FMS setup so that the classmates control the robots directly and the FMS send signals to the classmate to change the mode (teleop, auto, disabled)? I understand physically how the network is setup I asking from a logical stand point.

Here is scenario 1.
Classmate ---- FMS ---- AP ---- Robot
FMS ---- Classmate ---- AP ---- Robot



A bad cable between the driver station and the switch or the alliance switch and the main switch is my theory. We only had comm problems on the red alliance. I have come across plenty of network cables that provide partial connectivity.

Re: ARENA Fault
 

Alliance station status indicators:

Green--E-stop.
Flashing red or blue--no communication, hunting for communication, that sort of thing.
Solid red or blue--match running, communication.
All off--match waiting to start, communication. Either that or the entire system lost power...

Re: ARENA Fault
 

I'm aware of that I more specifically interested in how communication or lack there of is determined?
No communication with the Robot or
No communication with the Classmate or
Classmate can't communicate with robot?

Re: ARENA Fault
 

I don't know how they check robot comms, other than the RSL. Classmate-FMS is viewed in the Dashboard. I'd guess you can check the dashboard (some of the indicators would tell you).

Re: ARENA Fault
 

Eric and Kyle,
I believe I was the inspector. The team (and many other teams) brought me an adapter that they couldn't ping. Neither could I, even after two manufacturer resets. I could not connect, I could not see the web interface although it appeared to be handshaking to the computer. A power reset at one point, sent the device into never never land and it never completed. The team took it from me when it would not boot. I merely confirmed what they already thought.
In the final match in Wisconsin, three robots failed. I was informed that two of them lost the Classmate due to dead batteries and the third was a shutdown of the USB hub powered from the Classmate, also a battery issue.
I wish I had a nickel for every match I have missed because I was helping a team. I could buy a nice lunch.
I believe the alliance station also flashes during autonomous if a robot is not running autonomous code.

Re: ARENA Fault
 

Ray, thank you for your insightful analysis. As I indicated in the posts about the problems surrounding the Israeli Regional (http://www.chiefdelphi.com/forums/sh...793#post938793), our FTA and support staff worked tirelessly to resolve issues. We could always move backwards to a simpler system; but that system cannot offer the processing power available in the cRio. Challenges are what FIRST is all about. We all have to accept the challenge and move forward; we will be better for it.

We do a disservice as mentors if we simply complain. We must inspire our students by showing them how we rise to the challenge and solve the problems we face.

Re: ARENA Fault
 

Al, We thank you for trying to help us with our radio last second. We were in the web interface and configuring it fine. We simply had a configuration question. From what I understand from other threads the program where you simply enter the team number does not work with the newer radios. Would have been nice to know that then. Oh well. After it was reset completely we did not have the time that it would take to configure it from scratch because after applying nearly every setting you have to wait for the device to restart which takes nearly a minute. The robot worked in the next match so the drivers told us to forget about messing with it.

Al, Do you by chance recall which teams were because of dead classmate batteries? We didn't have a USB hub so I would assume one of them was us but I also know that team 1736 let us use there fully charge classmate battery because First did not provide a way (that we were aware of) to charge the classmates during the final matches. I recently found out about the inverter option with a spare robot battery.

The reason I want to know how the comm status is determined is because there are multiple network connection between the classmate, FMS, and the robot. I fully understand a robot radio loosing comms is our fault. There are a countless number of things that could cause this. If the classmate battery is fully charged, ethernet cable securely connected and the classmate looses comms to the FMS I would have to blame the field/arena. The classmate is an unmodified (software and hardware) product from First so nothing the team does aside from the previously mentioned network cable being plugged in properly would influence the comms between the classmate and the FMS. If I am wrong here correct me.

Re: ARENA Fault
 

as an FYI the frc director's blog got a post today that said power strips are now available in the queue for charging the classmate while waiting for the match.

According to rspurlin the FMS indicators are pulled from the classmate. I'd assume that those are used to determine communication, as well as an extra indicator for classmate communication. Also you should remember that the FMS is a fairly big thing. There are multiple fault points, however most of them would cause widespread failures.

One more thing out of curiosity: In the event that a ball is scored and properly returned on the ball return, however the return sensor fails and doesn't detect the ball, does the scorekeeper have a way to manually override the DOGMA penalties to keep the game from killing an alliance for a field error? Also, what happens if there is the reverse, a ball is missed by the ball counter and is caught by the ball return. Any idea how the system handles this?

Re: ARENA Fault
 

I'm not sure what all the paths are but by looking at your Dashboard under the Diagnostics tab I can tell whether:
the DS is talking to FMS
what mode FMS/DS is in (auto/teleop/enabled/disabled)
the bridge is talking to FMS
the bridge is talking to the cRio
the cRio is running code
you have decent battery voltage

BTW this is more detailed information than we have ever had before, even in the rosy IFI days some people remember so fondly. I might not be able to tell you what the source of your problem is, but I can pretty well tell you where to look.

Re: ARENA Fault
 

On the topic of FMS-classmate-robot or classmate-FMS-robot, I do see one big security issue in the former. We all know about the FMS Locked state, but few seem to know that FMS Locked can be bypassed with F1 for enable and Space bar for disable (those also work in regular mode. Much more convenient than messing with the mouse). If the FMS somehow loses connection with the classmate, but the classmate-robot communications can continue, if the classmate activates the bypass would the robot start operating without being under FMS control?

Re: ARENA Fault
 

That helps all the teams from the previous regionals.....It will be nice in the future though. Btw does anyone know why First has never and from what we can tell never will provide power at the driver station. It seems like a simple thing that would make everyone's lives easier and the question has been asked from multiple people on our team. Quite frankly we don't have a darn clue.

The blue ball counters rarely worked at WI. It took the refs a few matches to realize that the teams were not as slow as they thought. We originally lost a match because it said we got three penalties for not getting balls back in time. When we looked the match results the next day we saw the penalties were gone. The majority of the matches the refs / others kept score manually.

I just confirmed that our classmate battery was 75% charged. If this causes a problem then from the sounds of it maybe the classmate is not the computer to be using a driver station.....or as previously stated it should be able to be powered during competition.

Re: ARENA Fault
 

Someone in another thread said that the power spots for last year's DS have been taken up by the scoring systems now. 3 units (2 goals and the return) to replace the 3 DS spots. FIRST probably just didn't want to add more power strips to the field.

75% seemed fine for us. We ran matches below 20% by the end of the day Friday and we didn't have any battery-related issues

Re: ARENA Fault
 

Kyle,
I didn't even see the final match as I was helping out with some other issues. I was told afterward by field volunteer who spoke with the FTA. No teams were identified.
If you were able to open the web interface than I was thinking of someone else. On the device I was on I couldn't do anything.
In Wisconsin, volunteers were counting balls as they passed through the goal. I don't know if or how the Dogma was working. I only know that the ref asked for a count from each volunteer, one at each goal.

Re: ARENA Fault
 

@kylelanman

I wish I could definitively answer your questions, but there is much more that I don't know than I do. Here are my best guesses:

I think Eric H. is correct above in his description of the alliance station light. This light is different this year than in previous years. You want this solid. I'm not sure what controls it specifically, but it has a different flash pattern than the Robot signal light.

The robot signal light is discussed here. Once again you want this on solid. In finding this document I was reminded that there needs to be a jumper installed for the amber light to accurately reflect the LED status on the digital sidecar.

The classmate is able to exchange packets with the robot prior to match begin. FMS will not allow a start with robots not ready. If a robot-DS connection cannot be made, your robot can be bypassed to enable the match to begin, but at events where I have been, you would know that was the case before the MC begins the countdown. So what we need to know is why your connection dropped.

I'm guessing based on your screen name, but assume that you know something about what a non-routable network is. This is essentially what an FRC field is. This is why if robot communications fails to just one robot, the culprit is most likely either on the robot itself or the classmate and its individual ethernet cable. A failure in the rest of the wiring loop would most likely affect multiple robots.

I do not know how the classmate is controlled by FMS. I think it is most likely detecting the presence of the FMS and then behaving slightly differently, but I'm not certain. At first glance, I'd try to do it this way since once the match is started, packets to and from the robot are then handled only by the communications hardware (router, switch, access point) and not through any upper level software program (which could reduce bandwidth). It would be simple enough to have the driving program on the classmate poll the FMS every 100 ms or so and enable/disable as required. Of course, that might not be how it is really implemented.

I'm not a big fan of Windows*, especially on underpowered devices, so I'm not happy with the boot time and other idiosyncrasies of the classmate. However, it's much better than last year's blue box of death. I have a laptop that forgets it has a DVD drive every time someone closes the lid, so I'm not sure I'd recommend your logout/login regimen. We've already heard reports that USB connections can get wonky in certain circumstances. I think there are issues we are still uncovering as our first season with this device unfolds.

I'm pretty sure you can't connect with the field if your classmate is logged in as developer. You may have been able to have the same effect by simply logging out and logging back in. I've heard that it is necessary to do this after each match to clear the FMSlock.

At your next event, please check with the FTA early on practice day and let him.her know what you have experienced so far. Perhaps by that time more answers will be available.

At my next scorekeeping event, I will be taking even better notes about what is happening and what our best guess is regarding the causes. Hopefully this will help improve gameplay for all teams.


* I am an IT guy IRL, so please spare me the flame war. I haven't met the perfect computer/OS/programming language/DBMS/app/etc in over thirty years of working with them, so let's just focus on making what we have better, ok?

Re: ARENA Fault
 

When you say you looked at match results do you mean match review in FMS?

I can't say about the classmate specifically, but some laptops do treat attached peripherals differently based on power profile and will select different power profiles based on the amount of perceived charge.

I've already ordered a power cable to connect an inverter to the standard robot battery. I don't want to go on the field with a classmate that is less than 100%. Especially considering the potentially long queue waits at championships.

Re: ARENA Fault
 

http://www2.usfirst.org/2010comp/eve...chresults.html

Match 2

Re: ARENA Fault
 

Some information on how the FMS works can be gleaned from the FMS lite package that was released for the 2009 game. Here's the user's manual:
http://usfirst.org/uploadedFiles/Com...Guide_RevC.pdf


I would add that I find it extremely frustrating that not only are we not allowed to look at the source code for the FMS or DS, we're expected to deal with their imperfections without even a general idea of how the field works (or doesn't).

Re: ARENA Fault
 

The posted results do not list components of the score, just the final result. So I'm confused. Do you mean that after the match, red was announced as the winner, yet sometime later some penalties disappeared and blue became the winner?

As a hypothetical situation, had such a situation happened at our event (discovery after the match results were announced that an error needed to be corrected affecting the outcome) the head ref would go to each of the teams involved in the match to explain what had happened and what the new ruling was. After all teams are notified, a general announcement would be made by the MC and/or head ref to the crowd, explaining what happened and what the new ruling was. Because the revised score would be committed to FMS as soon as the new ruling is reached, rankings would reflect the fact first.

I'm surprised about the ball counters at your event. At ours, we think they worked flawlessly. We tested them Wed night, and Thursday, Friday and Saturday mornings. I'm not aware of any complaints that a score was incorrect relative to the ball counters at our event.

Re: ARENA Fault
 

Dealing with imperfections is a theme FIRST recognizes in areas other than programming. I wonder if the source code to Windows would be more helpful to troubleshooting DS problems than the source to FMS. :yikes:

Seriously, I'd be more worried if the source were freely available that a large well funded team could exploit some weakness.

While I'm not dumb enough to assert that FMS or the DS software is bug free, it seems pretty solid from my perspective and will get better if we help make it so.

Re: ARENA Fault
 

Yes. It was a nice surprise when we saw it that night seeing as how it was our originally our only loss.

I know the ref did not talk to myself personally and everyone on the teams was shocked when they saw that we ended up going undefeated so I do not believe the head ref told anyone.

It worked during practice and possibly for Q1. From then on out the live score on the board was never correct for the blue alliance and you had to wait till the end to see it. It made it nearly impossible to try and keep a 1 point margin between the winners and losers not knowing the score. So we altered our strategy and went with score as many points as possible at all times.

Re: ARENA Fault
 

I'm sorry to hear your event didn't work out as well as it could. Some of the scoring system issues I've read about sound as if the wires were not plugged in or plugged in correctly. I will be passing that thought on.

Hope to see you at championships.

Re: ARENA Fault
 

I have seen it repeatedly stated in this thread that a system error should effect an entire side and not just a single robot, but at VCU there were considerably more problems with robots being controlled from the center blue player station. During a seeding match, we were using that station and assured that we had a comm, but our robot was completely unresponsive. Our diagnostics in the pit showed no errors with our robot. We didn't make any changes and never had any similar problems at any of the other player stations. Our partners in the quarterfinals who were playing at that station had the same issue twice in a row. Both rounds of the quarterfinals they moved in autonomous mode but were unable to move as soon as teleop started, directly leading to our loss of both of the quarterfinal matches. They never had any issues at any other player station throughout the entire competition. We heard similar stories from other teams that were at that player station.

While I am extremely annoyed about what has already happened, I have been told that that same field will be used for the Raleigh regional where we will be both volunteering and participating. I would like to know what kind of errors could be associated with a single player station, why those errors wouldn't be flagged by the system, and what our team/volunteers can do to prove/disprove that there is an arena fault at the time, rather than doing some post-mortem statistical analysis to prove there was an issue.

Re: ARENA Fault
 

Goood point...bad logic. Often it is infact the fault of the Team, and an issue with the Robot. ex) We lost a match because when we went over a bump, one of our power cables (only for a split second) would come loose, causing our cRio to re-boot. After 30 mins of testing, we figured it out. At a glance, and even in depth, it appears to be an FMS problem, when in actuality, it isnt.

Another common problem is voltage drops. if hte voltage of your robot drops below 10, ther is a chance your wireless link will reboot itself.....causing comm issues.;

The FMS log will show al this.

The ONLY time in teh two regionals that I saw an Arena Fault (we decided it wasnt worth it to call it, and have teh match redone) was when Autonomous mode started, and 2 seconds later, thye game was disabled, only to become re-started at the START of autonomous mode again. This caused problems with teams' autonomous mode, as they had already driven a little before the second auto mode begining (causing them to miuss baslls due to improper spacing).

Re: ARENA Fault
 

Oh Pat, you and your law are the reason for so much of our frustrations...

In Ann Arbor, we didn't know if it was better to be red or blue. Red lost or messed up comms about half the time, but blue had faulty balls counters... So I guess it works out in the end.

I'm just glad there are the brave few who step up to work with this field, and deal with the frustration from the teams. Thanks!

Re: ARENA Fault
 

What were the conditions of the multitude of DS status lights and what was the RSL showing?

I saw similar symptoms several times working NJ, but the cases I personally witnessed were not field faults.
Symptoms were:

• The robot communication and robot code lights on the DS remained green
• FMS showed good communication with both the DS and robot as well
• The robot drove in autonomous (strong hint!)
• The robot did not respond to driver controls in Teleop

Of course, that's obviously a driver controls issue not a field issue and is easily solved during a match as long as the drive team keeps their wits about them. Waste time blaming the field and you're doing nothing to help yourself.

It usually turned out to be a USB hub that was either defective or attempting to draw too much power from the Classmate. Often the IO board was plugged into the Hub (should be plugged directly into the Classmate because it's a real power hog).

Quick solution is to plug your joysticks directly into the Classmate to see if they operate.
You can also try hitting the F1 key to force the Classmate to reenumerate the joysticks.

Re: ARENA Fault
 

That sounds like our symptoms except we weren't running an automode. Is the DS information logged somewhere in our system so we can examine it when we get back to the pits?
I'm doubtful this would be the problem since we used the same USB hub the whole regional and we only had an issue that one time, but I'll keep that in mind.
We weren't using the IO board during the competition, so no issues there.

But I've gotten a bit off topic here. I will certainly concede that it's possible our robot had some sort of random error for one match. But there seemed to be a general pattern of that one driver station having issues across several different teams, and those teams never had issues in any of the other driver stations. I'm trying to figure out if there's anything field related that we should be watching for once we've eliminated the usual suspects?

Re: ARENA Fault
 

If you notice that, talk to the FTA. It might be a cable going bad in the drivers' station. The FTA would then swap the cable and see if that improved things.

Re: ARENA Fault
 

Just a few tips for teams who may be having issues with the robot suddenly dying in a match...

1. Always make sure you have a fully charged battery in your robot. Keep in mind also that during a match, your battery voltage fluctuates immensely depending on the current draw of your robot. The more current you draw at one time, the lower your battery voltage will fluctuate. If your battery drops below 8 or 9 volts in a match (which is logged by FMS) you are highly suspect.

2. Always make sure you have full voltage on your Classmate. As well as all the latest software. If your classmate dies, you are dead in the water.

3. Try to secure your cables the best you can. Vibrations and shock to your robot can cause cables to jump and lose connection, or cause the Rio to reboot.

4. It is good practice not to run signal and power wires together. Keep them separate to reduce noise.

5. Try to put your gaming adapter somewhere in open space on your robot. The more internal to the robot, the better chance that you will lose signal.

6. Check your code! Make sure when you were fooling around with it that you didn't goof something up!

I know it can be very frustrating to teams when the robot goes dead, but you can greatly reduce your risk of anything happening by just taking care of your wiring. In two years with this new control system, to my knowledge, my team has never run into a communication problem with the field. I feel that is because we take extra care with the signal wires, placement of components, and absorbing of shock. Take it from me...I work with all these network cables and such every day. They get flaky just sitting by themselves over time. In a robot, they are subjected to tons of vibration and shock, which expedites degradation of the physical connection. Good wiring practices make a difference.

Re: ARENA Fault
 

There is no substitute for using all the status lights and the error messages the Driver Station provides on the Diagnostics tab. When there's a problem your coach should be noting and remembering everything those indicators show. The drivers should switch to that display before the match or during a failure on the field. The coach should note what the drivers are doing when a status light goes red. The most common example is when communication goes red when the driver leans on the joysticks. That indicates the cRIO or bridge just rebooted due to battery voltage suddenly being pulled really low by the drive train. If the RSL goes out, then it's the cRIO that rebooted.

The primary thing that FMS "controls" that your drivers should watch for, is Communications:

The DS displays the status of communication with both the FMS and the robot. When the robot fails to respond those status lights on the DS Diagnostic panel are the first thing your drivers should look at. The DS is just pinging each of those devices by-the-way. If the lights are all green, the problem is not communications.

FMS keeps a count of all the packets sent by both the DS and by the robot, tracks the number of dropped packets and when no packets are received. Status lights go red at the FMS station when one or the other fails to communicate. If the lights are all green, the problem is not communications.

The packets going between the DS and robot are just going through switches and the wireless field router. Technically, FMS doesn't really control this, it just monitors it. So when communications fail it could be:

• The Ethernet cable/connections between the DS and the Field end switch (affects only one Driver Station)
• The Ethernet cable/connection from the field-end switch to the field router (affects all single-alliance Driver Stations)
• The field router (affects all robots)
• The robot bridge (affects one robot)
• The Ethernet cable/connections between the robot bridge and the cRIO (affects one robot)

FMS will note and record lost packets caused by any of these failures, so they are all traceable from the FMS station.
It's just a LAN, nothing special. FMS is just a monitor.

The FMS is also responsible for ordering the DS into whatever mode the field wants- Disabled, Auto, Teleop.
These modes also show up on the DS, so the drivers should take note of these too when problems with the field arise.

Re: ARENA Fault
 

I have read through this thread and it seems most of the time it is just a freak think with the robot. Teams don't check something, wiring issue, coding issue, or Murphy's Law. I think a lot of this is because of the sophistication of the control system. There are a lot of components that are very high tech and many things can go wrong. I'm not bashing it, I really like the advanced level of what we've got. But that's what you have to sacrifice.

What Mark McLeod said was key: we need to learn how to diagnose and trouble shoot well. That way we can figure out if it was a FMS problem or a robot problem, and if so, in what way. Have your drivers read this thread and tell them what they need to know.

I agree that the lights and the diagnostic tab are great, but you need to know how to interpret them. They don't tell you everything. Our team compiled a list of the what all the lights on the robot mean. I suggest doing this too. I'll try to see if we have an electronic copy of ours somewhere and attach it if we do. The one complain I have is that the robot light has so many different meanings; it ultimately just tells you if something is wrong or not. This can get annoying.

Re: ARENA Fault
 

Start by noticing and writing down the status info provided to you on the classmate. Are you still showing comms? What is your battery voltage? Try to get the attention of the FTA to take a look at your drivers station and your robot. If you're on a field where I am scorekeeping, I've probably already radioed him to head your way if I see anything wrong.

The points of failure that are likely to affect only one robot (other than your classmate and the robot itself) are the ethernet wire and the port it is plugged into on the station cabinet underneath the center driver station. Past that point, network/radio traffic problems should affect either three or six robots.

At an off season event we ran last year, I had a team tell me there was a problem with the red 3 station after they lost robot controls. They cited that the team two matches previous had also lost controls so there was an obvious field fault. What they didn't know that I did is that the team in the previous match had their battery become disconnected. It's easy to make assumptions that don't turn out to be correct.

In an attempt to understand what might have happened, I looked at the match results from VCU. Team 435 played 3 of its 9 qualification matches in blue 2 and 1 of its 5 elimination matches in blue 2. Your post above indicates that 435 lost in quarterfinals, but VCU match results list 435 as being on the red alliance. Of the 4 matches 435 played in blue 2, did any of them work correctly? What am I missing?

Re: ARENA Fault
 

I think it would be nice if teams could write logs to a txt file on the classmate and on the crio. This way programmers could have something to look at from an unbiased computer.

We saw it during our rounds at the Boilermaker Regional. One round our main driver (a mechanical guy) drove and diagnosed the robot as not having any control. Next round we had our kicker operator (a programmer) actually drive the robot he was able to isolate the problem. Eventually the programming team found that we had an outdated C-Rio image.

It all comes back to one person blames another, mechanical team says its programming, programming says its electrical, electrical says it mechanical

Re: ARENA Fault
 

And now you know why the first 4 items on the inspection checklist are: WPA encryption, team number, cRIO image version, and DS firmware version. If you need to reimage the cRIO, check the version that you're imaging with to make sure it's the right one.

By the way, computers have no biases, feelings, or understanding of any language save the one they're designed to work with. As such, every computer is unbiased. The same cannot be said of their operators, leading to the dreaded exchange:

Team: It's the field, we know it is, we've been doing XYZ every time we've been out here!
Field personnel: It's your robot, we know it is, all our stuff says all your stuff is good!

Re: ARENA Fault
 

Oops, I meant to say our partners had issues in the semi-finals when we were on the blue side, my mistake.

It was match 49 where we had the issue ourselves, and aside from that one round we didn't have any other issues. After that round we had another team approach us and explain that they had the same problem in that DS shortly before us and suggested that other teams had also had issues in that DS. Yeah, I know, two datapoints and some anecdotes is not overwhelming evidence, but it just made us aware. Then when our partners had a similar problem in that DS in the semi-finals it really got our attention.

Now that I'm looking at the match results page I see that for semi 1-1 it lists us in blue 2, I was thinking 134 was in blue 2 both semi rounds. Maybe it was just Murphy messing with us if our partners were dead after auto from two different DSs.

Thanks for all your feedback. It will help our drive team to be better prepared in case something happens again.

Re: ARENA Fault
 

I remember watching those matches at VCU (339, one of your alliance partners, was my high school team so I was watching them closely). During the first semifinal, 134 never moved outside of autonomous. Their DS light remained solid, however, indicating that communications were fine. I then heard one of 339's drivers say that 134, "never left autonomous mode." I'm not sure if by this she meant that the DS still showed "autonomous enabled," or just that the robot wouldn't move in teleop.

339, the alliance captain, tried to replace 134 with an alternate, but gave up after finding that the first 5 teams on the list had already packed up and left. As a side note, something needs to be done about this: at least the first and second teams in the bull pen should have their robot ready for play during elims. Otherwise the resource is useless.

During the second semifinal, an FTA stood directly behind 134 to help diagnose the problem if it occurred again. Despite what match results say, I can confirm that 134 was, in fact, still in the center DS for this match. And sure enough, they didn't move outside of autonomous. The FTA seemed to be looking through their DS diagnostic pages during the match. Unfortunately, he was distracted when, with about 25 seconds remaining, 339 lost communications (DS light was blinking). At the time, this made me even more certain of a field fault, but in hindsight, I can't imagine that one robot sticking in autonomous mode and another robot losing comms 1.5 minutes in are at all related.

After much discussion with the FTA, 339 told me that they weren't given a lot of answers, other than being told as usual that nothing was wrong with the field, and it was probably an error in 134's code. Out of curiosity, what type of code must one write to force a robot to stick in autonomous mode (DS says "autonomous enabled" during teleop)? I can't imagine how one could achieve this in the user program on purpose, much less by accident. Perhaps it's easier than I think.

I can't really come to any conclusions as to what caused the failures. On one hand, I can't imagine what code 134 could have written to cause the robot to intermittently stay in autonomous mode. On the other hand, I can't imagine a field failure that would cause one robot to stay in autonomous mode while everything else works fine. Maybe they were performing some task in autonomous that hogged the CPU, causing it to miss the notification that teleop had started. But wouldn't the DS still get this message and switch to teleop?

I do believe FIRST should allow teams to examine the FMS and DS code. Who wrote these programs, anyway? Was it WPI? I know that this year I found (an reported) several bugs in WPILib - and that's after teams have had a year of access to the source code. I'm not criticizing anyone, but merely acknowledging that no software is perfect, and the more eyes we have looking at it, the better it will become.

I certainly hope we can trust teams to be graciously professional enough not to exploit FMS flaws to give themselves an unfair advantage; a simple "no modification of DS software or interference with FMS operation is allowed" rule should take care of that. Furthermore, even if a team so completely against the spirit of FIRST so as to try such an exploit existed, I don't know how they could sabotage anything without being blatantly obvious. Disable opponent's robots? That would be fairly obvious. Control the robot during autonomous? Alliances can't touch their controls until teleop starts. Remotely alter the electronic score? The refs would eventually notice that.

Finally, if it is already possible (as the FTA says) for a team to force their robot into a mode other than the one the FMS is sending simply by altering the user program, it seems to me that exploits are already possible.

Re: ARENA Fault
 

Yeah, I didn't even want to bring that one up, but the last 25 seconds no one had a comm signal, all the RSLs for the blue alliance were blinking fast. I suspect they didn't want to give us a reset because they were already hours behind schedule and assumed with 134 going dead right after automode that we didn't have a chance to win.
Buffer overflow from automode data that shouldn't be there? I seem to recall that in the match where we our robot didn't work one of our partners had an automode that didn't work correctly (possibly never left automode?). I really need to find a video of that match because I'm sure after all the matches I watched I'm getting some stuff confused.

Re: ARENA Fault
 

Just to set the record straight, part of the inspection process is to insure that the teams are using the boost regulated output of the Power Distribution to feed the wireless access point. This regulator makes a good 12 volts out, down to at least 5 volts on the battery. If the voltage gets that low, below 5.5 volts, the Crio inhibits outputs until the voltage rises. In the event that the voltage droops below the cutout voltage of the regulator, the wireless reboots. This is pretty obvious condition and takes 10-15 seconds on a good day and much longer on a regular day. In the event that the Crio reboots (below 4.5 volts), you all know how that varies with code on your own robots. Add to this, loose power connectors on the wireless access, loose Ethernet, dead Classmate batteries and code errors and we have some real problems.
However, not all issues can be explained away. A team that is showing comms but has no control has a problem. A team that is stuck in auto mode with no changes to code has a problem. A team that syncs up and never moves has a problem. A team that works in Blue 1 but not in Blue 2 has a problem. If an alliance wins an event because the opposing alliance has two dead robots, then that is a problem. There is still something else that is wrong.

Re: ARENA Fault
 

The robots not moving after a successful autonomous are almost certainly a robot problem, not a field problem. I know lots of ways for this to occur, most of them are code mistakes, but there are also dead driver controls and the typical bad connections on the robot as possibilities.

The major causes seem to be (assuming communications remain solid):

1. Robot effectively kills itself in autonomous (lots of ways to do this, even if it didn't do it the last match:) )
2. Robot Autonomous code refuses to give up control (language & framework dependent)
3. Autonomous code is running that the team didn't think was running (saw this for ~10 teams at NJ-"but we're not running any Autonomous code!"-Oh yes they were...:) )
4. Autonomous rattles an electrical connection loose
5. Bad battery (you'd be surprised how often this happens when teams should never let it occur)
6. Failed/underpowered USB hub/ driver controls

FMS isn't really in the pathway of communications from the Driver Station to the robot after everythings established. It just monitors the network traffic and defines which Driver Station must be which team. The communications go:

DS -> switch -> field router -> robot bridge -> cRIO

My home network is much more complicated.
If the RSL is showing good communications, then your network connection is fine and you should be looking at robot causes.


FMS signals the DS when it should change modes, but the RSL will verify if it's in the correct mode or not, and the DS will display the mode-how easy is that?

-------------
There are common communications problems where the DS or robot fails to connect to the field initially, but the field status lights and RSL tell that story if you look.
FMS can mess up if the same team number appears at two different Driver Stations, the field router/switches fail, Ethernet cables/connections can go bad, but these conditions show up as idiot status lights at the FMS station. If a single Ds or robot is not communicating, then FMS will not start a match. The scorekeeper must override and bypass any comminications failures to start a match, so the FTA has to decide to run without one team.
Where FMS gets complicated is in the scoring system, eStops, Team numbers, field status displays. It can fail to count balls correctly, etc.


The strangest thing I ran into was how team's didn't test properly back in the Pits.
"Well it worked when we ran Teleop in the Pits!"
Well, that's not how the failure occurred did it? That's looking under the street light for keys dropped across the street. Yes, it's easy to look there, but it isn't relevant.
Duplicate the conditions. Classmate on battery, same battery on the robot, and run a Practice match! The Practice match is even built into the DS controls this year.
Actually, I was advising teams to bring everything back to the pit (or to the side of the field if in eliminations) still powered up if it wasn't dangerous, so it could be examined and tested as is. Don't wiggle the connections or touch anything until you've attempted to recreate the problem, as is. You want it to break in the Pits after all.

Re: ARENA Fault
 

I'd like to throw out a request / suggestion; it's been a few years since I discussed this and it went nowhere then so maybe now...

It's kinda hard for a team to troubleshoot a system when they aren't allowed to have one of the most critical components in the loop. When you are in the pits, you are required to be on tether. At least this year you can use a radio on the practice field (although not your own) to see how the system works wireless rather than tethered, but that still doesn't tell you if your robot radio is functioning correctly. The only time you are allowed to have your own radio in the loop is when you are on the competition field.

What do we need to do to get the rules changed to check our own com's at some point before stepping onto the field? Can we designate specific times each hour when teams can check their signals? Why can't we do it when there are no matches taking place? Can we develop a procedure to use a field radio at the practice field that has the capability to broadcast to any given team (one at a time)? Can someone design a big tempest box or cage to put robots and controls in that contains the signal?

What about the other end of the equation. Should we have some kind of "control control" - a classmate and a cRIO with a battery sitting on the side of the field - that we can plug into any driver station to show that the field is working?

The assumption that if you troubleshoot each component in a system and find them all OK, then the system must be OK is not correct. Systems work differently than the sum of their components, and to troubleshoot the system you need the whole system together.

Re: ARENA Fault
 

@Gary:

At Peachtree this year and last, we did just about everything short of requiring a team to bring its robot to the field for a test before qualifications started. As a result, each team already knew which wires to connect and how long it should take to get everything going. I'd run them through a practice match to show that autonomous started, transitioned to teleop and then stopped at match end. For the teams that could not do this on Thursday, we came in early Friday to give them an opportunity to do it.

We had an HP PDA device that could emulate a driver station and a spare classmate available for testing. But you cant spend thirty minutes troubleshooting every robot that fails to connect to the field.

As far as I can tell, anyone can download FMS light. While it does not have the automatic scoring functions or match setup functions, otherwise it seems to be a fairly complete version of FMS. If teams were to run that I think they might find some of these issues and be able to correct them.

@FRC4ME:

As far as I know, FMS and its special supporting hardware were designed and built by 4FX Design. The Owner is Bob Pitzer, who occasionally posts here. I don't think WPI was involved with FMS.

The classmate drivers station software is an application written in Labview. I think it was done by employees of NI, but might be wrong there. There seems to be additional information at www.ni.com/first. I'd have to improve to be a neophyte at Labview, but from what I have experienced with it, I can imagine that there are many way to mess us the code that would result in unexpected behavior. Half the time I don't really understand why the code works as it does. It would be nice if NI had someone at every event to help. I do know from experience with my own team the the claim 'we didn't change anything' actually means 'we didn't change anything that we can remember or that we think might actually affect anything'. From decades in the computer business, it is usually the nothing change that brings the system down.

@ everyone else:

Thanks for adding to the understanding that will help get more robots working more of the time.

Re: ARENA Fault
 

I would like to suggest we alternate the ends we play from each match in the finals. If there is some advantage/disadvantage from any DS, it would either even out, or likely become more obvious from the swap. Other sports swap ends halfway through, why should FIRST be any different?

Re: ARENA Fault
 

Zach,
Teams would have to change bumpers way too often for that.

Re: ARENA Fault
 

The guidelines are ~10 minutes for a bumper change. The transition time for the semi-finals is like this:
your team completes the match you are in
3 minutes for the match reset
2.25 minutes for the alternate match
3 minutes for the match reset
---
8.25 minutes total

So the announcer would have to stall for a couple minutes if the bumper changes weren't complete. If a match went to a 3rd round maybe you don't change sides for that.

For the finals awards could be presented between matches to allow for extra changing time.

Re: ARENA Fault
 

I dislike the presentation of awards between matches format. Just take a look at Championship. It kills the excitement of the finals.

Really what we need to do here is fix the issues with the field rather than trying to console affected teams. (Note: I am not criticizing the creators of FMS. No software is perfect.)

Re: ARENA Fault
 

Awards aren't always presented in the same place as the game is played. Trying to do them at the same time would be totally unworkable at Boilermaker, for example.

Re: ARENA Fault
 

Why the implicit assumption that the problem is with the field? As you note, no software is perfect. Presumably that would include the code on the robot.

What we need is to understand how teams can lose controls to their robot while everything fieldwise looks normal. Only then can we implement the changes, whether to be done by teams to their robots or FIRST to the field (or both of course), that everyone wants to see happen.

I've spent a good bit of time the last few days researching possible causes for this issue here on CD as well as doing other research. There seem to be a lot of ways that USB connections, battery levels (classmate and robot), loose wiring and coding errors can cause these symptoms. Outside of the driver station ethernet connection (the loss of which would be indicated on the FMS screen), I'm having a hard time coming up with a way for FMS to affect just one team in a way that matches the symptoms described. Hopefully better minds than mine will find the answer soon.


----------------

ETA:

Check this thread for info about Labview coding that causes the robot to run autonomous, but then not be able to break out of it. Effectively resulting in no driver controls during teleop.

Re: ARENA Fault
 

I agree, the DS ethernet cable seems the most likely culprit if a single DS seems to have issues. In my experience there are always possible conditions with an ethernet cable that may not be automatically detectable. Give me a dozen ethernet cables and a rolling office chair and I can probably make one of these notorious cables for you.

Re: ARENA Fault
 

In this post a mentor describes unplugging and replugging the USB connection to regain control of the robot. If this solution works, is this cause for an arena fault? I would argue that it seems to be some interaction between the FMS and the Classmate that causes this problem since we haven't seen this behavior while a robot is tethered.

Re: ARENA Fault
 

Just last night I was fighting a USB problem on this very computer. The device was connecting and disconnecting repeatedly. It had worked in that port many times before. Plugging it into a powered hub solved the problem.

I suppose it is possible that the presence of the FMS could cause classmates to act differently in this area, but I doubt it is necessary. I think the classmate, Windows and the devices can screw themselves up without help from the FMS. Perhaps such problems are not noticed when tethered because you're not running in a match format?

Something else to look at.

Re: ARENA Fault
 

Even then, at the Wisconsin Regional, they could have switched the teams to the opposite sides of the field. After looking at the video again, 3 Robots were dead on the field. 2481, 706, and 1732 (2 Red, 1 Blue). We (2481) died right out of autonomous. Our Robot did not die during autonomous, it died around 1 second after autonomous. At that time 706 lost camera but still could drive. Then later in the match they lose complete communication going under the tower. 1732 appeared to lose communication while going over the bump. I am not entirely sure on their case.

The fair thing to do was at least switch sides. We would have accepted defeat if our robot was not operating on the other side, but the amount of problems the red side of the field had that day was outstanding.

It also made me angry when at the Midwest Regional they stopped a game because 2+ teams were inactive and called it on communication error.

Re: ARENA Fault
 

That is a large amount of effort and an even longer awards ceremony just to compensate for issues that shouldn't exist in the first place while taking away from an alliance's repair time.

Re: ARENA Fault
 

The problem here is that there are potential issues that are difficult to detect, and if they're difficult to detect then it's hard to get the issues fixed. If the teams were to switch ends and teams consistently had fewer issues on one end than the other, then it would become more obvious that there is an issue to fix. If the team has issues on both ends of the field then it shows the issue is specific to that team. The problem may still be something in the robot-FMS interaction, but at least you know the driver station isn't cursed (cough*centerbluestation*cough).

Re: ARENA Fault
 

I have found an issue for us with the Classmate and Intel's CPU power saving scheme called SpeedStep. By going into the Classmate's BIOS and disabling the SpeedStep option I was able to eliminate periodic system watchdog errors that we had been getting (approx 4 per minute).

While several people have said that the FMS doesn't care about watchdogs, the system watchdog is an indicator that communications between the Classmate and the cRIO has been lost (at least temporarily). I'm hoping that this was the reason that our robot stopped running in multiple rounds at the Suffield, CT scrimmage - I guess we will find out later this week at Hartford.

You can view the thread I started on this issue here (please post to this thread if this solution improves your robot communication issues):
http://www.chiefdelphi.com/forums/sh...d.php?p=943661

Merle Yoder
The GRUNTS Team#3146

Re: ARENA Fault
 

GDC response to my recommendation for managing operation off tether at competition - they will be considering how to do this for next year

http://forums.usfirst.org/showthread.php?t=15210