ARENA Fault

[rspurlin]

I'm sorry to hear your event didn't work out as well as it could. Some of the scoring system issues I've read about sound as if the wires were not plugged in or plugged in correctly. I will be passing that thought on.

Hope to see you at championships.

[Zach Purser]

I have seen it repeatedly stated in this thread that a system error should effect an entire side and not just a single robot, but at VCU there were considerably more problems with robots being controlled from the center blue player station. During a seeding match, we were using that station and assured that we had a comm, but our robot was completely unresponsive. Our diagnostics in the pit showed no errors with our robot. We didn't make any changes and never had any similar problems at any of the other player stations. Our partners in the quarterfinals who were playing at that station had the same issue twice in a row. Both rounds of the quarterfinals they moved in autonomous mode but were unable to move as soon as teleop started, directly leading to our loss of both of the quarterfinal matches. They never had any issues at any other player station throughout the entire competition. We heard similar stories from other teams that were at that player station.

While I am extremely annoyed about what has already happened, I have been told that that same field will be used for the Raleigh regional where we will be both volunteering and participating. I would like to know what kind of errors could be associated with a single player station, why those errors wouldn't be flagged by the system, and what our team/volunteers can do to prove/disprove that there is an arena fault at the time, rather than doing some post-mortem statistical analysis to prove there was an issue.

[BEEKMAN]

Originally Posted by kylelanman

ARENA as it is defined in the rules.

The ARENA includes all elements of the game infrastructure that are required to play Breakaway: the FIELD, the ALLIANCE STATIONS, the GOALS, the BALLS, and all supporting communications, arena control, and scorekeeping equipment.

The only rule I see regarding ARENA faults.

<T18> If, in the judgment of the Head Referee, an “ARENA fault” occurs that affects either the play or the outcome of the MATCH, the MATCH will be replayed. Example ARENA faults include broken field elements, power failure to a portion of the field, improper activation of the field control system, errors by field personnel, etc.

A common response that I have heard is "After reviewing the log of the entire match nothing seems out the ordinary so the match will not be replayed". How is 1 to 4 robots loosing coms not something wrong? Something is clearly wrong. Thousands of fans can tell with out looking at a log that something went wrong.

Outside of FIRST when someone has an IT related problem and it can not be identified from the log we don't tell the person(s) they are out of luck and we aren't going to help them. The IT department looks into the problem and rectifies the situation as soon as possible.

The least they could do is have a professional on site at each regional to inspect the robot after the match back in the pit and help figure out what is wrong. How can a bunch of high-schoolers be expected to find the problem when the inspectors at the event can't. From the looks and sounds of it the only reason that help is not offered is that no one actually knows what causes all the comm problems.

If FIRST can not help us troubleshoot the mysterious comm problems that never happen at the shop but randomly happen on the field then teams should be given the benefit of the doubt and the match should be replayed when they occurr.

Goood point...bad logic. Often it is infact the fault of the Team, and an issue with the Robot. ex) We lost a match because when we went over a bump, one of our power cables (only for a split second) would come loose, causing our cRio to re-boot. After 30 mins of testing, we figured it out. At a glance, and even in depth, it appears to be an FMS problem, when in actuality, it isnt.

Another common problem is voltage drops. if hte voltage of your robot drops below 10, ther is a chance your wireless link will reboot itself.....causing comm issues.;

The FMS log will show al this.

The ONLY time in teh two regionals that I saw an Arena Fault (we decided it wasnt worth it to call it, and have teh match redone) was when Autonomous mode started, and 2 seconds later, thye game was disabled, only to become re-started at the START of autonomous mode again. This caused problems with teams' autonomous mode, as they had already driven a little before the second auto mode begining (causing them to miuss baslls due to improper spacing).

[Not2B]

Originally Posted by pathew100

Be careful there. I'm a scorekeeper. But yes, sometimes my "law" does come into play unfortunately.

Oh Pat, you and your law are the reason for so much of our frustrations...

In Ann Arbor, we didn't know if it was better to be red or blue. Red lost or messed up comms about half the time, but blue had faulty balls counters... So I guess it works out in the end.

I'm just glad there are the brave few who step up to work with this field, and deal with the frustration from the teams. Thanks!

[Mark McLeod]

Originally Posted by Zach Purser

... assured that we had a comm, but our robot was completely unresponsive.

What were the conditions of the multitude of DS status lights and what was the RSL showing?

I saw similar symptoms several times working NJ, but the cases I personally witnessed were not field faults.
Symptoms were:

• The robot communication and robot code lights on the DS remained green
• FMS showed good communication with both the DS and robot as well
• The robot drove in autonomous (strong hint!)
• The robot did not respond to driver controls in Teleop

Of course, that's obviously a driver controls issue not a field issue and is easily solved during a match as long as the drive team keeps their wits about them. Waste time blaming the field and you're doing nothing to help yourself.

It usually turned out to be a USB hub that was either defective or attempting to draw too much power from the Classmate. Often the IO board was plugged into the Hub (should be plugged directly into the Classmate because it's a real power hog).

Quick solution is to plug your joysticks directly into the Classmate to see if they operate.
You can also try hitting the F1 key to force the Classmate to reenumerate the joysticks.

[Zach Purser]

Originally Posted by Mark McLeod

What were the conditions of the multitude of DS status lights and what was the RSL showing?

I saw similar symptoms several times working NJ, but the cases I personally witnessed were not field faults.
Symptoms were:

* The robot communication and robot code lights on the DS remained green
* FMS showed good communication with both the DS and robot as well
* The robot drove in autonomous (strong hint!)
* The robot did not respond to driver controls in Teleop

That sounds like our symptoms except we weren't running an automode. Is the DS information logged somewhere in our system so we can examine it when we get back to the pits?

It usually turned out to be a USB hub that was either defective or attempting to draw too much power from the Classmate.

I'm doubtful this would be the problem since we used the same USB hub the whole regional and we only had an issue that one time, but I'll keep that in mind.

Often the IO board was plugged into the Hub (should be plugged directly into the Classmate because it's a real power hog).

We weren't using the IO board during the competition, so no issues there.

But I've gotten a bit off topic here. I will certainly concede that it's possible our robot had some sort of random error for one match. But there seemed to be a general pattern of that one driver station having issues across several different teams, and those teams never had issues in any of the other driver stations. I'm trying to figure out if there's anything field related that we should be watching for once we've eliminated the usual suspects?

[EricH]

Originally Posted by Zach Purser

But there seemed to be a general pattern of that one driver station having issues across several different teams, and those teams never had issues in any of the other driver stations. I'm trying to figure out if there's anything field related that we should be watching for once we've eliminated the usual suspects?

If you notice that, talk to the FTA. It might be a cable going bad in the drivers' station. The FTA would then swap the cable and see if that improved things.

[Andy Grady]

Just a few tips for teams who may be having issues with the robot suddenly dying in a match...

1. Always make sure you have a fully charged battery in your robot. Keep in mind also that during a match, your battery voltage fluctuates immensely depending on the current draw of your robot. The more current you draw at one time, the lower your battery voltage will fluctuate. If your battery drops below 8 or 9 volts in a match (which is logged by FMS) you are highly suspect.

2. Always make sure you have full voltage on your Classmate. As well as all the latest software. If your classmate dies, you are dead in the water.

3. Try to secure your cables the best you can. Vibrations and shock to your robot can cause cables to jump and lose connection, or cause the Rio to reboot.

4. It is good practice not to run signal and power wires together. Keep them separate to reduce noise.

5. Try to put your gaming adapter somewhere in open space on your robot. The more internal to the robot, the better chance that you will lose signal.

6. Check your code! Make sure when you were fooling around with it that you didn't goof something up!

I know it can be very frustrating to teams when the robot goes dead, but you can greatly reduce your risk of anything happening by just taking care of your wiring. In two years with this new control system, to my knowledge, my team has never run into a communication problem with the field. I feel that is because we take extra care with the signal wires, placement of components, and absorbing of shock. Take it from me...I work with all these network cables and such every day. They get flaky just sitting by themselves over time. In a robot, they are subjected to tons of vibration and shock, which expedites degradation of the physical connection. Good wiring practices make a difference.

[Mark McLeod]

Originally Posted by Zach Purser

I'm trying to figure out if there's anything field related that we should be watching for once we've eliminated the usual suspects?

There is no substitute for using all the status lights and the error messages the Driver Station provides on the Diagnostics tab. When there's a problem your coach should be noting and remembering everything those indicators show. The drivers should switch to that display before the match or during a failure on the field. The coach should note what the drivers are doing when a status light goes red. The most common example is when communication goes red when the driver leans on the joysticks. That indicates the cRIO or bridge just rebooted due to battery voltage suddenly being pulled really low by the drive train. If the RSL goes out, then it's the cRIO that rebooted.

The primary thing that FMS "controls" that your drivers should watch for, is Communications:

The DS displays the status of communication with both the FMS and the robot. When the robot fails to respond those status lights on the DS Diagnostic panel are the first thing your drivers should look at. The DS is just pinging each of those devices by-the-way. If the lights are all green, the problem is not communications.

FMS keeps a count of all the packets sent by both the DS and by the robot, tracks the number of dropped packets and when no packets are received. Status lights go red at the FMS station when one or the other fails to communicate. If the lights are all green, the problem is not communications.

The packets going between the DS and robot are just going through switches and the wireless field router. Technically, FMS doesn't really control this, it just monitors it. So when communications fail it could be:

• The Ethernet cable/connections between the DS and the Field end switch (affects only one Driver Station)
• The Ethernet cable/connection from the field-end switch to the field router (affects all single-alliance Driver Stations)
• The field router (affects all robots)
• The robot bridge (affects one robot)
• The Ethernet cable/connections between the robot bridge and the cRIO (affects one robot)

FMS will note and record lost packets caused by any of these failures, so they are all traceable from the FMS station.
It's just a LAN, nothing special. FMS is just a monitor.

The FMS is also responsible for ordering the DS into whatever mode the field wants- Disabled, Auto, Teleop.
These modes also show up on the DS, so the drivers should take note of these too when problems with the field arise.

[RoboMaster]

I have read through this thread and it seems most of the time it is just a freak think with the robot. Teams don't check something, wiring issue, coding issue, or Murphy's Law. I think a lot of this is because of the sophistication of the control system. There are a lot of components that are very high tech and many things can go wrong. I'm not bashing it, I really like the advanced level of what we've got. But that's what you have to sacrifice.

What Mark McLeod said was key: we need to learn how to diagnose and trouble shoot well. That way we can figure out if it was a FMS problem or a robot problem, and if so, in what way. Have your drivers read this thread and tell them what they need to know.

I agree that the lights and the diagnostic tab are great, but you need to know how to interpret them. They don't tell you everything. Our team compiled a list of the what all the lights on the robot mean. I suggest doing this too. I'll try to see if we have an electronic copy of ours somewhere and attach it if we do. The one complain I have is that the robot light has so many different meanings; it ultimately just tells you if something is wrong or not. This can get annoying.

[rspurlin]

Originally Posted by Zach Purser

. . . there seemed to be a general pattern of that one driver station having issues across several different teams, and those teams never had issues in any of the other driver stations. I'm trying to figure out if there's anything field related that we should be watching for once we've eliminated the usual suspects?

Start by noticing and writing down the status info provided to you on the classmate. Are you still showing comms? What is your battery voltage? Try to get the attention of the FTA to take a look at your drivers station and your robot. If you're on a field where I am scorekeeping, I've probably already radioed him to head your way if I see anything wrong.

The points of failure that are likely to affect only one robot (other than your classmate and the robot itself) are the ethernet wire and the port it is plugged into on the station cabinet underneath the center driver station. Past that point, network/radio traffic problems should affect either three or six robots.

At an off season event we ran last year, I had a team tell me there was a problem with the red 3 station after they lost robot controls. They cited that the team two matches previous had also lost controls so there was an obvious field fault. What they didn't know that I did is that the team in the previous match had their battery become disconnected. It's easy to make assumptions that don't turn out to be correct.

In an attempt to understand what might have happened, I looked at the match results from VCU. Team 435 played 3 of its 9 qualification matches in blue 2 and 1 of its 5 elimination matches in blue 2. Your post above indicates that 435 lost in quarterfinals, but VCU match results list 435 as being on the red alliance. Of the 4 matches 435 played in blue 2, did any of them work correctly? What am I missing?

[aechmtwash11]

I think it would be nice if teams could write logs to a txt file on the classmate and on the crio. This way programmers could have something to look at from an unbiased computer.

We saw it during our rounds at the Boilermaker Regional. One round our main driver (a mechanical guy) drove and diagnosed the robot as not having any control. Next round we had our kicker operator (a programmer) actually drive the robot he was able to isolate the problem. Eventually the programming team found that we had an outdated C-Rio image.

It all comes back to one person blames another, mechanical team says its programming, programming says its electrical, electrical says it mechanical

[EricH]

Originally Posted by aechmtwash11

Next round we had our kicker operator (a programmer) actually drive the robot he was able to isolate the problem. Eventually the programming team found that we had an outdated C-Rio image.

And now you know why the first 4 items on the inspection checklist are: WPA encryption, team number, cRIO image version, and DS firmware version. If you need to reimage the cRIO, check the version that you're imaging with to make sure it's the right one.

By the way, computers have no biases, feelings, or understanding of any language save the one they're designed to work with. As such, every computer is unbiased. The same cannot be said of their operators, leading to the dreaded exchange:

Team: It's the field, we know it is, we've been doing XYZ every time we've been out here!
Field personnel: It's your robot, we know it is, all our stuff says all your stuff is good!

[Zach Purser]

Originally Posted by rspurlin

In an attempt to understand what might have happened, I looked at the match results from VCU. Team 435 played 3 of its 9 qualification matches in blue 2 and 1 of its 5 elimination matches in blue 2. Your post above indicates that 435 lost in quarterfinals, but VCU match results list 435 as being on the red alliance. Of the 4 matches 435 played in blue 2, did any of them work correctly? What am I missing?

Oops, I meant to say our partners had issues in the semi-finals when we were on the blue side, my mistake.

It was match 49 where we had the issue ourselves, and aside from that one round we didn't have any other issues. After that round we had another team approach us and explain that they had the same problem in that DS shortly before us and suggested that other teams had also had issues in that DS. Yeah, I know, two datapoints and some anecdotes is not overwhelming evidence, but it just made us aware. Then when our partners had a similar problem in that DS in the semi-finals it really got our attention.

Now that I'm looking at the match results page I see that for semi 1-1 it lists us in blue 2, I was thinking 134 was in blue 2 both semi rounds. Maybe it was just Murphy messing with us if our partners were dead after auto from two different DSs.

Thanks for all your feedback. It will help our drive team to be better prepared in case something happens again.

[FRC4ME]

Originally Posted by rspurlin

Start by noticing and writing down the status info provided to you on the classmate. Are you still showing comms? What is your battery voltage? Try to get the attention of the FTA to take a look at your drivers station and your robot. If you're on a field where I am scorekeeping, I've probably already radioed him to head your way if I see anything wrong.

The points of failure that are likely to affect only one robot (other than your classmate and the robot itself) are the ethernet wire and the port it is plugged into on the station cabinet underneath the center driver station. Past that point, network/radio traffic problems should affect either three or six robots.

At an off season event we ran last year, I had a team tell me there was a problem with the red 3 station after they lost robot controls. They cited that the team two matches previous had also lost controls so there was an obvious field fault. What they didn't know that I did is that the team in the previous match had their battery become disconnected. It's easy to make assumptions that don't turn out to be correct.

In an attempt to understand what might have happened, I looked at the match results from VCU. Team 435 played 3 of its 9 qualification matches in blue 2 and 1 of its 5 elimination matches in blue 2. Your post above indicates that 435 lost in quarterfinals, but VCU match results list 435 as being on the red alliance. Of the 4 matches 435 played in blue 2, did any of them work correctly? What am I missing?

Originally Posted by Zach Purser

Oops, I meant to say our partners had issues in the semi-finals when we were on the blue side, my mistake.

It was match 49 where we had the issue ourselves, and aside from that one round we didn't have any other issues. After that round we had another team approach us and explain that they had the same problem in that DS shortly before us and suggested that other teams had also had issues in that DS. Yeah, I know, two datapoints and some anecdotes is not overwhelming evidence, but it just made us aware. Then when our partners had a similar problem in that DS in the semi-finals it really got our attention.

Now that I'm looking at the match results page I see that for semi 1-1 it lists us in blue 2, I was thinking 134 was in blue 2 both semi rounds. Maybe it was just Murphy messing with us if our partners were dead after auto from two different DSs.

Thanks for all your feedback. It will help our drive team to be better prepared in case something happens again.

I remember watching those matches at VCU (339, one of your alliance partners, was my high school team so I was watching them closely). During the first semifinal, 134 never moved outside of autonomous. Their DS light remained solid, however, indicating that communications were fine. I then heard one of 339's drivers say that 134, "never left autonomous mode." I'm not sure if by this she meant that the DS still showed "autonomous enabled," or just that the robot wouldn't move in teleop.

339, the alliance captain, tried to replace 134 with an alternate, but gave up after finding that the first 5 teams on the list had already packed up and left. As a side note, something needs to be done about this: at least the first and second teams in the bull pen should have their robot ready for play during elims. Otherwise the resource is useless.

During the second semifinal, an FTA stood directly behind 134 to help diagnose the problem if it occurred again. Despite what match results say, I can confirm that 134 was, in fact, still in the center DS for this match. And sure enough, they didn't move outside of autonomous. The FTA seemed to be looking through their DS diagnostic pages during the match. Unfortunately, he was distracted when, with about 25 seconds remaining, 339 lost communications (DS light was blinking). At the time, this made me even more certain of a field fault, but in hindsight, I can't imagine that one robot sticking in autonomous mode and another robot losing comms 1.5 minutes in are at all related.

After much discussion with the FTA, 339 told me that they weren't given a lot of answers, other than being told as usual that nothing was wrong with the field, and it was probably an error in 134's code. Out of curiosity, what type of code must one write to force a robot to stick in autonomous mode (DS says "autonomous enabled" during teleop)? I can't imagine how one could achieve this in the user program on purpose, much less by accident. Perhaps it's easier than I think.

I can't really come to any conclusions as to what caused the failures. On one hand, I can't imagine what code 134 could have written to cause the robot to intermittently stay in autonomous mode. On the other hand, I can't imagine a field failure that would cause one robot to stay in autonomous mode while everything else works fine. Maybe they were performing some task in autonomous that hogged the CPU, causing it to miss the notification that teleop had started. But wouldn't the DS still get this message and switch to teleop?

I do believe FIRST should allow teams to examine the FMS and DS code. Who wrote these programs, anyway? Was it WPI? I know that this year I found (an reported) several bugs in WPILib - and that's after teams have had a year of access to the source code. I'm not criticizing anyone, but merely acknowledging that no software is perfect, and the more eyes we have looking at it, the better it will become.

I certainly hope we can trust teams to be graciously professional enough not to exploit FMS flaws to give themselves an unfair advantage; a simple "no modification of DS software or interference with FMS operation is allowed" rule should take care of that. Furthermore, even if a team so completely against the spirit of FIRST so as to try such an exploit existed, I don't know how they could sabotage anything without being blatantly obvious. Disable opponent's robots? That would be fairly obvious. Control the robot during autonomous? Alliances can't touch their controls until teleop starts. Remotely alter the electronic score? The refs would eventually notice that.

Finally, if it is already possible (as the FTA says) for a team to force their robot into a mode other than the one the FMS is sending simply by altering the user program, it seems to me that exploits are already possible.