Re: ARENA Fault
 

Yeah, I didn't even want to bring that one up, but the last 25 seconds no one had a comm signal, all the RSLs for the blue alliance were blinking fast. I suspect they didn't want to give us a reset because they were already hours behind schedule and assumed with 134 going dead right after automode that we didn't have a chance to win.
Buffer overflow from automode data that shouldn't be there? I seem to recall that in the match where we our robot didn't work one of our partners had an automode that didn't work correctly (possibly never left automode?). I really need to find a video of that match because I'm sure after all the matches I watched I'm getting some stuff confused.

Re: ARENA Fault
 

Just to set the record straight, part of the inspection process is to insure that the teams are using the boost regulated output of the Power Distribution to feed the wireless access point. This regulator makes a good 12 volts out, down to at least 5 volts on the battery. If the voltage gets that low, below 5.5 volts, the Crio inhibits outputs until the voltage rises. In the event that the voltage droops below the cutout voltage of the regulator, the wireless reboots. This is pretty obvious condition and takes 10-15 seconds on a good day and much longer on a regular day. In the event that the Crio reboots (below 4.5 volts), you all know how that varies with code on your own robots. Add to this, loose power connectors on the wireless access, loose Ethernet, dead Classmate batteries and code errors and we have some real problems.
However, not all issues can be explained away. A team that is showing comms but has no control has a problem. A team that is stuck in auto mode with no changes to code has a problem. A team that syncs up and never moves has a problem. A team that works in Blue 1 but not in Blue 2 has a problem. If an alliance wins an event because the opposing alliance has two dead robots, then that is a problem. There is still something else that is wrong.

Re: ARENA Fault
 

The robots not moving after a successful autonomous are almost certainly a robot problem, not a field problem. I know lots of ways for this to occur, most of them are code mistakes, but there are also dead driver controls and the typical bad connections on the robot as possibilities.

The major causes seem to be (assuming communications remain solid):

1. Robot effectively kills itself in autonomous (lots of ways to do this, even if it didn't do it the last match:) )
2. Robot Autonomous code refuses to give up control (language & framework dependent)
3. Autonomous code is running that the team didn't think was running (saw this for ~10 teams at NJ-"but we're not running any Autonomous code!"-Oh yes they were...:) )
4. Autonomous rattles an electrical connection loose
5. Bad battery (you'd be surprised how often this happens when teams should never let it occur)
6. Failed/underpowered USB hub/ driver controls

FMS isn't really in the pathway of communications from the Driver Station to the robot after everythings established. It just monitors the network traffic and defines which Driver Station must be which team. The communications go:

DS -> switch -> field router -> robot bridge -> cRIO

My home network is much more complicated.
If the RSL is showing good communications, then your network connection is fine and you should be looking at robot causes.


FMS signals the DS when it should change modes, but the RSL will verify if it's in the correct mode or not, and the DS will display the mode-how easy is that?

-------------
There are common communications problems where the DS or robot fails to connect to the field initially, but the field status lights and RSL tell that story if you look.
FMS can mess up if the same team number appears at two different Driver Stations, the field router/switches fail, Ethernet cables/connections can go bad, but these conditions show up as idiot status lights at the FMS station. If a single Ds or robot is not communicating, then FMS will not start a match. The scorekeeper must override and bypass any comminications failures to start a match, so the FTA has to decide to run without one team.
Where FMS gets complicated is in the scoring system, eStops, Team numbers, field status displays. It can fail to count balls correctly, etc.


The strangest thing I ran into was how team's didn't test properly back in the Pits.
"Well it worked when we ran Teleop in the Pits!"
Well, that's not how the failure occurred did it? That's looking under the street light for keys dropped across the street. Yes, it's easy to look there, but it isn't relevant.
Duplicate the conditions. Classmate on battery, same battery on the robot, and run a Practice match! The Practice match is even built into the DS controls this year.
Actually, I was advising teams to bring everything back to the pit (or to the side of the field if in eliminations) still powered up if it wasn't dangerous, so it could be examined and tested as is. Don't wiggle the connections or touch anything until you've attempted to recreate the problem, as is. You want it to break in the Pits after all.

Re: ARENA Fault
 

I'd like to throw out a request / suggestion; it's been a few years since I discussed this and it went nowhere then so maybe now...

It's kinda hard for a team to troubleshoot a system when they aren't allowed to have one of the most critical components in the loop. When you are in the pits, you are required to be on tether. At least this year you can use a radio on the practice field (although not your own) to see how the system works wireless rather than tethered, but that still doesn't tell you if your robot radio is functioning correctly. The only time you are allowed to have your own radio in the loop is when you are on the competition field.

What do we need to do to get the rules changed to check our own com's at some point before stepping onto the field? Can we designate specific times each hour when teams can check their signals? Why can't we do it when there are no matches taking place? Can we develop a procedure to use a field radio at the practice field that has the capability to broadcast to any given team (one at a time)? Can someone design a big tempest box or cage to put robots and controls in that contains the signal?

What about the other end of the equation. Should we have some kind of "control control" - a classmate and a cRIO with a battery sitting on the side of the field - that we can plug into any driver station to show that the field is working?

The assumption that if you troubleshoot each component in a system and find them all OK, then the system must be OK is not correct. Systems work differently than the sum of their components, and to troubleshoot the system you need the whole system together.

Re: ARENA Fault
 

@Gary:

At Peachtree this year and last, we did just about everything short of requiring a team to bring its robot to the field for a test before qualifications started. As a result, each team already knew which wires to connect and how long it should take to get everything going. I'd run them through a practice match to show that autonomous started, transitioned to teleop and then stopped at match end. For the teams that could not do this on Thursday, we came in early Friday to give them an opportunity to do it.

We had an HP PDA device that could emulate a driver station and a spare classmate available for testing. But you cant spend thirty minutes troubleshooting every robot that fails to connect to the field.

As far as I can tell, anyone can download FMS light. While it does not have the automatic scoring functions or match setup functions, otherwise it seems to be a fairly complete version of FMS. If teams were to run that I think they might find some of these issues and be able to correct them.

@FRC4ME:

As far as I know, FMS and its special supporting hardware were designed and built by 4FX Design. The Owner is Bob Pitzer, who occasionally posts here. I don't think WPI was involved with FMS.

The classmate drivers station software is an application written in Labview. I think it was done by employees of NI, but might be wrong there. There seems to be additional information at www.ni.com/first. I'd have to improve to be a neophyte at Labview, but from what I have experienced with it, I can imagine that there are many way to mess us the code that would result in unexpected behavior. Half the time I don't really understand why the code works as it does. It would be nice if NI had someone at every event to help. I do know from experience with my own team the the claim 'we didn't change anything' actually means 'we didn't change anything that we can remember or that we think might actually affect anything'. From decades in the computer business, it is usually the nothing change that brings the system down.

@ everyone else:

Thanks for adding to the understanding that will help get more robots working more of the time.

Re: ARENA Fault
 

I would like to suggest we alternate the ends we play from each match in the finals. If there is some advantage/disadvantage from any DS, it would either even out, or likely become more obvious from the swap. Other sports swap ends halfway through, why should FIRST be any different?

Re: ARENA Fault
 

Zach,
Teams would have to change bumpers way too often for that.

Re: ARENA Fault
 

The guidelines are ~10 minutes for a bumper change. The transition time for the semi-finals is like this:
your team completes the match you are in
3 minutes for the match reset
2.25 minutes for the alternate match
3 minutes for the match reset
---
8.25 minutes total

So the announcer would have to stall for a couple minutes if the bumper changes weren't complete. If a match went to a 3rd round maybe you don't change sides for that.

For the finals awards could be presented between matches to allow for extra changing time.

Re: ARENA Fault
 

I dislike the presentation of awards between matches format. Just take a look at Championship. It kills the excitement of the finals.

Really what we need to do here is fix the issues with the field rather than trying to console affected teams. (Note: I am not criticizing the creators of FMS. No software is perfect.)

Re: ARENA Fault
 

Awards aren't always presented in the same place as the game is played. Trying to do them at the same time would be totally unworkable at Boilermaker, for example.

Re: ARENA Fault
 

Why the implicit assumption that the problem is with the field? As you note, no software is perfect. Presumably that would include the code on the robot.

What we need is to understand how teams can lose controls to their robot while everything fieldwise looks normal. Only then can we implement the changes, whether to be done by teams to their robots or FIRST to the field (or both of course), that everyone wants to see happen.

I've spent a good bit of time the last few days researching possible causes for this issue here on CD as well as doing other research. There seem to be a lot of ways that USB connections, battery levels (classmate and robot), loose wiring and coding errors can cause these symptoms. Outside of the driver station ethernet connection (the loss of which would be indicated on the FMS screen), I'm having a hard time coming up with a way for FMS to affect just one team in a way that matches the symptoms described. Hopefully better minds than mine will find the answer soon.


----------------

ETA:

Check this thread for info about Labview coding that causes the robot to run autonomous, but then not be able to break out of it. Effectively resulting in no driver controls during teleop.

Re: ARENA Fault
 

I agree, the DS ethernet cable seems the most likely culprit if a single DS seems to have issues. In my experience there are always possible conditions with an ethernet cable that may not be automatically detectable. Give me a dozen ethernet cables and a rolling office chair and I can probably make one of these notorious cables for you.

Re: ARENA Fault
 

In this post a mentor describes unplugging and replugging the USB connection to regain control of the robot. If this solution works, is this cause for an arena fault? I would argue that it seems to be some interaction between the FMS and the Classmate that causes this problem since we haven't seen this behavior while a robot is tethered.

Re: ARENA Fault
 

Just last night I was fighting a USB problem on this very computer. The device was connecting and disconnecting repeatedly. It had worked in that port many times before. Plugging it into a powered hub solved the problem.

I suppose it is possible that the presence of the FMS could cause classmates to act differently in this area, but I doubt it is necessary. I think the classmate, Windows and the devices can screw themselves up without help from the FMS. Perhaps such problems are not noticed when tethered because you're not running in a match format?

Something else to look at.

Re: ARENA Fault
 

Even then, at the Wisconsin Regional, they could have switched the teams to the opposite sides of the field. After looking at the video again, 3 Robots were dead on the field. 2481, 706, and 1732 (2 Red, 1 Blue). We (2481) died right out of autonomous. Our Robot did not die during autonomous, it died around 1 second after autonomous. At that time 706 lost camera but still could drive. Then later in the match they lose complete communication going under the tower. 1732 appeared to lose communication while going over the bump. I am not entirely sure on their case.

The fair thing to do was at least switch sides. We would have accepted defeat if our robot was not operating on the other side, but the amount of problems the red side of the field had that day was outstanding.

It also made me angry when at the Midwest Regional they stopped a game because 2+ teams were inactive and called it on communication error.