Some thoughts on rules, refs, and ranking

[AmoryG]

I do agree with the OP on some point, particularly with the occasional horrible penalty call. I empathize with your team because I remember a time when I felt the rules screwed my team out of a win that would of sent our alliance to the semifinals.

I also agree that the best teams aren't always on top. I occasionally see teams I did not expect at the top of the rankings, or even elite teams that somehow are in the bottom half of the rankings. Take 254 at the San Diego Regional. They finished with a 5-5 record, but were picked by the #1 seed even though they struggled throughout the event. And on the other side of the spectrum are teams that are ranked near the top, but are declined by many higher seeding teams. I agree that the rankings can be a little messed up, but generally they're pretty good and almost always the best teams form the best alliances and they will usually win the events.

Also, keep in mind that FIRST really is doing the best job it can do. And given that the competitions are almost exclusively run by volunteers, I think FIRST is doing a pretty good job. And if you don't think so, just look at other competitions such as in college and professional sports, and you'll know FIRST is doing something right. College sports are dominated by a few teams, and it's very rare other teams rise above the rest. Every regional is worth watching in FIRST because there's always a few outstanding teams that aren't recognized outside of the region. And while our volunteer referees and judges aren't perfect, is our system so much worse than the systems in professional sports? I'm always hearing how the greatest players are favored by referees, and the blatant cheating that participants get away because of how difficult it is to objectively call fouls and penalties. And in every sport I can think of, the perennial playoff contenders are almost universally hated. Great teams in FIRST are universally respected, and we actually cheer for winners as often as we cheer for the underdogs. And think of all the smack players talk about players from other teams. It really is amazing how a bunch of high school students act more maturely than these "professional" athletes.

So I agree FIRST has problems, but I can't agree that it hasn't been doing a good job for the most part.

[alex334]

Originally Posted by AmoryG

I do agree with the OP on some point, particularly with the occasional horrible penalty call. I empathize with your team because I remember a time when I felt the rules screwed my team out of a win that would of sent our alliance to the semifinals.

I also agree that the best teams aren't always on top. I occasionally see teams I did not expect at the top of the rankings, or even elite teams that somehow are in the bottom half of the rankings. Take 254 at the San Diego Regional. They finished with a 5-5 record, but were picked by the #1 seed even though they struggled throughout the event. And on the other side of the spectrum are teams that are ranked near the top, but are declined by many higher seeding teams. I agree that the rankings can be a little messed up, but generally they're pretty good and almost always the best teams form the best alliances and they will usually win the events.

Also, keep in mind that FIRST really is doing the best job it can do. And given that the competitions are almost exclusively run by volunteers, I think FIRST is doing a pretty good job. And if you don't think so, just look at other competitions such as in college and professional sports, and you'll know FIRST is doing something right. College sports are dominated by a few teams, and it's very rare other teams rise above the rest. Every regional is worth watching in FIRST because there's always a few outstanding teams that aren't recognized outside of the region. And while our volunteer referees and judges aren't perfect, is our system so much worse than the systems in professional sports? I'm always hearing how the greatest players are favored by referees, and the blatant cheating that participants get away because of how difficult it is to objectively call fouls and penalties. And in every sport I can think of, the perennial playoff contenders are almost universally hated. Great teams in FIRST are universally respected, and we actually cheer for winners as often as we cheer for the underdogs. And think of all the smack players talk about players from other teams. It really is amazing how a bunch of high school students act more maturely than these "professional" athletes.

So I agree FIRST has problems, but I can't agree that it hasn't been doing a good job for the most part.

Yes, FIRST does a very good job. It got me to where I am today. The volunteers are amazing and you can never thank them enough. Yes, the attitude of FIRST is pretty great. However, that doesn't mean it cannot greatly improve. You should always be seeking to improve. I wanted to call attention to a few things that I noticed that could/might be improved. I wanted to see some explanations for both sides of the argument in all cases. If something is not possible, that's fine, I'll accept that. I just wanted clarifications on a few thoughts.

The thing for me with powerhouse teams is that they inspire others. I always drew inspiration from 254. I always wanted to and still do want to become more like them. If they shine in competition and seed high, that gives teams a clear goal. But if teams like that are not in the top spots, what message does that send?

[EricH]

Originally Posted by alex334

But if teams like that are not in the top spots, what message does that send?

It sends a message that even the best have an off event now and again. They have their issues like the rest of us. And, there is hope that maybe we can be asking them to join our alliance instead of the other way around.

Incidentally, I'd like to point out some other, really annoying ranking systems.
2001--besides the 4v0, IIRC it was just the scores of the teams--if your colored ball was on the goal you got 10 extra points.
2003--details escape me, but I remember it being some function of scoring.
2010--When Coopertition Score was the biggest driving factor of the standings, things got "interesting", including a "planned" 6v0 match where the loser shot ahead of the winner in the standings.
2012--The bridge points affected an awful lot of rankings. 254 couldn't get a good ranking at one event because for one reason or another nobody went for the Coop Bridge with them.

This year, it's back to the good ol' win-loss-tie system, which like a republic is the worst system around except for all the others.

[slijin]

A long post justifies a long response.

Originally Posted by alex334

Firstly, the ranking system. This year's seemed to particularly wonky. At a regional as large as NYC, eight matches per team simply don't cut it. How can a robot's abilities be assessed fairly in comparison with the rest when it mathematically does not have the chance to play with/against every other team. A lot of luck seemed to be involved.

Luck will always be involved. Time will always be a constraint. Unless a better, viable alternative emerges, this is how things will work.

Originally Posted by alex334

I would propose a larger implementation of the district model. That seems to work a lot better and attract smaller amounts of teams.

The district model also requires a more expansive program, even more experienced and efficient volunteers, and stronger outreach than NYC offers. The unfortunate reality is that despite efforts to do so, we haven't been able to introduce NYC to the district model precisely because we don't have the resources to make the district model viable here.

Originally Posted by alex334

The large amount of penalties (I saw 120+ points scored on just penalties) also seemed to skew the rankings, especially when fouls were counted as goal/climb points. Let's face it, not everyone reads the rules. Why punish teams that do by dragging them down? Why not find a better way to rank based on individual performance.

Because teams shouldn't be ranked just by individual performance, but how they contribute to an alliance. That match you mention? We were on the receiving end of those penalty points; we won that match 10-165 because of those penalty points. Our very next match, we were partnered with the team that incurred those penalties, coached them so that they wouldn't do so, and went on to win that match as well.

Moreover, individual performance isn't exactly something you can judge quantitatively - e.g. how do you compare offense with defense? 3137 played fantastic defense throughout eliminations and were critical to our wins, but they didn't earn our alliance any actual points.

Originally Posted by alex334

Secondly, rules. In general, I liked the rules a lot. The ones that bothered me were those regarding penalties (this is going to be a common theme in this post). "Obstructing the flow of the game." What's that all about? It just sounds like defense to me, not to mention that it's extremely subjective. When does a good defense begin to obstruct the game?

IMO, its intent is to encourage scoring over blockading - that is, to encourage active gameplay and discourage anything that inhibits active gameplay. Maybe it's not a "fair" rule, persay; certainly, it is very subjective precisely because of its ambiguity, and makes defense harder, but it's part of the game. And when the game itself prevents you from executing a certain strategy, you work around it.

Originally Posted by alex334

Instead of creating rules like this, why not just create a game in which this is not possible? Instead of placing two obnoxiously large pyramids in the middle of the field, with either side being easily blocked, why not create a climbing wall and leave a wide open field? Maybe place a different, smaller element there instead.

It's not exactly a new rule:

Originally Posted by 2012 Competition Manual

[G23]
Robots on the same Alliance may not work together to blockade the Court in an attempt to stop the flow of the Match.
This rule has no effect on individual Robot-to-Robot defense.
Violation: Technical-Foul

Originally Posted by 2011 Competition Manual

<G48-C> ALLIANCE ROBOTS may not work together to blockade the FIELD in an attempt to stop the
flow of the MATCH. This rule has no effect on individual ROBOT-to-ROBOT defense.

Originally Posted by alex334

Which brings me to my final complaint. This exact bias in ruling occurred during the elimination matches of the NYC regional this year. (http://www.ustream.tv/recorded/29860861). Not only do I find this unfair, but it all of the refs seemed to disagree on what the large penalty was for.

This is an inevitable problem that will arise when making judgments with respect to ambiguous rules - that is why there's a head ref, so that there can be someone to bring a close to the discussion.

Originally Posted by alex334

I asked him to consider watching videos of previous matches in which I would gladly point out more extreme scenarios only a few matches prior which were not penalized.

You aren't the first, nor you will you be the last person to argue with this policy, which FIRST explicitly lays out in the manual:

Originally Posted by 2013 Competition Manual

5.5.3 Referee Interaction
The Head Referee has the ultimate authority in the ARENA during the event, but may receive input from additional sources, e.g. Game Designers, FIRST personnel, and technical staff. The Head Referee rulings are final. The Head Referee will not review recorded replays under any circumstances.

Originally Posted by alex334

If you could prove your point using the footage available, should that not count?

If the footage you use to prove your point offers a misleading perspective of the situation, should it count?

This isn't something our Head Referee decided on the spot. This is an enduring FRC-wide policy precisely, because as Sam explained:

Originally Posted by IndySam

There is no way that FIRST can allow video reviews of matches. The NFL spends millions of dollars on equipment and gives the head refs many different angles to review a call and they still don;t always get it right. There is just no way for FIRST to implement a fair replay system, period.

Originally Posted by alex334

He refused and told me to relax, because "it's just a game" and the refs are "just volunteers." Well, needless to say, the majority of us are just volunteers. I personally spent two weeks of my college vacation helping my team and countless hours on the bus to see the regional. He told me that we still accomplished the mission of FIRST, since we learned a lot. He then walked away.

I am completely disappointed with this attitude. It is unprofessional. It is disrespectful to everyone doing FIRST and their efforts. I'm still pondering what the lesson learned was for my team. That life isn't fair? Is that such a great message? I personally don't care about winning. I'm done with FIRST. I just want others to get the experience that I got without having such great hopes shattered.

Originally Posted by 2013 Competition Manual

T13
If a TEAM needs clarification on a ruling or score, one (1) pre-college student from that TEAM should address the Head Referee after the ARENA reset signal (i.e. PLAYER STATION LED strings turn green). A TEAM member signals their desire to speak with the Head Referee by standing in the Red or Blue Question Box, which are located on the floor at each end of the scoring table. Depending on timing, the Head Referee may postpone any requested discussion until the end of the subsequent MATCH.

That he even listened to you instead of turning you away, as he easily could have done, is gracious in itself.

And you're wondering what the lesson learned for your team was? The lesson was your build season. You guys pulled yourself together, harnessed your resources, and put together an amazing robot. During both a strategy meeting yesterday and a celebratory dinner tonight, the consensus was that you guys had the most competitive robot there. There wasn't a single person at New York that wasn't impressed by your performance. The way you got there - the little things you did, the changes you went through, the experience you gained - that's what you come away with.

We consider our Lunacy robot, Michael1, to be the best robot we've ever built, even though it didn't go to CMP (which was partly because of abstruse FMS issues specific to our programming that caused us to lose matches) - and the lessons we've gained from Michael's season have driven our success as a team. Virtually everything about how our team operates today - our build criteria and process, the importance we place on driver practice, the effort we put into scouting - is driven by the lessons we learned from Michael's year. And as the cliché goes, those lessons we learned that year - about the importance of simplicity, of implementation, of efficiency - extend not just to FRC but life in general.

That's the kind of thing you walk away with.

[fsracer]

Ouch 694:

"Moreover, individual performance isn't exactly something you can judge quantitatively - e.g. how do you compare offense with defense? 3137 played fantastic defense throughout eliminations and were critical to our wins, but they didn't earn our alliance any actual points."

As the coach of 3171, I saw a consistent 30+ points each match by us. 18 auton, 10 hang and a few during tele when we were not defending. Should we deduct the 30 we did not score from the match totals??

Great alliance! Quick changes in strategy during the elims, called by 694 BTW, were critical to the success of the alliance.

Team connectivity and replays: all teams should be required to demonstrate updated software before going on the field for practice.
BTW have you ever tried to do an emergency fix on a problem with 2 people looking over your shoulder and stamping their feet? Now try it with 65 teams worth of people. Not an easy job for volunteers.

Losing the ability to use cameras during elims: We used 2. One to aim, one to spot disks for pick up. OUCH again.

Referees' calls: I was steamed. It appeared to me that there were at least 3 instances when we were on the pyramid and got whacked by an opposing alliance member. Once while we were attempting to hang (It bent our hanging hooks) No call on any of these instances. When questioned, the reply was: we did not see it. I am also of the opinion that student's efforts should not result in poor payback because of bad calls. BUT
Reality: The refs are human, and volunteers. It will happen. I needed to calm down. (my bad)

On rankings: Good scouting over rules ranking. Our scouting database results were consistent with most of the picks for alliances.

Again, thanks to all teams for their efforts, especially our alliance partners.
And thanks to all the volunteers that make this event a reality!

Team 3137 words of wisdom: There is plenty of time to to it right, but never enough time to do it over(especially during a 2:15 minute match)

Good luck all for the remainder of the season!

fsracer

[jblay]

Originally Posted by fsracer

Ouch 694:

"Moreover, individual performance isn't exactly something you can judge quantitatively - e.g. how do you compare offense with defense? 3137 played fantastic defense throughout eliminations and were critical to our wins, but they didn't earn our alliance any actual points."

As the coach of 3171, I saw a consistent 30+ points each match by us. 18 auton, 10 hang and a few during tele when we were not defending. Should we deduct the 30 we did not score from the match totals??

Not sure exactly what you mean here with the ouch 694? Hope no one was offended by any of the posts by members of our team. I think the meaning here was that you can't rank a team purely on their scoring, because great defense and other factors are very difficult to quantify. 3137 gave us consistent 28 points in auto + climb, but they also robbed our opponents of a lot of points with their stingy defense.

With regards to penalties, learn the rules and have your scouts figure out how they are being called, you can argue till you are blue in the face. If you want your partners not to commit penalties, take the initiative to have them understand the rules and don't rely on them to have great knowledge going in. We have a list of rules and common penalties that we go over with all our partners before every single match no matter the partner, we even went over them with 375 who I believe has won NYC more times than any other team.

With regards to the semifinals, not pictured is 1635 hitting us in the protected zone almost every other time we went to the feeder station, these weren't all called but I'm pretty sure the fouls you are referring to in the disputed match were mostly that and 375 contacting our pyramid and our alliance. Just, because the call was not in your favor or in the stream does not mean it did should not have been called.

With blockading I interpreted the rules and I think the refs did as well, the way that blockading has been called in the past specifically in 2011. Two robots teaming up on one robot to stop them from crossing the field. In my mind this refers to only one lane or in front of the pyramid. 1 robot cannot stop a gap from existing in either of these 3 zones, 2 robots can, that is why the rule exists. 2 robots on opposite ends of the field cannot be evaluated as a blockade especially because that would mean that if one robot is on one side playing defense that means there is no way for their partner to enter the other side of the field because there is now no area they can cross without creating this interpretation of a blockade. Also the idea that if the robot can go under the pyramid or not has an effect on the call is pretty crazy. What if I have a robot that can go under but the bot breaks and is stuck in the above position? This is not only unfair because you have made a design choice around not being able to use this part of the field but is also up to the referee to evaluate if you can under the pyramid or not to see if you are being blockaded. To me the rule is pretty black and white. Two robots sitting in front of their opponent trying to stop them from getting somewhere is a blockade, everything else is just defense.

[bduddy]

Originally Posted by jblay

With blockading I interpreted the rules and I think the refs did as well, the way that blockading has been called in the past specifically in 2011. Two robots teaming up on one robot to stop them from crossing the field. In my mind this refers to only one lane or in front of the pyramid. 1 robot cannot stop a gap from existing in either of these 3 zones, 2 robots can, that is why the rule exists. 2 robots on opposite ends of the field cannot be evaluated as a blockade especially because that would mean that if one robot is on one side playing defense that means there is no way for their partner to enter the other side of the field because there is now no area they can cross without creating this interpretation of a blockade. Also the idea that if the robot can go under the pyramid or not has an effect on the call is pretty crazy. What if I have a robot that can go under but the bot breaks and is stuck in the above position? This is not only unfair because you have made a design choice around not being able to use this part of the field but is also up to the referee to evaluate if you can under the pyramid or not to see if you are being blockaded. To me the rule is pretty black and white. Two robots sitting in front of their opponent trying to stop them from getting somewhere is a blockade, everything else is just defense.

I'm sorry, I think it's a pretty ridiculous stretch for you, a ref, or anyone else to say that essentially any 2v1 defense on a robot constitutes blockad[ing] the FIELD in an attempt to stop the flow of the MATCH. You can't just pull the word "blockading" out of that and treat that as the rule, and "stop[ping] one robot from crossing the field" does not, in my opinion, come close to meeting the actual definition of the rule, nor does anything that happened in that match. Would you honestly look at the rule the same way if it hadn't been your team that had benefited from that call?

[alex334]

Alright, I feel that this discussion is drawing towards it's natural conclusion.
I just wanted to summarize some of the things that I've seen repeated and that we can all agree on.

• Refs work hard and their job is very difficult. Yes, they are just volunteers. We greatly appreciate their work even though it often goes unthanked. However, this does not mean that they are always right. We would like to see more accountability in terms of decision making. Perhaps in the form of explanations of large penalties after every match and a more refined system for appeals.
• The GDC should work towards removing ambiguity in the rules. We understand that this is a difficult task, but a few more examples of blockades, etc. would have greatly benefited the competition. Try to explain rules, especially if they can be left open to interpretation.
• Field communications seem to be getting really wonky. Please look into better solutions than having every team disable their cameras.
• FIRST needs more volunteers. So go help out! I know I will be from now on.

Thank you everyone for your responses. A lot was clarified and many topics were illuminated. It was a great season for 334 and I am tremendously proud of all the students on the team. I hope that the season improves for later competitions in terms of uncontrollable issues and wish you all good luck. I know that I will be sticking around for a long time to come, likely as a volunteer as some of you suggested. 334 will definitely be back with a vengeance as the work for next year has already begun.

Once again. Thank you.

[FrankJ]

Maybe add pay attention at the driver's meeting. This is the place to find out the event ref's interpretation of the rules. Ask well thought out questions there. Listen to the answers.

[kamehameHA]

First, I'd like to thank all the judges, referees, and volunteers that I met and interacted with at the NYC regional. Second, I'd like to voice my concern over the communication errors that occurred during the elimination matches. I understand that having 65 teams continually shuffle through the field is difficult, but forcing teams to turn off their cameras in the finals wouldn't fly at most other regional competitions and definitely not at championships. Losing our Kinect feed completely crippled our ability to shoot.
More frustratingly, our robot lost communication during the teleop periods of our only elimination matches. Although the first couple of elimination matches were replayed, ours was not. Despite talking to some of the field management people, the ruling was never made and we were considered out without taking a single shot.

[AmoryG]

Originally Posted by alex334

Yes, FIRST does a very good job. It got me to where I am today. The volunteers are amazing and you can never thank them enough. Yes, the attitude of FIRST is pretty great. However, that doesn't mean it cannot greatly improve. You should always be seeking to improve. I wanted to call attention to a few things that I noticed that could/might be improved. I wanted to see some explanations for both sides of the argument in all cases. If something is not possible, that's fine, I'll accept that. I just wanted clarifications on a few thoughts.

I definitely agree that FIRST has glaring issues, and that we should work to improve FIRST by eliminating these problems. It just seemed to me that when you said you lost a lot of respect and that you were done with FIRST, you were giving it up because of these existing problems.

Originally Posted by alex334

The thing for me with powerhouse teams is that they inspire others. I always drew inspiration from 254. I always wanted to and still do want to become more like them. If they shine in competition and seed high, that gives teams a clear goal. But if teams like that are not in the top spots, what message does that send?

To me it means that the elite sometimes struggle and the lesser teams sometimes excel. It says nothing other than there's parity within FIRST that often doesn't exist in other competitions. Parity means teams have to work hard to remain the best, and with hard work you can become the best. I actually think that's a pretty good message to send.

[Jeanne Boyarsky]

I wrote this before reading the replies. I wholeheartedly agree with Racer26. Sam (as senior on 694) wrote an excellent post as well. Great job Sam!

~~~~~~~~~~~~~~~~~~~~~~~~~~

On the "ranking system", I thought the "picking your alliance" was supposed to compensate for that. Personally, I refer to it as a seeding system rather than a ranking system. It's not a simple ordered ranking but a guide. Like in tennis. The #1 seed isn't necessarily the best - he/she is the one who played/won the matches. Many people realized #334 was the best. I didn't see the match you lost so I don't know if it was bad luck or a jam or what. But you were clearly a top robot and the #1 seed certainly recognized that.

We were on the opposing alliance of the one that got 120+ points in penalties. (I think it was more because when they asked for a re-count on their missing 10 points, our score went up about 40 points.) It was early in the day so I'm guessing the penalty calling was overly aggressive. That said, pushing a robot while it is under it's own pyramid isn't a good strategy. Too much risk of pushing the opponent into the pyramid. (The high penalty match was #4 - can't find it on ustream for a link)

"not everyone reads the rules" - Sorry, but this is a cop out. We were all given the rules. We are all supposed to read them. If a team is out of dimensions, it doesn't pass inspection. Yes, it is unfortunate that the alliance partners get assessed the penalties too. All I can see, this increases the need for the experienced teams (like both of ours) to *talk* to our alliance partners to make sure they know the rules. Particularly the ones relating to the match strategy. Or any of the 3 coaches to notice the penalties and tell the driver to back off.

I'm sorry you feel "done with FIRST" after your team had such a great day. Your team fielded one of the best robots there if not the best. I hope they learned a lot. They inspired others. That is what FIRST is about. And mentoring for the matter. Granted, I'm thrilled we won - but that wasn't the most important thing to me that happened yesterday.

[Andrew Schreiber]

Originally Posted by alex334

If they shine in competition and seed high, that gives teams a clear goal. But if teams like that are not in the top spots, what message does that send?

The rankings are defined just like the robot rules. To shine in the competition implies ranking high. If teams do not rank high it says they are not excelling. Play the game as it is written NOT how you think it should be.

[Mr. Van]

A few thoughts:

If you have real issues with how events are refereed, the most direct solution is to volunteer as a referee. Be the new standard that all other refs will look to.

Just about all of us have been on the bad end of bad calls by referees in FIRST. It is also very likely that we have been on the "good" end of bad calls by referees (a glancing blow to a pyramid that wasn't noticed, etc.). Sometimes it seems things don't go your way so much that you develop a reputation. Believe me, I know what it is like to have "that talk" with a team all too well...
It is difficult, but it is what really makes a difference - a real difference.

As for the ranking system, the problem isn't so much that really good teams don't rank high (this is really rare), but the the alliance picking system places many, many teams in a "too good for a third pick" category. In many (most?) regionals now, the battle for the championship is between a handful of elite powerhouse teams who pair up and then look for a solid third pick. With all respect to those third picks, they are (because of the system) not generally among the top robots at the event. This places the robots in the 7th through 20th or so seedings (generally speaking) in a place where they almost never have a chance to form an alliance with a really competitive dominant team. There is a much better chance that a team in the lower ranks will be selected to round out those dominant alliances.

With the chance to participate in the Championship on the line, sometimes I ask "is this fair?" The clear answer is no. But then I realize that I've been drawn into the trap of thinking that this is about robots and scores and winning.

Reboot. Remember why we do what we do.

- Mr. Van
Coach, Robodox