[beyondinspection] 2015 Ranking System

[jwfoss]

New Year = New Game = New Content
The folks behind beyondinspection would like to wish everyone a happy new year and best of luck in the construction of their robots for the 2015 FRC Game: Recycle Rush. As we head into competition season we would love to have event coverage all over the globe, so if you are interested let us know. Some of our fellow mentors have taken the time to write up articles about the new ranking system and they are ready for your reading pleasure.

[SA] 2015 Ranking: QA vs QS
[SA] 2015 Ranking: Visualization

Special thanks to our two contributors this week: Andrew Schreiber and Scott Meredith

As always let us know what you think and what other content you would like to see developed (or content you would like to provide).

[Andrew Schreiber]

Originally Posted by jwfoss

New Year = New Game = New Content
The folks behind beyondinspection would like to wish everyone a happy new year and best of luck in the construction of their robots for the 2015 FRC Game: Recycle Rush. As we head into competition season we would love to have event coverage all over the globe, so if you are interested let us know. Some of our fellow mentors have taken the time to write up articles about the new ranking system and they are ready for your reading pleasure.

[SA] 2015 Ranking: QA vs QS
[SA] 2015 Ranking: Visualization

Special thanks to our two contributors this week: Andrew Schreiber and Scott Meredith

As always let us know what you think and what other content you would like to see developed (or content you would like to provide).

I'm actually hoping some folks will weigh in on the limitations of that simulation. Mostly that I don't account for teams who are genuinely worth negative points.

If there's interest I might do another version that handles ranking based on the 2014 WLT method to show the difference.

[Aren_Hill]

This is an awesome analysis, thanks for posting these.
I had a hunch this is how things would work out, I just never ran any numbers.

-Aren

[dellagd]

I was always curious about doing a "schedule strength" calculation. Cool to see it actually done out, and very interesting. Awesome!

[Caleb Sykes]

Thanks for doing this, these are very interesting.

Originally Posted by Andrew Schreiber

I'm actually hoping some folks will weigh in on the limitations of that simulation. Mostly that I don't account for teams who are genuinely worth negative points.

I've also been wondering recently how many teams will "hurt" their alliances this year. I imagine that there might be more teams with negative OPRs this year than we have seen in years past. The fact that penalties now subtract from your alliance's score instead of adding to the opposing alliance's score will lower OPR values overall, but it seems that there will be less penalties this year than in 2014. Additionally, there is the inevitable fact that stacks will get knocked over by robots.

According to the 2834 scouting database, last year 151/2696 teams (roughly 6%) had a max OPR that was negative. In 2013, 249/2509 teams (10%) had a max OPR that was negative. Interesting that there were more negative teams in 2013 than in 2014, I might have guessed that it would be the other way around.

I'm not quite sure how to use this data for a reasonable estimate, but it seems unlikely to me that the percentage for this year will be less than the percentage from 2013, since penalties were not subtractive in 2013 and it was difficult to de-score any points that year*. However, robots that could not do much more than drive that year were often assigned to be defenders, which will not be an option this year, raising OPRs all around. My guess is that the percentage of negative OPRs this year will be a bit higher than 10%.

*Although I do recall a few robots inadvertently knocking their own partners off of pyramids.

[SoftwareBug2.0]

I'm glad to see that this year's ranking system appears to correlate with a traditional measure of robot performance. I've had different opinions on FIRST's different tie breakers over the years but never thought enough about it to quantify them. If we wanted to take this really seriously we might be able to come up with some better metrics than correlation coefficient with OPR.

So, let's take this way too seriously: First, the final output is ordinal rather than continous data, so we might want to correlate the rankings between systems rather than the raw numbers. Second, only a subset of the results actually matter to the tournament so we might want to consider only that portion of the results. In particular, the rankings determine who can be which alliance captain so your team being ranked 1 vs 8 matters but being ranked 17th vs 30th means nothing.

We might also want to consider how a ranking effects the tournament. For example, consider the following two cases:

1) The best robot is ranked last, but all other are in order
2) The worst robot is ranked first, but all others are in order

In the first case, the robot that should have been ranked first will most likely be picked early and the alliances will look sort of like what they would have if the rankings had been perfect.

In the second case, I think the results would be much more severe. First, it guarantees a team a slot in the eliminations that shouldn't be there. Second, it's likely to change all of the aliance pairings because there will be a bunch of declines. This reduces the value of being the 2nd or 3rd alliance captain and some team ranked just out of the top 8 will be hosed.

I wonder what you'd get for something like:
1) Determine a rank based on the OPR from that event
2) Determine a rank by desired metric (QS, QA, etc.)
3) For each of the top 10 (or so) positions, if the OPR rank is lower than the rank by the desired metric then add that how much it is lower by.
4) Compare totals & the metric with the lower total is better

For example, if you had something like this:

QS rank|OPR rank
1      |3      
2      |5      
3      |2      
4      |1      
5      |12
6      |8
7      |4
8      |11
9      |6
10     |9

Then you'd get: 2+3+0+0+7+2+0+3+0+0=17 for QS

Anyway, this is pretty ad-hoc. I'm sure there's a nicer way to do this.

[thatprogrammer]

[quote=SoftwareBug2.0;1429519]

In the first case, the robot that should have been ranked first will most likely be picked early and the alliances will look sort of like what they would have if the rankings had been perfect.

In the second case, I think the results would be much more severe. First, it guarantees a team a slot in the eliminations that shouldn't be there. Second, it's likely to change all of the aliance pairings because there will be a bunch of declines. This reduces the value of being the 2nd or 3rd alliance captain and some team ranked just out of the top 8 will be hosed.

/QUOTE]

The multiple denials would lead to more teams outside of the top eight being picked, The teams just outside of the top right might even get to play with an amazing team (other than the number 1, who wouldn't be very good). The overall strength of alliances would still be fairly high, excluding alliance 1. Still, that one team outside of the top eight that gets selected by the number one would be unfortunate.

[IKE]

I would recommend using 2008 OPR distributions to see what negative penalties could do to the game. It was a game that tracked well to OPR and there were a fair amount of penalties.

Also, how well does your curve shape match the OPR curve shapes for 2013, 2012 (modified one), 2010, and 2008?

[Andrew Schreiber]

Originally Posted by IKE

I would recommend using 2008 OPR distributions to see what negative penalties could do to the game. It was a game that tracked well to OPR and there were a fair amount of penalties.

Also, how well does your curve shape match the OPR curve shapes for 2013, 2012 (modified one), 2010, and 2008?

Dunno, never graphed it/compared to OPR. I'll see if I can drag out an OPR curve for 2008 and use it to generate skill levels.


I recognized that the only output that matters is the final ranking, there's a whole analysis section that only deals with it. I just felt it was interesting to watch how teams moved through rankings as matches progressed. Specifically, how the rankings eventually reached a stable point for teams in the top/bottom of them. The middle needs more matches to settle out since they tend to be closer in skill. I was looking into adding a Average Error value in (summing abs(actualRank - expectedRank) for each team and divide by number of teams)
but I just didn't get around to it before this went live (something something build season)


I supposed we could take this model even further and simulate picks (assume each team picks the best available robot, and some sort of metric for declines) then we could play out elims to see what teams end up "qualifying" from the event. Since, really, for Regionals/CMP Divisions/CMP the only output that matters is the winning alliance. But I question the value of this since it is much more driven by team's ability to pick an alliance than by FIRST's rules (at least this year, other years are another story).

[Ty Tremblay]

Despite the fact that QA is a better representation of the higher scoring (often better) robots this year. An important thing to remember during alliance selection is to build an ALLIANCE that can score as many points as possible. Make sure your alliance consists of 3 robots that will work well together.

[Andrew Schreiber]

Originally Posted by Ty Tremblay

Despite the fact that QA is a better representation of the higher scoring (often better) robots this year. An important thing to remember during alliance selection is to build an ALLIANCE that can score as many points as possible. Make sure your alliance consists of 3 robots that will work well together.

Absolutely, I don't think anyone is saying anything different. Scott and I were mostly trying to analyze how good the new ranking system was. Since there have been years when it was possible to completely screw good teams over.