Win-Loss, Average, RP?

[Knufire]

Originally Posted by Caleb Sykes

I really have trouble imagining any ranking system that isn't WLT which won't cause qualification gameplay to differ from playoff gameplay. Take this year as an example, the ranking system discouraged many teams from playing defense in the quals matches where they would have if we had a WLT ranking system or if they were playing in the playoffs.

I think i see where you're coming from. Generally speaking, any ranking system that factors in performance relative to a standard outside the scope of that single match (i.e. bonus RP for secondary objectives, tournament average, etc) leaves room for teams in a given match to collude in order to maximize their own gains. I think this is a necessary evil, as purely looking at each match in a vacuum (which is basically what WLT does) leaves you more at the mercy of the schedule gods. I don't think this really happens enough in FRC to be a major concern, and with the 2016 manual, is explicitly discouraged by FiRST.

The differences between qualifications and playoff rounds that concern me were more related to game design, such as tasks being worth RP in quals and points in eliminations (2016), disappear completely in playoffs (co-op 2015, co-op 2012), or don't exist in the qualification rounds but are important to playoff rounds (triple-balancing 2012). These are almost all related to game design and not the ranking system. IMO, make everything worth points and let the ranking system do the work of filtering teams into rank.

[Chris is me]

I think the FIRST GDC has tried for a long time to figure out a way to make secondary objectives influence ranking, without much success. But in 2016 they finally figured out a good solution that added a lot of strategic depth and value to the game. I particularly like how only one of the two extra RP required full cooperation of the entire alliance, and how the RPs provided extra incentive for correct strategic play in terms of defense crossings. It was very well done, and I hope that if they do "WLT Plus Something Else" in the future they do it like 2016 and not 2012.

Would be really interesting for someone to calculate what the rankings would be with straight WLT, and elimination rules for breach / capture (so extra points added to final score). I think they would be a bit less accurate, as "WLT Plus Something Else" games tend to allow those that prioritize a top seed to seed higher simply by knowing the rules and playing correctly.

[Caleb Sykes]

2012 would have been hands down my favorite game if the quals rules had been the same as the playoff rules. I agree that 2016 was the best implementation so far of the WLT+something system, but I fear that good games will be worse off in the future due to poor implementations like 2012's. I'm willing to let the GDC play around more with it though, I just hope they recognize why 2016 was so much better than 2012.

[TDav540]

Originally Posted by Knufire

I think i see where you're coming from. Generally speaking, any ranking system that factors in performance relative to a standard outside the scope of that single match (i.e. bonus RP for secondary objectives, tournament average, etc) leaves room for teams in a given match to collude in order to maximize their own gains. I think this is a necessary evil, as purely looking at each match in a vacuum (which is basically what WLT does) leaves you more at the mercy of the schedule gods. I don't think this really happens enough in FRC to be a major concern, and with the 2016 manual, is explicitly discouraged by FiRST.

The differences between qualifications and playoff rounds that concern me were more related to game design, such as tasks being worth RP in quals and points in eliminations (2016), disappear completely in playoffs (co-op 2015, co-op 2012), or don't exist in the qualification rounds but are important to playoff rounds (triple-balancing 2012). These are almost all related to game design and not the ranking system. IMO, make everything worth points and let the ranking system do the work of filtering teams into rank.

I thought the thing that 2016 did best was making sure that factors which influenced ranking (breaches and captures) converted to something of value during the playoffs, unlike in 2015 and 2012. However, I also agree that the 2015 version did the best work to separate the best teams, but I think using that exact system in any game that requires defense would, by virtue of team intelligence, result in a no defense game, at least until the playoffs. However, a system similar to 2015, but using scoring differential, might be a better way to separate teams.

A danger could be that you could see teams try to run up the score, but I think that's much more difficult to do in a 150 second match than a 60 minute football/basketball/hockey/etc game.

[Caleb Sykes]

Originally Posted by Chris is me

Would be really interesting for someone to calculate what the rankings would be with straight WLT, and elimination rules for breach / capture (so extra points added to final score). I think they would be a bit less accurate, as "WLT Plus Something Else" games tend to allow those that prioritize a top seed to seed higher simply by knowing the rules and playing correctly.

I missed this part:

and elimination rules for breach / capture (so extra points added to final score)

I'll go back and redo it later, here are my results without modifying scores.

Rank using the 2016 system had about 80% correlation with OPR and a WLT system in 2016 would have only had about 73% correlation with OPR.

However, it should be noted that, if the ranking system were WLT, teams would have behaved differently, and it is probably safe to assume that the correlation of WLT with OPR would have increased. It is questionable though if it would have increased to 80+%.

I have attached a bunch of graphs comparing different things at Palmetto, comparing ranking score against OPR value produced the strongest correlation, so I took that comparison and applied it to 8 other events. Those results are also attached.