Win-Loss, Average, RP?

[Rangel(kf7fdb)]

I really liked 2016 in that it rewarded an alliance for performing very well regardless if they won or lost but still made wins and losses important to the rankings. Though it might be an unpopular opinion, I absolutely hate when coopertition is a very important part of the ranking system. I think extra ranking for having good teamwork within your alliance should be rewarded vs teamwork with the opposing alliance(2016 vs 2012). Despite co-op trying to promote positive interactions with opponents, I feel as if it does the exact opposite. Especially when one alliance doesn't want to co-op for legitimate competitive reasons.

[EricH]

Originally Posted by Rangel(kf7fdb)

I really liked 2016 in that it rewarded an alliance for performing very well regardless if they won or lost but still made wins and losses important to the rankings. Though it might be an unpopular opinion, I absolutely hate when coopertition is a very important part of the ranking system. I think extra ranking for having good teamwork within your alliance should be rewarded vs teamwork with the opposing alliance(2016 vs 2012). Despite co-op trying to promote positive interactions with opponents, I feel as if it does the exact opposite. Especially when one alliance doesn't want to co-op for legitimate competitive reasons.

THIS.

2016 allowed you to get a "win" in rankings if you played well, regardless of whether you won or lost. 2012 required you to work with your opponent, and is the reason that every time I discuss the "rankings game", I make very clear that if a team makes an agreement with their opponents to do something, they need to KEEP it. There was quite a bit of "bad blood" that year, whether intentional or not I can't say. 2015 was OK... but required you to design to do it, and work with your opponent.

[asid61]

2016 was probably the best system IMO because it balanced multiple tasks, and the tasks that would net you RPs would also net a considerable number of points in eliminations, keeping you competitive (unlike 2015 as Caleb mentioned). The dynamic feel of the game also helped to keep Elims exciting even if the teams were not evenly matched.

[Knufire]

Note that the 2016 ranking system was very much tied to the 2016 game. If you want to apply the same ranking system to future games, you need a game with several different major activities. Applying the same concept to a similar game, such as 2014, might prove difficult (RP for assists, trusses, and winning?).

These are what I look for in a ranking system:
1. Produces rankings that correlate closely to individual robot ability.
2. Simple enough to explain to a general audience who may not be invested in FRC.
3. Universally applicable to different game types, such that FIRST could just use the same ranking system year after year.


Given these criteria, I am a very big fan of the 2015 ranking system. 2016 was definitely fun to participate in but not easy to explain to a more general audience.

I also agree with Caleb on "changing the game" between qualifications and eliminations; however, I see this as a game design issue and not a ranking system issue.

[Peyton Yeung]

And here I am still liking the 2010 system

[Caleb Sykes]

Originally Posted by Knufire

I also agree with Caleb on "changing the game" between qualifications and eliminations; however, I see this as a game design issue and not a ranking system issue.

I really have trouble imagining any ranking system that isn't WLT which won't cause qualification gameplay to differ from playoff gameplay. Take this year as an example, the ranking system discouraged many teams from playing defense in the quals matches where they would have if we had a WLT ranking system or if they were playing in the playoffs.

[Taylor]

2016>2012>2010>everything else>2015.

[Knufire]

Originally Posted by Caleb Sykes

I really have trouble imagining any ranking system that isn't WLT which won't cause qualification gameplay to differ from playoff gameplay. Take this year as an example, the ranking system discouraged many teams from playing defense in the quals matches where they would have if we had a WLT ranking system or if they were playing in the playoffs.

I think i see where you're coming from. Generally speaking, any ranking system that factors in performance relative to a standard outside the scope of that single match (i.e. bonus RP for secondary objectives, tournament average, etc) leaves room for teams in a given match to collude in order to maximize their own gains. I think this is a necessary evil, as purely looking at each match in a vacuum (which is basically what WLT does) leaves you more at the mercy of the schedule gods. I don't think this really happens enough in FRC to be a major concern, and with the 2016 manual, is explicitly discouraged by FiRST.

The differences between qualifications and playoff rounds that concern me were more related to game design, such as tasks being worth RP in quals and points in eliminations (2016), disappear completely in playoffs (co-op 2015, co-op 2012), or don't exist in the qualification rounds but are important to playoff rounds (triple-balancing 2012). These are almost all related to game design and not the ranking system. IMO, make everything worth points and let the ranking system do the work of filtering teams into rank.

[Chris is me]

I think the FIRST GDC has tried for a long time to figure out a way to make secondary objectives influence ranking, without much success. But in 2016 they finally figured out a good solution that added a lot of strategic depth and value to the game. I particularly like how only one of the two extra RP required full cooperation of the entire alliance, and how the RPs provided extra incentive for correct strategic play in terms of defense crossings. It was very well done, and I hope that if they do "WLT Plus Something Else" in the future they do it like 2016 and not 2012.

Would be really interesting for someone to calculate what the rankings would be with straight WLT, and elimination rules for breach / capture (so extra points added to final score). I think they would be a bit less accurate, as "WLT Plus Something Else" games tend to allow those that prioritize a top seed to seed higher simply by knowing the rules and playing correctly.

[Caleb Sykes]

2012 would have been hands down my favorite game if the quals rules had been the same as the playoff rules. I agree that 2016 was the best implementation so far of the WLT+something system, but I fear that good games will be worse off in the future due to poor implementations like 2012's. I'm willing to let the GDC play around more with it though, I just hope they recognize why 2016 was so much better than 2012.

[Caleb Sykes]

Originally Posted by Chris is me

Would be really interesting for someone to calculate what the rankings would be with straight WLT, and elimination rules for breach / capture (so extra points added to final score). I think they would be a bit less accurate, as "WLT Plus Something Else" games tend to allow those that prioritize a top seed to seed higher simply by knowing the rules and playing correctly.

I missed this part:

and elimination rules for breach / capture (so extra points added to final score)

I'll go back and redo it later, here are my results without modifying scores.

Rank using the 2016 system had about 80% correlation with OPR and a WLT system in 2016 would have only had about 73% correlation with OPR.

However, it should be noted that, if the ranking system were WLT, teams would have behaved differently, and it is probably safe to assume that the correlation of WLT with OPR would have increased. It is questionable though if it would have increased to 80+%.

I have attached a bunch of graphs comparing different things at Palmetto, comparing ranking score against OPR value produced the strongest correlation, so I took that comparison and applied it to 8 other events. Those results are also attached.

[TDav540]

Originally Posted by Knufire

I think i see where you're coming from. Generally speaking, any ranking system that factors in performance relative to a standard outside the scope of that single match (i.e. bonus RP for secondary objectives, tournament average, etc) leaves room for teams in a given match to collude in order to maximize their own gains. I think this is a necessary evil, as purely looking at each match in a vacuum (which is basically what WLT does) leaves you more at the mercy of the schedule gods. I don't think this really happens enough in FRC to be a major concern, and with the 2016 manual, is explicitly discouraged by FiRST.

The differences between qualifications and playoff rounds that concern me were more related to game design, such as tasks being worth RP in quals and points in eliminations (2016), disappear completely in playoffs (co-op 2015, co-op 2012), or don't exist in the qualification rounds but are important to playoff rounds (triple-balancing 2012). These are almost all related to game design and not the ranking system. IMO, make everything worth points and let the ranking system do the work of filtering teams into rank.

I thought the thing that 2016 did best was making sure that factors which influenced ranking (breaches and captures) converted to something of value during the playoffs, unlike in 2015 and 2012. However, I also agree that the 2015 version did the best work to separate the best teams, but I think using that exact system in any game that requires defense would, by virtue of team intelligence, result in a no defense game, at least until the playoffs. However, a system similar to 2015, but using scoring differential, might be a better way to separate teams.

A danger could be that you could see teams try to run up the score, but I think that's much more difficult to do in a 150 second match than a 60 minute football/basketball/hockey/etc game.