Win-Loss, Average, RP?
 

Just wondering, since we've seen there different scoring systems in the last three games, what seems like the best system that rewards commitment, yet still encourages underfunded teams?

Re: Win-Loss, Average, RP?
 

I really loved the 2016 ranking system. It made it such that even if your alliance was over matched, you still had goals to try to achieve during a match. It also created a ton of interesting strategic choices about how to play each match.

Re: Win-Loss, Average, RP?
 

I agree that 2016 was a great ranking system in that you always had a way to keep your ranking from completely tanking, but still needed to win matches to rank the highest. It took away defense from Quals more than it would've been otherwise, but it was for strategic benefit. It created a slight difference from Quals to playoffs which I enjoyed.

Re: Win-Loss, Average, RP?
 

Statistically speaking, I think 2015 was the most accurate. No matter what ranking system you have, having a good schedule can skew your rank but having the averages were great for ranking because no defense means no low score matches due to the other alliance (except at worlds where can burglars decided the whole match). Now nobody wants to really play with no defense again, but I think that the 2015 ranking system is the most accurate for most non-defense games in the future (please, don't do another non-defense game...).

Re: Win-Loss, Average, RP?
 

I actually started going through regionals/championship divisions in 2015 to see how teams would have been ranked if they had used the 2014 ranking system. I'll post a link once I'm done with the spreadsheet.

Re: Win-Loss, Average, RP?
 

Rank in 2015 was much more highly correlated with ability than in any other year, and I value this correlation pretty highly with any ranking system.

My problem with systems like those of 2012, 2015 or 2016 is that the game becomes fundamentally different in the playoff matches from what it was in qualification matches. I don't think it is fair that abilities which are important to ranking high (see, making coopertition stacks) will not necessarily be as useful in the playoffs, and that an ability that is an invaluable asset in the playoffs (see, fast canburglars) can be essentially useless during qualifications. This forces teams to make a choice, either explicitly or implicitly, between focusing on seeding high or contributing to the maximum in the playoffs. That is just silly, other sports' playoff matches are essentially the same as qualification matches, just with better teams, let's have that again please, like we did in 2013 and 2014.

Re: Win-Loss, Average, RP?
 

2016 was the best system to play under. The three methods of getting RP made the game more interesting and always gave an alliance a reason to try.

2010 produced the most accurate results, but I think the issues many had with the ranking system probably mean we'll never see anything like it again. I personally thought it was great as long as teams were willing to score in the proper goal...

Re: Win-Loss, Average, RP?
 

I really liked 2016 in that it rewarded an alliance for performing very well regardless if they won or lost but still made wins and losses important to the rankings. Though it might be an unpopular opinion, I absolutely hate when coopertition is a very important part of the ranking system. I think extra ranking for having good teamwork within your alliance should be rewarded vs teamwork with the opposing alliance(2016 vs 2012). Despite co-op trying to promote positive interactions with opponents, I feel as if it does the exact opposite. Especially when one alliance doesn't want to co-op for legitimate competitive reasons.

Re: Win-Loss, Average, RP?
 

THIS.

2016 allowed you to get a "win" in rankings if you played well, regardless of whether you won or lost. 2012 required you to work with your opponent, and is the reason that every time I discuss the "rankings game", I make very clear that if a team makes an agreement with their opponents to do something, they need to KEEP it. There was quite a bit of "bad blood" that year, whether intentional or not I can't say. 2015 was OK... but required you to design to do it, and work with your opponent.

Re: Win-Loss, Average, RP?
 

2016 was probably the best system IMO because it balanced multiple tasks, and the tasks that would net you RPs would also net a considerable number of points in eliminations, keeping you competitive (unlike 2015 as Caleb mentioned). The dynamic feel of the game also helped to keep Elims exciting even if the teams were not evenly matched.

Re: Win-Loss, Average, RP?
 

Note that the 2016 ranking system was very much tied to the 2016 game. If you want to apply the same ranking system to future games, you need a game with several different major activities. Applying the same concept to a similar game, such as 2014, might prove difficult (RP for assists, trusses, and winning?).

These are what I look for in a ranking system:
1. Produces rankings that correlate closely to individual robot ability.
2. Simple enough to explain to a general audience who may not be invested in FRC.
3. Universally applicable to different game types, such that FIRST could just use the same ranking system year after year.


Given these criteria, I am a very big fan of the 2015 ranking system. 2016 was definitely fun to participate in but not easy to explain to a more general audience.

I also agree with Caleb on "changing the game" between qualifications and eliminations; however, I see this as a game design issue and not a ranking system issue.

Re: Win-Loss, Average, RP?
 

And here I am still liking the 2010 system :D

Re: Win-Loss, Average, RP?
 

To tack onto this bit, the thing that I really hated about 2015 was if you had one match where you browned out, your rank would fall like a stone. 2016 didn't punish you as much for that.

Re: Win-Loss, Average, RP?
 

I really have trouble imagining any ranking system that isn't WLT which won't cause qualification gameplay to differ from playoff gameplay. Take this year as an example, the ranking system discouraged many teams from playing defense in the quals matches where they would have if we had a WLT ranking system or if they were playing in the playoffs.

Re: Win-Loss, Average, RP?
 

2016>2012>2010>everything else>2015.

Re: Win-Loss, Average, RP?
 

I think i see where you're coming from. Generally speaking, any ranking system that factors in performance relative to a standard outside the scope of that single match (i.e. bonus RP for secondary objectives, tournament average, etc) leaves room for teams in a given match to collude in order to maximize their own gains. I think this is a necessary evil, as purely looking at each match in a vacuum (which is basically what WLT does) leaves you more at the mercy of the schedule gods. I don't think this really happens enough in FRC to be a major concern, and with the 2016 manual, is explicitly discouraged by FiRST.

The differences between qualifications and playoff rounds that concern me were more related to game design, such as tasks being worth RP in quals and points in eliminations (2016), disappear completely in playoffs (co-op 2015, co-op 2012), or don't exist in the qualification rounds but are important to playoff rounds (triple-balancing 2012). These are almost all related to game design and not the ranking system. IMO, make everything worth points and let the ranking system do the work of filtering teams into rank.

Re: Win-Loss, Average, RP?
 

I think the FIRST GDC has tried for a long time to figure out a way to make secondary objectives influence ranking, without much success. But in 2016 they finally figured out a good solution that added a lot of strategic depth and value to the game. I particularly like how only one of the two extra RP required full cooperation of the entire alliance, and how the RPs provided extra incentive for correct strategic play in terms of defense crossings. It was very well done, and I hope that if they do "WLT Plus Something Else" in the future they do it like 2016 and not 2012.

Would be really interesting for someone to calculate what the rankings would be with straight WLT, and elimination rules for breach / capture (so extra points added to final score). I think they would be a bit less accurate, as "WLT Plus Something Else" games tend to allow those that prioritize a top seed to seed higher simply by knowing the rules and playing correctly.

Re: Win-Loss, Average, RP?
 

2012 would have been hands down my favorite game if the quals rules had been the same as the playoff rules. I agree that 2016 was the best implementation so far of the WLT+something system, but I fear that good games will be worse off in the future due to poor implementations like 2012's. I'm willing to let the GDC play around more with it though, I just hope they recognize why 2016 was so much better than 2012.

Re: Win-Loss, Average, RP?
 

I thought the thing that 2016 did best was making sure that factors which influenced ranking (breaches and captures) converted to something of value during the playoffs, unlike in 2015 and 2012. However, I also agree that the 2015 version did the best work to separate the best teams, but I think using that exact system in any game that requires defense would, by virtue of team intelligence, result in a no defense game, at least until the playoffs. However, a system similar to 2015, but using scoring differential, might be a better way to separate teams.

A danger could be that you could see teams try to run up the score, but I think that's much more difficult to do in a 150 second match than a 60 minute football/basketball/hockey/etc game.

Re: Win-Loss, Average, RP?
 

I missed this part:

I'll go back and redo it later, here are my results without modifying scores.

Rank using the 2016 system had about 80% correlation with OPR and a WLT system in 2016 would have only had about 73% correlation with OPR.

However, it should be noted that, if the ranking system were WLT, teams would have behaved differently, and it is probably safe to assume that the correlation of WLT with OPR would have increased. It is questionable though if it would have increased to 80+%.

I have attached a bunch of graphs comparing different things at Palmetto, comparing ranking score against OPR value produced the strongest correlation, so I took that comparison and applied it to 8 other events. Those results are also attached.