Ranking by Median Score

[Kevin Leonard]

Let me start off by saying that this year's ranking system is generally one of the best we've had in a while at making sure the top teams are at the top. However this past weekend, my team was ranked in the top 8, then had two bad freak matches, and ended up near the 20th seed due to those two matches being factored into our average score.

I think an even better representation of robot ability would be the median score. This would cause rankings to still decrease and increase when your team has a good or bad match, but punishes teams much less for some freak accident (like a programming issue that caused the forks on our elevator to oscillate open and closed for the entire match.)

What does CD think? I can see a few problems with the game becoming fundamentally different, but I don't think an argument like that holds a lot of weight.

I'd also love to see one of you statistical wizards showing how that would have affected rankings at some events.
Thanks!

[Nathan Streeter]

Seems that something like your suggestion would be (even) better than the current ranking system.

My iterative suggestion is to perhaps average the middle half of your matches... I think this would probably be even more accurate to teams' capabilities throughout the rankings. My suspicion is that this year's rankings work phenomenally well in events where:

- # of matches to # of teams ratio is high
- Most teams aren't improving dramatically
- No teams have 'outlier-like' performance

but probably become weaker, particularly in the middle and lower ranks, when there are more teams, fewer matches, and 1 or 2 powerhouses. Since a powerhouse can put up so many points, you'll see teams that would have a QA of 50 get bumped up to 60, just by playing a single match with a team that puts up 150pts. Particularly in the upper-middle section of the rankings, having a terrible match can drop you like a stone... this would also assist with that scenario.

So my recommendation is to take the middle 50% of your match scores.... average all those together. This nearly eliminates the impact of having 1 or 2 fiercely unlucky (or lucky) matches, but will still reward the 'consistent performers' who are just very good every time they go out there.

[Andrew Schreiber]

Originally Posted by Nathan Streeter

Seems that something like your suggestion would be (even) better than the current ranking system.

My iterative suggestion is to perhaps average the middle half of your matches... I think this would probably be even more accurate to teams' capabilities throughout the rankings. My suspicion is that this year's rankings work phenomenally well in events where:

- # of matches to # of teams ratio is high
- Most teams aren't improving dramatically
- No teams have 'outlier-like' performance

but probably become weaker, particularly in the middle and lower ranks, when there are more teams, fewer matches, and 1 or 2 powerhouses. Since a powerhouse can put up so many points, you'll see teams that would have a QA of 50 get bumped up to 60, just by playing a single match with a team that puts up 150pts. Particularly in the upper-middle section of the rankings, having a terrible match can drop you like a stone... this would also assist with that scenario.

So my recommendation is to take the middle 50% of your match scores.... average all those together. This nearly eliminates the impact of having 1 or 2 fiercely unlucky (or lucky) matches, but will still reward the 'consistent performers' who are just very good every time they go out there.

For anyone who would like to try this out, I've got a very simple sim of the ranking system (demo http://schreiaj.github.io/2015_ranki...s=64&matches=8 and code https://github.com/schreiaj/2015_ranking_sim) To implement these changes you'd simply have to change https://github.com/schreiaj/2015_ran...ain.coffee#L30

Rebuilding this to easily swap out the ranking system has been a todo item for a while.

[dougwilliams]

I'm generally please withe the averages approach this year, with one huge caveat...the number of matches vs the number of teams at an event.

At VA we only had 8 matches, and 64 teams, and I don't feel that was enough to adequately sort. The sort was starting to make sense, but I don't think it had enough to truly reflect what was going on; and in that sense failed the teams some.

I haven't looked at medians and how that progressed, but I'll poke around and see what my feelings are on that. My gut feel is mean is better than median.

[Citrus Dad]

Originally Posted by Kevin Leonard

Let me start off by saying that this year's ranking system is generally one of the best we've had in a while at making sure the top teams are at the top. However this past weekend, my team was ranked in the top 8, then had two bad freak matches, and ended up near the 20th seed due to those two matches being factored into our average score.

I think an even better representation of robot ability would be the median score. This would cause rankings to still decrease and increase when your team has a good or bad match, but punishes teams much less for some freak accident (like a programming issue that caused the forks on our elevator to oscillate open and closed for the entire match.)

What does CD think? I can see a few problems with the game becoming fundamentally different, but I don't think an argument like that holds a lot of weight.

I'd also love to see one of you statistical wizards showing how that would have affected rankings at some events.
Thanks!

I was thinking of going the other way to solve this problem. In some ways this year's game is too stable in the rankings. Once a team gets to the second day, they aren't going to change their ranking much. It doesn't really allow for a late charge to the top which can make this more interesting.

So instead the ranking could be based on the best 5 performances (or 50%). A bad match will be thrown out, but a good match can move a team up quite a bit. The rankings could be quite reshuffled the second day.

[Kevin Leonard]

Originally Posted by Citrus Dad

I was thinking of going the other way to solve this problem. In some ways this year's game is too stable in the rankings. Once a team gets to the second day, they aren't going to change their ranking much. It doesn't really allow for a late charge to the top which can make this more interesting.

So instead the ranking could be based on the best 5 performances (or 50%). A bad match will be thrown out, but a good match can move a team up quite a bit. The rankings could be quite reshuffled the second day.

So your idea is to make rankings less indicative of robot ability.

I like median because it removed outliers from having either really good partners or having some freak accident.