How many matches are really needed to determine final rankings?

[tctc]

It's funny how it works out that the top teams tend to end up in the top 8 spots by the end of qualifications. Maybe magical. Early in my involvement with FIRST someone said that there is a mathematical theory behind it. So, I assumed teams change a lot in rank early in the competition; but by the end the amount of change for all teams should level off, indicating they are sorted well. But the theory was never explained, and I could never test it. Until now. Based on a semi-random sample of eight regional competitions from the 2015 season, this post shows the extent to which teams really are sorted and draws a few conclusions about what it means for scouting.

For the last 5+ years, teams were ranked by their qualification score and quite often secondary and tertiary terms. Those extra terms were never available in the match results posted online, making it very difficult to determine how teams change in their rank during the course of the competition. This year, however, since the qualification score is the team's average score, it's very rare to need tie-breakers after the 2nd round. Yeah!

I started with the 10,000 Lakes and North Star Regionals since that is where I am located. But I expanded the analysis to include Colorado, Ventura (since they have 12 matches), Silicon Valley (since I heard the average qualification score was high), Wisconsin (close to MN), UNH District (12 matches), and New York City (many teams, only 8 matches).

The attached spreadsheet (FRC Team Rankings Chart.xlsx) has all the analysis in it plus some extra plots. For your convenience, I'll post a few charts here.



This chart shows how each team's rank changed over the course of qualifications at the North Star Regional. The number and extent of changes in rankings between match 9 and 10 are quite a bit more than I expected. However, what is interesting is that the changes in ranking for the top 10 teams or so has leveled off. The next chart removes some of the clutter and shows the path of the top 10 teams.



These teams follow the path I was expecting: lots of change early on and relatively little change by the end of their matches. So, from this graph (and the graph for the Ventura regional), it seems safe to say that the top 8 really are the top 8, sorted in that order.

But, unfortunately, this is not the case with all of the regionals. Let's look at the Wisconsin Regional.



Even though the top nine teams have been identified, it looks like there's still potential for change if they played more matches. So what happens if there are more matches?

The University of New Hampshire District Event and Ventura Reginonals both had 12 matches. One would hope that after 9 or 10 matches, they would be stable; but that is not necessarily the case.



At the UNH District event, top 8 teams were changing by as many as 3 places between the 11th and 12th matches.

Another way to look at the data is to ask how early the top 3 or 5 or 8 could be identified. They might not be sorted, but they are at least in the top spots (see 'TopRankAnalysis' sheet).

The top one or two teams for nearly every regional were identified right away, by the second or third match. Taking all sampled regionals into consideration, one could very safely say the top team is identified by round 6; and it could be safe to say the top three are identified by round 8. With some risk you could say by match 5, the top three teams will stay there.

So, how many matches are really needed to determine final rankings? Apparently many many more than just 9 or 10 if we want every team to be sorted. However, since it is the top 8 who are picking their alliance partners, I think it is safe to say those teams, at least, have been identified by the end of qualifications. More matches could be helpful to further sort those top 8. But I also think most regionals run into time constraints. Those Fridays and Saturdays are long days.

I think another outcome, especially for those top 8 when they are choosing their alliance partners, is that the top 8 should not base their picks based on how teams are sorted beyond the top 10. Scouts really do need to look at how many points each robot is able to contribute to the final score and not assume teams are completely sorted. A team ranked 11 isn't necessarily better than a team ranked 15, for example.

And now that I've run out of time, one other musing I have is the effect of top teams. The Cheesey Poofs, for example, had an average score of 200, while most teams averaged about 65 points. Since the Cheesey Poofs' totes count towards their alliance partners' scores, how did that effect the rankings of their alliance partners? Perhaps another day. The spreadsheet also has histograms showing how many teams averaged between 50-60, 60-70, 70-80 points, and so on at each competition. Based on the peaks of the curves, one can tell fairly quickly which regionals are more competitive than others. Perhaps this, too, can be elaborated on another day.