paper: Relative Success of Championship Teams based on Qualification Method

[Ken Streeter]

Originally Posted by Kims Robot

... What I find interesting is that the CA line is trending downward, but the HoF line is not.

This brings up a few interesting questions... Any theories?

I have a couple thoughts about this. However, since this issue is complex, and I tend to be verbose, this note is likely to be a "tl;dr" for many.

First off, I really like the "Representation Index" as a way to compare the "relative success" of teams from these different qualification categories. Essentially, the "Representation Index" indicates whether or not a disproportionate number of the teams in a given category are making it into the CMP elimination rounds. Very high numbers mean that almost all teams in that category make it to CMP elims. Very low numbers mean that almost none of the teams in that category end up in CMP elims.

Looking at the graph, one will see that the bigger categories have smoother trend lines, since the larger numbers of teams in these groups average out noise. Regional Winners are about 40% of the teams at CMP, while about 15% of the teams belong to each of RCA, EIS, and RAS categories. As expected by basic statistics, these categories have much more stable trend lines than the smaller categories, as whether or not one or two teams end up making eliminations or not doesn't change the statistics much for a bigger category.

On the other hand, with the categories for "Original & Sustaining", "Last Year's Winners", and "Hall of Fame" each being only a few percent of teams, those trend lines are much more volatile, despite being helped by the 4-year moving average.

Now, on to Kim's specific questions:

Originally Posted by Kims Robot

... 1. What is the correlation between CA teams and "good robots"?

In general, I think that there is a pretty strong correlation between CA teams and "good robots." However, the criteria of being "in CMP elims" (the criteria which matters for the "Representation Index") is a pretty stringent restriction. The "CMP elims" robots aren't just "good robots" but consist of only 96 robots in the world, out of around 2550 FIRST teams in 2013. Ignoring the fact that the "CMP elimination" teams aren't really the "top 96" in the world due to some excellent robots not attending CMP, or not making it to the elimination rounds for other reasons, the "CMP elimination caliber" robots are about the top 4% of teams worldwide.

This top 4% of teams worldwide aren't just "good robots" but are "exceptional robots." So, the question really is, "What is the correlation between CA teams and exceptional robots?"

Well, what the statistics show is that being an RCA team isn't as well-correlated with being an exceptional robot as things such as winning CMP last year, qualifying by rank from a district CMP, being a HoF team, winning a regional, or being one of the hanful of original & sustaining teams.

That said, I think that RCA teams generally do have "good robots" -- they just have proportionately less "exceptional robots" than some of the other categories.

I think what we're seeing is that some teams focus more on the robot, some teams focus more on CA activities, and some teams try to excel at both. Teams that focus more on the robot are likely to have a bit of an edge over similarly capable teams that focus on CA activities or teams that balance both. It requires a lot more team effort to build both an "exceptional robot" and be an RCA team than it does to only build an "exceptional robot."

Originally Posted by Kims Robot

... 2. Do the incredibly strong CA teams have strong all around programs thus resulting in good robots? or are CCA judges picking teams that have strong robots in addition to strong CA criteria?

In general, I think the correlation between CA teams and "good robots" is primarily due to the strength of the program from those teams. From what I've seen from the outside, the RCA judges seem to make their decisions without really considering the strength of the team's robot. The CCA judges are picking the very best RCA team each year -- the level of competition for the CCA is so high that only elite programs are even in the running. Such elite programs are excellent not only in CA qualities, but also in the robot design, build, and operation, meaning they are very likely to have "exceptional robots." I think this is why the HoF teams are so highly represented in the CMP elimination rounds.

Originally Posted by Kims Robot

3. Why is the RCA line dipping? Is it because there are more RCA's given out, thus the pool has to be spread more thinly? Is it because RCA teams are concentrating more on their RCA performance than Robot performance?

Well, everything else being equal, one actually would expect the RCA line to dip a little bit every year, as the very best CA team from the prior year becomes a Hall-of-Fame team for the next year, and thus the RCA category loses one of its best teams, which then gets back-filled by a lesser team. This new HoF team was surely an "elite program" or they wouldn't have won the CCA; accordingly, that team is very likely to produce an "exceptional robot." However, the downward trend is moving faster than just one team shifting to the HoF category each year.

This is speculation on my part, but I think the primary reason for the downward trend in the RCA line is the arrival on the scene of the new "By Rank from District CMP" category. Since these are all at least "very good robots" within their district, these teams have earned higher representation in the elimination rounds, which has tended to displace RCA and RAS teams from elimination round berths.

However, I also find it interesting that EI teams have not seen the downwards trend experienced by RAS and RCA teams. It would seem that over the past 6 years, EI teams have become just as strongly correlated with "exceptional robots" than RCA teams, although that was not the case back in the 2007 time frame.

I would also note that the fast drop in the RAS trend line makes sense, too, as with more and more veteran teams each year bringing up the average level of capability, it gets harder and harder for a rookie team to build an "exceptional robot" in their first year.

Having said all of the above, I would be really interested in seeing the "Regional Winner" trend line broken out into three separate categories for "Regional Winner Captain", "Regional Winner 1st-Pick", and "Regional Winner 2nd-Pick" as I think at least one of those lines would be significantly different than the others! However, that data may be very difficult to add to the spreadsheet that was used to generate these charts, if it isn't already there.