Veteran Teams have No Advantage

**A low team number means absolutely NOTHING when it comes to predicting the team’s performance in qualification rounds. **

Now that I’ve got your attention, allow me to explain. I was inspired by discussion in this thread “Random” match Schedules](http://www.chiefdelphi.com/forums/showthread.php?t=55178) discussing the apparent lack of randomness in assigning alliances for qualifying matches. It was stated that FIRST intentionally schedules matches so that alliances have roughly equal seniority, based on the assumption (presumably) that this will lead to a more balanced match.

That didn’t seem fair to me…

So I downloaded the results from the first weekend’s competitions, and dropped them into excel. Based on these 270 data points there is abolutely NO relationship between team number and seeding. The teams numbered below 300 had an average seed of 26, while teams numbered above 2000 had an average seed of 27.8. Hardly an advantage (although, admittedly I didn’t calculate exactly how much difference would be required to be statistically significant here). Running a correlation coefficient over the whole data set shows a .007 coefficient of correlation between team number and seeding… and in some regionals the coefficient is negative (but also insignficant.)

This leads me to three possible hypothesis:

  1. **We are all mistaken about the advantage that senior teams have. ** Hey, I’ve done it… you’ve done it… and now it is alleged that FIRST is doing it. We’re looking at a few very successful veteran teams and saying “wow… watch out for the teams with low numbers” and completely forgetting that for every veteran out there rocking the rack that there is another one struggling with their design, and a rookie team that is doing even better.

  2. Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

  3. My statistical analysis is incorrect or incomplete. I’m always willing to admit the possibility that I might be wrong. I challenge anyone, however, to prove that veteran teams have done signficantly better in qualification matches than newer teams. Math, here, please people… not anecdotes. We humans are really good at seeing relationships that don’t really exist.

I’m open to suggestions, and to someone willing to examine finals matches and outcomes to see if veteran teams have any statistical advantage on Saturday afternoon… or to someone who can find some better predictor of success (perhaps previous year’s rankings can be a reliable indicator) but until then… I say that veteran teams have no advantage when it comes to winning matches and that a scheduling system that (allegedly) uses team numbers as a factor is not only unjustified, but unscientific.

Jason

P.S. Yes, veteran teams (my gosh… we’re one now, I think…) have many advantages (and challenges) that junior teams might not have… but not ones that significantly affect the outcomes of qualifying matches.

Does anyone have, or know where I can get qualifying match data from last year? I’d like to do something similar, but for last year, so I can get some idea as to how much the new match algorithm matters.

Thanks.

http://www2.usfirst.org/2006comp/events/NH/rankings.html is the list for BAE last year. Replace the NH with your respective regional town (SJ for Silicon Valley, WI for Wisconsin, etc)

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

I agree… you may also want to take into account the number of teams per regional. See how many closely numbered teams were playing other close numbered teams and then factor in the number of teams at the event. I don’t know how to do that, I’m not a math person.

I don’t disagree, however my analysis is based on this year’s data. Please see hypothesis two concerning it’s interpretation.

I’m looking forward to seeing an analysis of last year’s numbers (it sounds as if someone is running that right now…). If lower numbered teams did significantly better last year relative to newer teams than this year, and FIRST has implemented a new scheduling system, then I would say that the scheduling system has served to undermine years of hard work and development on veteran teams.

“IF”, however, is a pretty big word. I’ll wait to see the results. Personally, until I see otherwise I think the “low number=power house” hypothesis is based on the same irrational pattern recognition that makes us silly humans believe in lucky rabbit’s feet and horseshoes.

Jason

I’m currently running data analysis on last year’s data, and it seems to support the notion that lower number teams have no advantage. I think that notion comes due to the existence several low-number power houses - 25 and 254 serve as great examples. However, many other low numbered teams are not nearly as strong. I’ll present some data tonight, and get some more online tomorrow.

I remember seeing a thread a while back that compared '05 and '06 success at nationals. There may be some relevant data in the graphs in that thread if anyone remembers what thread I am referring to.

There are many struggling veteran teams. Just because they have a low number doesn’t mean there is an institutional advantage. Many of the lower number teams “recreate” themselves and go through any number of crises and evolve into a new team. Their number may be one of the few things that stays the same and the lessons learned don’t necessarily get passed on. I don’t have any hard data to back up what I just said; only personal experience.
For every statistic, there is another story.

Measuring the true capability of any team (regardless of how long they have competed) requires more than simply analyzing the outcome of the matches that they participated in.

The results may mask many other things that a single participant in any alliance would have had to deal with. Not many veteran teams can overcome, incompatible alliance pairings, mechanical breakdowns, strategic mistakes, etc. - all the things that could go wrong or right that often determine the outcome of a match.

Unfortunately, the scores and match outcomes is what is often used to evaluate (statistically) these kinds of debates/discussions.

Yes, one would assume that more veteran teams would improve over time, utilize the things that they have witnessed, attempted, failed, and succeeded at.

There are far too many variables involved to make statements one way or the other, as to whom has an advantage over whom.

One can only assume, that veteran teams have learned to deal with the challenges better than the lesser experienced team, strickly due to level of exposure.

I think that one thing you definitely need to look at in addition to what you have done so far (which seems sound) is to look at how likely veteran teams are to place in the top 8 compared to younger teams. Which is different from the analysis you have done so far. It is an interesting question though. Gives me a good idea for an assignment for AP Statistics class while I am gone for the regional this weeked.

OK, I also just took a look at some of the data and it occurs to me that your result is exactly what FIRST would be trying to achieve by randomizing matches as it did.

What makes a veteran team? Participating in one year then returning the next should qualify that statement. Or maybe being one of the founding teams qualifies at team as a veteran.
Who gives a team this seniority? Our team for example is #1038. Our student population changes every two to three years. Are we less experienced than Team #45? We have mentors who are very active on our team that have been building FIRST robots since 1994, as a matter of fact, several of these robots have been very successful in the FIRST contests - (Sunny D, 1994 National Champs, 1996 National Semi + Chairmans Award, and the list for our former team - #144, goes on until we switched schools in 1998)
The students on some teams change every year…The mentors change occasionally, too. So, is there really an advantage to participation from year to year? It totally depends on the composition of individual teams and therefore cannot be controlled in an algorithm for match scheduling that is intended to have “veteran” teams play matches with other experienced teams, and “rookies” play more rookies. We sometimes have great years, followed by some “re-building” years. Sometimes we figure the game out, other times we totally miss it. That is part of life, part of FIRST. Your team number does not imply any “powerhouse” status. That is earned by consistant performance and should not be accounted for in any match scheduling.
I agree with the title of this thread. With returning mentors and students there are always lessons learned and re-applied to following years, but with a student-centric team, the personnel changes can be drastic from year to year and are not reflected accurately by a team number.

I did this last year. I found that older teams DID have an advantage in terms of scoring, if not seeding. This was based on all the week 1 and week 2 regionals in 2006.

http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4111&d=1142127063 - Average alliance team number versus score
http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4114&d=1142139985 - Average alliance age in years versus score
And Here’s the thread, with many more charts and things.

I remember seeing a thread a while back that compared '05 and '06 success at nationals. There may be some relevant data in the graphs in that thread if anyone remembers what thread I am referring to.

I think that was me as well. I downloaded the seed results of a bunch of teams from a bunch of regionals. I normalized them so that they were between 0 and 1, and the plotted the results. A point at (x,y) means that in 2005 a team got ranking x, and in 2006 they got ranking y. This graph shows that there is little year-to-year correlation for performance. Someone who does awful one year can get as high as the top 10% in the next year. Likewise, someone who wins regionals one year can go to the bottom 10% the next year. The powerhouse teams that win consistently from year to year aren’t as widespread as people think.
http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4724&d=1162868157

If anyone needs raw data from last year, I attached a zipped xml file that has all the regional stats. It can be viewed in excel. I generated it with my FIRSTXML program. I really need to update FIRST XML for this years regionals.

MASTERlite.zip (96.5 KB)


MASTERlite.zip (96.5 KB)

I’ll take a look it this soon, but you might have just saved me and another team member a lot of manual labor.

Thanks!

Guy

I found that guessing for the acronym was very difficult, so I went to google and did this:

<word from regional name> results site:www2.usfirst.org

And google will get it almost every time.

That is very interesting data and analysis, and suggests that perhaps there is a slight advantage to be gained from experience (which one might well expect) but certainly not an insurmountable one. It also meshes well with the idea that – particularly for a student centred team – there is an upper limit to how much experience one can gain. Does it make sense that a ten-year team would have any advantage over a four-year team?

I think that changes to the KOP and the availability of COTS mechanisms have also greatly helped rookies and evened up opportunities. A two-speed gearbox or mecanum wheel four years ago was something that required a lot of technical expertise… now it requires a credit card and some bolts. Which isn’t a bad thing, but it does level the playing field.

Mentoring and support also helps balance the playing field. I know our rookie neighbours (2273, who did quite well and placed 20 spots ahead of us in Portland) are using a couple of our old motors and speed controllers in their system because they didn’t have the inventory of spare parts on hand that we do. Their neighbours in the pits, team 114, not only loaned them a robot cart at first, but then built them one! (One of the many reasons 114 took home the J&J Sportsmanship award… well done, #114!) One of the great things about FIRST is that veterans help rookies, rather than pummel them.

So I look at this as follows… either:

  1. Veteran teams have NO inherent advantage in terms of winning qualifying matches. The data posted on scoring from last year, however, suggests that this might not be the case.

  2. The current scheduling system eliminates any advantage gained by experience by repeatedly pitting veteran teams against each other rather than by having them compete against a random selection of opponents. Assuming that this scheduling system is new this year, this would explain why last year veteran teams scored higher, but this year had not advantage in qualifying matches. (Yes, two slightly different measurments, but the best we’ve got right now.)

In any case I understand that FIRST is considering an adjustment to the match scheduling system. If they choose to eliminate the rookies vs. rookies and veterans vs. veterans part of the formula we can re-run the qualifying results from next weekend and see if there is more of a correlation between team number and seeding.

I think that would be really interesting to look at…

Jason

P.S. What a great idea to assign this as a stats project to an AP stats class… I might save this as an assignment for some of my math classes, too.

Even easier: http://www.usfirst.org/community/frc/content.aspx?id=4188

Wow, I spent half an hour looking through their site yesterday for those before I gave up and did my google method. Thank you.