Veteran Teams have No Advantage

[dtengineering]

A low team number means absolutely NOTHING when it comes to predicting the team's performance in qualification rounds.

Now that I've got your attention, allow me to explain. I was inspired by discussion in this thread "Random" match Schedules discussing the apparent lack of randomness in assigning alliances for qualifying matches. It was stated that FIRST intentionally schedules matches so that alliances have roughly equal seniority, based on the assumption (presumably) that this will lead to a more balanced match.

That didn't seem fair to me....

So I downloaded the results from the first weekend's competitions, and dropped them into excel. Based on these 270 data points there is abolutely NO relationship between team number and seeding. The teams numbered below 300 had an average seed of 26, while teams numbered above 2000 had an average seed of 27.8. Hardly an advantage (although, admittedly I didn't calculate exactly how much difference would be required to be statistically significant here). Running a correlation coefficient over the whole data set shows a .007 coefficient of correlation between team number and seeding... and in some regionals the coefficient is negative (but also insignficant.)

This leads me to three possible hypothesis:

1) We are all mistaken about the advantage that senior teams have. Hey, I've done it... you've done it... and now it is alleged that FIRST is doing it. We're looking at a few very successful veteran teams and saying "wow... watch out for the teams with low numbers" and completely forgetting that for every veteran out there rocking the rack that there is another one struggling with their design, and a rookie team that is doing even better.

2) Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

3) My statistical analysis is incorrect or incomplete. I'm always willing to admit the possibility that I might be wrong. I challenge anyone, however, to prove that veteran teams have done signficantly better in qualification matches than newer teams. Math, here, please people... not anecdotes. We humans are really good at seeing relationships that don't really exist.

I'm open to suggestions, and to someone willing to examine finals matches and outcomes to see if veteran teams have any statistical advantage on Saturday afternoon... or to someone who can find some better predictor of success (perhaps previous year's rankings can be a reliable indicator) but until then.... I say that veteran teams have no advantage when it comes to winning matches and that a scheduling system that (allegedly) uses team numbers as a factor is not only unjustified, but unscientific.

Jason

P.S. Yes, veteran teams (my gosh... we're one now, I think...) have many advantages (and challenges) that junior teams might not have... but not ones that significantly affect the outcomes of qualifying matches.

[Guy Davidson]

Does anyone have, or know where I can get qualifying match data from last year? I'd like to do something similar, but for last year, so I can get some idea as to how much the new match algorithm matters.

Thanks.

[Nuttyman54]

Originally Posted by sumadin

Does anyone have, or know where I can get qualifying match data from last year? I'd like to do something similar, but for last year, so I can get some idea as to how much the new match algorithm matters.

Thanks.

http://www2.usfirst.org/2006comp/eve.../rankings.html is the list for BAE last year. Replace the NH with your respective regional town (SJ for Silicon Valley, WI for Wisconsin, etc)

[nuggetsyl]

Originally Posted by dtengineering

A low team number means absolutely NOTHING when it comes to predicting the team's performance in qualification rounds.

Now that I've got your attention, allow me to explain. I was inspired by discussion in this thread "Random" match Schedules discussing the apparent lack of randomness in assigning alliances for qualifying matches. It was stated that FIRST intentionally schedules matches so that alliances have roughly equal seniority, based on the assumption (presumably) that this will lead to a more balanced match.

That didn't seem fair to me....

So I downloaded the results from the first weekend's competitions, and dropped them into excel. Based on these 270 data points there is abolutely NO relationship between team number and seeding. The teams numbered below 300 had an average seed of 26, while teams numbered above 2000 had an average seed of 27.8. Hardly an advantage (although, admittedly I didn't calculate exactly how much difference would be required to be statistically significant here). Running a correlation coefficient over the whole data set shows a .007 coefficient of correlation between team number and seeding... and in some regionals the coefficient is negative (but also insignficant.)

This leads me to three possible hypothesis:

1) We are all mistaken about the advantage that senior teams have. Hey, I've done it... you've done it... and now it is alleged that FIRST is doing it. We're looking at a few very successful veteran teams and saying "wow... watch out for the teams with low numbers" and completely forgetting that for every veteran out there rocking the rack that there is another one struggling with their design, and a rookie team that is doing even better.

2) Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

3) My statistical analysis is incorrect or incomplete. I'm always willing to admit the possibility that I might be wrong. I challenge anyone, however, to prove that veteran teams have done signficantly better in qualification matches than newer teams. Math, here, please people... not anecdotes. We humans are really good at seeing relationships that don't really exist.

I'm open to suggestions, and to someone willing to examine finals matches and outcomes to see if veteran teams have any statistical advantage on Saturday afternoon... or to someone who can find some better predictor of success (perhaps previous year's rankings can be a reliable indicator) but until then.... I say that veteran teams have no advantage when it comes to winning matches and that a scheduling system that (allegedly) uses team numbers as a factor is not only unjustified, but unscientific.

Jason

P.S. Yes, veteran teams (my gosh... we're one now, I think...) have many advantages (and challenges) that junior teams might not have... but not ones that significantly affect the outcomes of qualifying matches.

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

[Jeremiah Johnson]

Originally Posted by nuggetsyl

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

I agree... you may also want to take into account the number of teams per regional. See how many closely numbered teams were playing other close numbered teams and then factor in the number of teams at the event. I don't know how to do that, I'm not a math person.

[dtengineering]

Originally Posted by nuggetsyl

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

I don't disagree, however my analysis is based on this year's data. Please see hypothesis two concerning it's interpretation.

I'm looking forward to seeing an analysis of last year's numbers (it sounds as if someone is running that right now...). If lower numbered teams did significantly better last year relative to newer teams than this year, and FIRST has implemented a new scheduling system, then I would say that the scheduling system has served to undermine years of hard work and development on veteran teams.

"IF", however, is a pretty big word. I'll wait to see the results. Personally, until I see otherwise I think the "low number=power house" hypothesis is based on the same irrational pattern recognition that makes us silly humans believe in lucky rabbit's feet and horseshoes.

Jason

[Guy Davidson]

I'm currently running data analysis on last year's data, and it seems to support the notion that lower number teams have no advantage. I think that notion comes due to the existence several low-number power houses - 25 and 254 serve as great examples. However, many other low numbered teams are not nearly as strong. I'll present some data tonight, and get some more online tomorrow.

[AdamHeard]

I remember seeing a thread a while back that compared '05 and '06 success at nationals. There may be some relevant data in the graphs in that thread if anyone remembers what thread I am referring to.

[Petey]

Originally Posted by dtengineering

[b][u]
2) Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

Or, it is possible that if you have two veteran teams and a younger team, the strength of that veteran team may be able to overcome the weakness of the younger team.

Perhaps a better statement would be that won/loss records mean nearly nothing when it comes to evaluating the strength of any robot. WPI had a much stronger robot than us this year and yet finished nearly 10 spots lower. Why? Because they had weak alliance partners and faced Buzz four times.

Why do you think you see so many teams keeping spreadsheets of individual robots performances? Why do you suppose that, after the usual top-8 incest (after all, if you're ranked #1, chances are you are dominant enough to carry both of your partners on your backs), we often see teams picking from the 20's or 30's instead of the teens?

It's because alliances, throughout which the strength and weakness is divided, =! teams.

Of course veteran teams have an advantage. They have infrastructure. They have institutional expertise. They have tried and failed and thus come out with more knowledge. You can have rookie teams that do quite well, but how often do you see a rookie team come out of the blue and dominate the field? A second year team? Even a third? If you could find the data, try seeing how many younger teams really dominate in any of the game components. I don't think it happens.

Your analysis isn't wrongheaded, it's just that the data is deceptive.

[RoboMom]

Originally Posted by Petey

Of course veteran teams have an advantage. They have infrastructure. They have institutional expertise. They have tried and failed and thus come out with more knowledge.

Your analysis isn't wrongheaded, it's just that the data is deceptive.

There are many struggling veteran teams. Just because they have a low number doesn't mean there is an institutional advantage. Many of the lower number teams "recreate" themselves and go through any number of crises and evolve into a new team. Their number may be one of the few things that stays the same and the lessons learned don't necessarily get passed on. I don't have any hard data to back up what I just said; only personal experience.
For every statistic, there is another story.

[meaubry]

Measuring the true capability of any team (regardless of how long they have competed) requires more than simply analyzing the outcome of the matches that they participated in.

The results may mask many other things that a single participant in any alliance would have had to deal with. Not many veteran teams can overcome, incompatible alliance pairings, mechanical breakdowns, strategic mistakes, etc. - all the things that could go wrong or right that often determine the outcome of a match.

Unfortunately, the scores and match outcomes is what is often used to evaluate (statistically) these kinds of debates/discussions.

Yes, one would assume that more veteran teams would improve over time, utilize the things that they have witnessed, attempted, failed, and succeeded at.

There are far too many variables involved to make statements one way or the other, as to whom has an advantage over whom.

One can only assume, that veteran teams have learned to deal with the challenges better than the lesser experienced team, strickly due to level of exposure.

[mathking]

I think that one thing you definitely need to look at in addition to what you have done so far (which seems sound) is to look at how likely veteran teams are to place in the top 8 compared to younger teams. Which is different from the analysis you have done so far. It is an interesting question though. Gives me a good idea for an assignment for AP Statistics class while I am gone for the regional this weeked.

OK, I also just took a look at some of the data and it occurs to me that your result is exactly what FIRST would be trying to achieve by randomizing matches as it did.

[Dave Campbell]

What makes a veteran team? Participating in one year then returning the next should qualify that statement. Or maybe being one of the founding teams qualifies at team as a veteran.
Who gives a team this seniority? Our team for example is #1038. Our student population changes every two to three years. Are we less experienced than Team #45? We have mentors who are very active on our team that have been building FIRST robots since 1994, as a matter of fact, several of these robots have been very successful in the FIRST contests - (Sunny D, 1994 National Champs, 1996 National Semi + Chairmans Award, and the list for our former team - #144, goes on until we switched schools in 1998)
The students on some teams change every year...The mentors change occasionally, too. So, is there really an advantage to participation from year to year? It totally depends on the composition of individual teams and therefore cannot be controlled in an algorithm for match scheduling that is intended to have "veteran" teams play matches with other experienced teams, and "rookies" play more rookies. We sometimes have great years, followed by some "re-building" years. Sometimes we figure the game out, other times we totally miss it. That is part of life, part of FIRST. Your team number does not imply any "powerhouse" status. That is earned by consistant performance and should not be accounted for in any match scheduling.
I agree with the title of this thread. With returning mentors and students there are always lessons learned and re-applied to following years, but with a student-centric team, the personnel changes can be drastic from year to year and are not reflected accurately by a team number.

[Bongle]

I did this last year. I found that older teams DID have an advantage in terms of scoring, if not seeding. This was based on all the week 1 and week 2 regionals in 2006.

http://www.chiefdelphi.com/forums/at...1&d=1142127063 - Average alliance team number versus score
http://www.chiefdelphi.com/forums/at...4&d=1142139985 - Average alliance age in years versus score
And Here's the thread, with many more charts and things.

I remember seeing a thread a while back that compared '05 and '06 success at nationals. There may be some relevant data in the graphs in that thread if anyone remembers what thread I am referring to.

I think that was me as well. I downloaded the seed results of a bunch of teams from a bunch of regionals. I normalized them so that they were between 0 and 1, and the plotted the results. A point at (x,y) means that in 2005 a team got ranking x, and in 2006 they got ranking y. This graph shows that there is little year-to-year correlation for performance. Someone who does awful one year can get as high as the top 10% in the next year. Likewise, someone who wins regionals one year can go to the bottom 10% the next year. The powerhouse teams that win consistently from year to year aren't as widespread as people think.
http://www.chiefdelphi.com/forums/at...4&d=1162868157

[The Lucas]

If anyone needs raw data from last year, I attached a zipped xml file that has all the regional stats. It can be viewed in excel. I generated it with my FIRSTXML program. I really need to update FIRST XML for this years regionals.