Veteran Teams have No Advantage

[dtengineering]

A low team number means absolutely NOTHING when it comes to predicting the team's performance in qualification rounds.

Now that I've got your attention, allow me to explain. I was inspired by discussion in this thread "Random" match Schedules discussing the apparent lack of randomness in assigning alliances for qualifying matches. It was stated that FIRST intentionally schedules matches so that alliances have roughly equal seniority, based on the assumption (presumably) that this will lead to a more balanced match.

That didn't seem fair to me....

So I downloaded the results from the first weekend's competitions, and dropped them into excel. Based on these 270 data points there is abolutely NO relationship between team number and seeding. The teams numbered below 300 had an average seed of 26, while teams numbered above 2000 had an average seed of 27.8. Hardly an advantage (although, admittedly I didn't calculate exactly how much difference would be required to be statistically significant here). Running a correlation coefficient over the whole data set shows a .007 coefficient of correlation between team number and seeding... and in some regionals the coefficient is negative (but also insignficant.)

This leads me to three possible hypothesis:

1) We are all mistaken about the advantage that senior teams have. Hey, I've done it... you've done it... and now it is alleged that FIRST is doing it. We're looking at a few very successful veteran teams and saying "wow... watch out for the teams with low numbers" and completely forgetting that for every veteran out there rocking the rack that there is another one struggling with their design, and a rookie team that is doing even better.

2) Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

3) My statistical analysis is incorrect or incomplete. I'm always willing to admit the possibility that I might be wrong. I challenge anyone, however, to prove that veteran teams have done signficantly better in qualification matches than newer teams. Math, here, please people... not anecdotes. We humans are really good at seeing relationships that don't really exist.

I'm open to suggestions, and to someone willing to examine finals matches and outcomes to see if veteran teams have any statistical advantage on Saturday afternoon... or to someone who can find some better predictor of success (perhaps previous year's rankings can be a reliable indicator) but until then.... I say that veteran teams have no advantage when it comes to winning matches and that a scheduling system that (allegedly) uses team numbers as a factor is not only unjustified, but unscientific.

Jason

P.S. Yes, veteran teams (my gosh... we're one now, I think...) have many advantages (and challenges) that junior teams might not have... but not ones that significantly affect the outcomes of qualifying matches.

[Guy Davidson]

Does anyone have, or know where I can get qualifying match data from last year? I'd like to do something similar, but for last year, so I can get some idea as to how much the new match algorithm matters.

Thanks.

[Nuttyman54]

Originally Posted by sumadin

Does anyone have, or know where I can get qualifying match data from last year? I'd like to do something similar, but for last year, so I can get some idea as to how much the new match algorithm matters.

Thanks.

http://www2.usfirst.org/2006comp/eve.../rankings.html is the list for BAE last year. Replace the NH with your respective regional town (SJ for Silicon Valley, WI for Wisconsin, etc)

[Bongle]

Originally Posted by Nuttyman54

http://www2.usfirst.org/2006comp/eve.../rankings.html is the list for BAE last year. Replace the NH with your respective regional town (SJ for Silicon Valley, WI for Wisconsin, etc)

I found that guessing for the acronym was very difficult, so I went to google and did this:

<word from regional name> results site:www2.usfirst.org

And google will get it almost every time.

[Joe Ross]

Originally Posted by Bongle

I found that guessing for the acronym was very difficult, so I went to google and did this:

<word from regional name> results site:www2.usfirst.org

And google will get it almost every time.

Even easier: http://www.usfirst.org/community/frc...t.aspx?id=4188

[Bongle]

Originally Posted by Joe Ross

Even easier: http://www.usfirst.org/community/frc...t.aspx?id=4188

Wow, I spent half an hour looking through their site yesterday for those before I gave up and did my google method. Thank you.

[Rich Ross]

Two things:

1. Off topic a bit, but every time that a thread like this pops up, we are justifying the current failure that is the match sheduling system. By saying that it levels the playing field, people from FIRST think more and more that this is acceptable. I'm going to be blunt and say it: I ama against the current algorithm (spelling?) and expect first to come up with something better that will pair us with and against a bigger variety of teams

2. There are verteran powerhouses (someone mentioned 25 and 254) but there are also rookie powerhouses. Maybe they dont win a regional every year, but every year you will see rookies that mow down veterans. 1503 won 3 regionals last year, with 1114 (both relatively new teams). It's a fact that rookies and veterans are both FIRST teams, and given similar knowledge, they will both perform well.

Let's let FIRST know that we want to play against everyone, rookies and vets.

[Nuttyman54]

It's not veteran status. It's not rookies status. It's infrastructure. The teams that are well organized, well directed and are set up to ensure longevity are the ones that do well.

In order to be successful, you have to learn from experience. Rookies that have experienced mentors and enthusiastic members will do well (see 1902 for a prime example).

On the flip side, veteran teams that have no continuity from year to year are going to do poorly, or have mixed results. Not having any information on past robots can be very detrimental, as the team has to re-learn all of it's previous mistakes. Team handbooks, dedicated sponsors and supportive communities are the marks of successful teams.

[Matt Reiland]

One advantage that vetern teams DO have is their lower number and their team history. I have been at many a regional where a Big Name team didn't have the strongest robot of the litter but they were picked based on reasons other than that particular years robot performance. Sometimes it is the experience under pressure, sometimes it is the way they sell themselves, it could have even been that it was an incredibly robust machine and went out for every single match. Either way you don't have to have the #1 seed robot to be on the winning team, you may even have the worst robot at the regional (Based on the qualifying match scores) especially this year when you see some of these matches go so lopsided when the right robots are out there together against and unevenly matched oponent.

[Guy Davidson]

Originally Posted by Matt Reiland

One advantage that vetern teams DO have is their lower number and their team history. I have been at many a regional where a Big Name team didn't have the strongest robot of the litter but they were picked based on reasons other than that particular years robot performance. Sometimes it is the experience under pressure, sometimes it is the way they sell themselves, it could have even been that it was an incredibly robust machine and went out for every single match. Either way you don't have to have the #1 seed robot to be on the winning team, you may even have the worst robot at the regional (Based on the qualifying match scores) especially this year when you see some of these matches go so lopsided when the right robots are out there together against and unevenly matched oponent.

This argument only supports the reason for a new algorithm. The less teams one is exposed to, the less quality information he is likely to have, the more likely he is to revert to picking a low-numbered team. Hence, we should revert to a random algorithm, that will let each team compete with and against more teams.

-Guy

EDIT: Results from the inference I ran on last year's regionals results are coming tomorrow. I promise

[ScoutingNerd175]

Our scouting team took data on the number of tetras/balls actually scored by robots at the regionals we were attending. This may be more reflective of how teams are actually scoring. The only data I have access to at the moment is the data for the 2005 Chesapeake Bay Regional. So, out of 55 teams at the regional, 25 (45%) were numbered above 1000 (arbitrary number) with 9 (16.3%) being rookies. Looking at just the top 8 (scorers, not seeds), 2 (25%) were numbered over 1000, 1 (12.5%) was a rookie team. If I pick an arbitrary number again, like 2 tetras, 36% of teams that scored 2+ tetras were numbered over 1000 and 18% were rookies.



So I'm not sure if this data is in any way meaningful, but it may be more meaningful if someone keeps data on scoring, or something along those lines, to compare that to team number (in a more meaningful way than this) or to compare scoring to seed this year and last year (for example what % of the top 8 seeds were also in the top 8 scoring this years as compared to last year) to see if this algorithm is having an effect on where the highest scoring teams seed. (I can't do that here because I don't know the seeds for 2005 Chesapeake)

Or it might all be meaningless. But match results definitely do not equal scoring ability (the highest scoring robot in the Curie division at 2005 Championships was seeded 11th). Although, scoring ability definitely does not equal robot quality.

[Doug Leppard]

Originally Posted by Nuttyman54

It's not veteran status. It's not rookies status. It's infrastructure. The teams that are well organized, well directed and are set up to ensure longevity are the ones that do well.

In order to be successful, you have to learn from experience. Rookies that have experienced mentors and enthusiastic members will do well (see 1902 for a prime example).

On the flip side, veteran teams that have no continuity from year to year are going to do poorly, or have mixed results. Not having any information on past robots can be very detrimental, as the team has to re-learn all of it's previous mistakes. Team handbooks, dedicated sponsors and supportive communities are the marks of successful teams.

As a mentor of 1902 I agree. My previous team 1083 the school decided not to continue the team. A team mom on 1083 from another school decided to run it from their home without official shool sponsership and so at last minute started 1902.

But we pulled in experienced students, college mentors, adult mentors. George has 10 years experience, I have 5.

We are lucky to have deep infrastructure:
Strong hardware and partnered with another team to fill in our weaknesses
Strong software that has developed successful auto modes
Strong driver strategy and strong drive team
Strong non-engineers that raise money and awareness

It takes all that and more to make a succesfull team.

But we are still growing and learning what infrastructure it takes to run a team and be succesful. We are still making rookie mistakes but are learning and adjusting to those mistakes.

Big one last year when the hardware guys gave us a day to to auto mode and it put us ahead of others. This year they worked hard and delivered the bot over a week ahead so we had lots of driver practice, spent 60 hours on details of software and auto mode and worked out hardware problems.

Final thing is the team is fun, that stage being set by our team leader Wendy, we work together well.

[nuggetsyl]

Originally Posted by dtengineering

A low team number means absolutely NOTHING when it comes to predicting the team's performance in qualification rounds.

Now that I've got your attention, allow me to explain. I was inspired by discussion in this thread "Random" match Schedules discussing the apparent lack of randomness in assigning alliances for qualifying matches. It was stated that FIRST intentionally schedules matches so that alliances have roughly equal seniority, based on the assumption (presumably) that this will lead to a more balanced match.

That didn't seem fair to me....

So I downloaded the results from the first weekend's competitions, and dropped them into excel. Based on these 270 data points there is abolutely NO relationship between team number and seeding. The teams numbered below 300 had an average seed of 26, while teams numbered above 2000 had an average seed of 27.8. Hardly an advantage (although, admittedly I didn't calculate exactly how much difference would be required to be statistically significant here). Running a correlation coefficient over the whole data set shows a .007 coefficient of correlation between team number and seeding... and in some regionals the coefficient is negative (but also insignficant.)

This leads me to three possible hypothesis:

1) We are all mistaken about the advantage that senior teams have. Hey, I've done it... you've done it... and now it is alleged that FIRST is doing it. We're looking at a few very successful veteran teams and saying "wow... watch out for the teams with low numbers" and completely forgetting that for every veteran out there rocking the rack that there is another one struggling with their design, and a rookie team that is doing even better.

2) Veteran teams do have an advantage, but something is being done to prevent that advantage from helping them to win matches. It is possible that a scheduling system that pits veterans against veterans removes the legitimate advantage that comes from years of hard work and development.

3) My statistical analysis is incorrect or incomplete. I'm always willing to admit the possibility that I might be wrong. I challenge anyone, however, to prove that veteran teams have done signficantly better in qualification matches than newer teams. Math, here, please people... not anecdotes. We humans are really good at seeing relationships that don't really exist.

I'm open to suggestions, and to someone willing to examine finals matches and outcomes to see if veteran teams have any statistical advantage on Saturday afternoon... or to someone who can find some better predictor of success (perhaps previous year's rankings can be a reliable indicator) but until then.... I say that veteran teams have no advantage when it comes to winning matches and that a scheduling system that (allegedly) uses team numbers as a factor is not only unjustified, but unscientific.

Jason

P.S. Yes, veteran teams (my gosh... we're one now, I think...) have many advantages (and challenges) that junior teams might not have... but not ones that significantly affect the outcomes of qualifying matches.

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

[Jeremiah Johnson]

Originally Posted by nuggetsyl

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

I agree... you may also want to take into account the number of teams per regional. See how many closely numbered teams were playing other close numbered teams and then factor in the number of teams at the event. I don't know how to do that, I'm not a math person.

[dtengineering]

Originally Posted by nuggetsyl

maybe the reason why your data is so close is because teams are playing numbers close to each other. Example 11 and 25 played each other 4 times. one team is going to be seeded lower then the other and the average will always be in the middle. Your excel sheet would be better if you used last years numbers.

I don't disagree, however my analysis is based on this year's data. Please see hypothesis two concerning it's interpretation.

I'm looking forward to seeing an analysis of last year's numbers (it sounds as if someone is running that right now...). If lower numbered teams did significantly better last year relative to newer teams than this year, and FIRST has implemented a new scheduling system, then I would say that the scheduling system has served to undermine years of hard work and development on veteran teams.

"IF", however, is a pretty big word. I'll wait to see the results. Personally, until I see otherwise I think the "low number=power house" hypothesis is based on the same irrational pattern recognition that makes us silly humans believe in lucky rabbit's feet and horseshoes.

Jason