There are many struggling veteran teams. Just because they have a low number doesn’t mean there is an institutional advantage. Many of the lower number teams “recreate” themselves and go through any number of crises and evolve into a new team. Their number may be one of the few things that stays the same and the lessons learned don’t necessarily get passed on. I don’t have any hard data to back up what I just said; only personal experience.
For every statistic, there is another story.
Measuring the true capability of any team (regardless of how long they have competed) requires more than simply analyzing the outcome of the matches that they participated in.
The results may mask many other things that a single participant in any alliance would have had to deal with. Not many veteran teams can overcome, incompatible alliance pairings, mechanical breakdowns, strategic mistakes, etc. - all the things that could go wrong or right that often determine the outcome of a match.
Unfortunately, the scores and match outcomes is what is often used to evaluate (statistically) these kinds of debates/discussions.
Yes, one would assume that more veteran teams would improve over time, utilize the things that they have witnessed, attempted, failed, and succeeded at.
There are far too many variables involved to make statements one way or the other, as to whom has an advantage over whom.
One can only assume, that veteran teams have learned to deal with the challenges better than the lesser experienced team, strickly due to level of exposure.
I think that one thing you definitely need to look at in addition to what you have done so far (which seems sound) is to look at how likely veteran teams are to place in the top 8 compared to younger teams. Which is different from the analysis you have done so far. It is an interesting question though. Gives me a good idea for an assignment for AP Statistics class while I am gone for the regional this weeked.
OK, I also just took a look at some of the data and it occurs to me that your result is exactly what FIRST would be trying to achieve by randomizing matches as it did.
What makes a veteran team? Participating in one year then returning the next should qualify that statement. Or maybe being one of the founding teams qualifies at team as a veteran.
Who gives a team this seniority? Our team for example is #1038. Our student population changes every two to three years. Are we less experienced than Team #45? We have mentors who are very active on our team that have been building FIRST robots since 1994, as a matter of fact, several of these robots have been very successful in the FIRST contests - (Sunny D, 1994 National Champs, 1996 National Semi + Chairmans Award, and the list for our former team - #144, goes on until we switched schools in 1998)
The students on some teams change every year…The mentors change occasionally, too. So, is there really an advantage to participation from year to year? It totally depends on the composition of individual teams and therefore cannot be controlled in an algorithm for match scheduling that is intended to have “veteran” teams play matches with other experienced teams, and “rookies” play more rookies. We sometimes have great years, followed by some “re-building” years. Sometimes we figure the game out, other times we totally miss it. That is part of life, part of FIRST. Your team number does not imply any “powerhouse” status. That is earned by consistant performance and should not be accounted for in any match scheduling.
I agree with the title of this thread. With returning mentors and students there are always lessons learned and re-applied to following years, but with a student-centric team, the personnel changes can be drastic from year to year and are not reflected accurately by a team number.
I did this last year. I found that older teams DID have an advantage in terms of scoring, if not seeding. This was based on all the week 1 and week 2 regionals in 2006.
http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4111&d=1142127063 - Average alliance team number versus score
http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4114&d=1142139985 - Average alliance age in years versus score
And Here’s the thread, with many more charts and things.
I remember seeing a thread a while back that compared '05 and '06 success at nationals. There may be some relevant data in the graphs in that thread if anyone remembers what thread I am referring to.
I think that was me as well. I downloaded the seed results of a bunch of teams from a bunch of regionals. I normalized them so that they were between 0 and 1, and the plotted the results. A point at (x,y) means that in 2005 a team got ranking x, and in 2006 they got ranking y. This graph shows that there is little year-to-year correlation for performance. Someone who does awful one year can get as high as the top 10% in the next year. Likewise, someone who wins regionals one year can go to the bottom 10% the next year. The powerhouse teams that win consistently from year to year aren’t as widespread as people think.
http://www.chiefdelphi.com/forums/attachment.php?attachmentid=4724&d=1162868157
If anyone needs raw data from last year, I attached a zipped xml file that has all the regional stats. It can be viewed in excel. I generated it with my FIRSTXML program. I really need to update FIRST XML for this years regionals.
MASTERlite.zip (96.5 KB)
MASTERlite.zip (96.5 KB)
I’ll take a look it this soon, but you might have just saved me and another team member a lot of manual labor.
Thanks!
Guy
I found that guessing for the acronym was very difficult, so I went to google and did this:
<word from regional name> results site:www2.usfirst.org
And google will get it almost every time.
That is very interesting data and analysis, and suggests that perhaps there is a slight advantage to be gained from experience (which one might well expect) but certainly not an insurmountable one. It also meshes well with the idea that – particularly for a student centred team – there is an upper limit to how much experience one can gain. Does it make sense that a ten-year team would have any advantage over a four-year team?
I think that changes to the KOP and the availability of COTS mechanisms have also greatly helped rookies and evened up opportunities. A two-speed gearbox or mecanum wheel four years ago was something that required a lot of technical expertise… now it requires a credit card and some bolts. Which isn’t a bad thing, but it does level the playing field.
Mentoring and support also helps balance the playing field. I know our rookie neighbours (2273, who did quite well and placed 20 spots ahead of us in Portland) are using a couple of our old motors and speed controllers in their system because they didn’t have the inventory of spare parts on hand that we do. Their neighbours in the pits, team 114, not only loaned them a robot cart at first, but then built them one! (One of the many reasons 114 took home the J&J Sportsmanship award… well done, #114!) One of the great things about FIRST is that veterans help rookies, rather than pummel them.
So I look at this as follows… either:
-
Veteran teams have NO inherent advantage in terms of winning qualifying matches. The data posted on scoring from last year, however, suggests that this might not be the case.
-
The current scheduling system eliminates any advantage gained by experience by repeatedly pitting veteran teams against each other rather than by having them compete against a random selection of opponents. Assuming that this scheduling system is new this year, this would explain why last year veteran teams scored higher, but this year had not advantage in qualifying matches. (Yes, two slightly different measurments, but the best we’ve got right now.)
In any case I understand that FIRST is considering an adjustment to the match scheduling system. If they choose to eliminate the rookies vs. rookies and veterans vs. veterans part of the formula we can re-run the qualifying results from next weekend and see if there is more of a correlation between team number and seeding.
I think that would be really interesting to look at…
Jason
P.S. What a great idea to assign this as a stats project to an AP stats class… I might save this as an assignment for some of my math classes, too.
Wow, I spent half an hour looking through their site yesterday for those before I gave up and did my google method. Thank you.
Two things:
-
Off topic a bit, but every time that a thread like this pops up, we are justifying the current failure that is the match sheduling system. By saying that it levels the playing field, people from FIRST think more and more that this is acceptable. I’m going to be blunt and say it: I ama against the current algorithm (spelling?) and expect first to come up with something better that will pair us with and against a bigger variety of teams
-
There are verteran powerhouses (someone mentioned 25 and 254) but there are also rookie powerhouses. Maybe they dont win a regional every year, but every year you will see rookies that mow down veterans. 1503 won 3 regionals last year, with 1114 (both relatively new teams). It’s a fact that rookies and veterans are both FIRST teams, and given similar knowledge, they will both perform well.
Let’s let FIRST know that we want to play against everyone, rookies and vets.
It’s not veteran status. It’s not rookies status. It’s infrastructure. The teams that are well organized, well directed and are set up to ensure longevity are the ones that do well.
In order to be successful, you have to learn from experience. Rookies that have experienced mentors and enthusiastic members will do well (see 1902 for a prime example).
On the flip side, veteran teams that have no continuity from year to year are going to do poorly, or have mixed results. Not having any information on past robots can be very detrimental, as the team has to re-learn all of it’s previous mistakes. Team handbooks, dedicated sponsors and supportive communities are the marks of successful teams.
One advantage that vetern teams DO have is their lower number and their team history. I have been at many a regional where a Big Name team didn’t have the strongest robot of the litter but they were picked based on reasons other than that particular years robot performance. Sometimes it is the experience under pressure, sometimes it is the way they sell themselves, it could have even been that it was an incredibly robust machine and went out for every single match. Either way you don’t have to have the #1 seed robot to be on the winning team, you may even have the worst robot at the regional (Based on the qualifying match scores) especially this year when you see some of these matches go so lopsided when the right robots are out there together against and unevenly matched oponent.
This argument only supports the reason for a new algorithm. The less teams one is exposed to, the less quality information he is likely to have, the more likely he is to revert to picking a low-numbered team. Hence, we should revert to a random algorithm, that will let each team compete with and against more teams.
-Guy
EDIT: Results from the inference I ran on last year’s regionals results are coming tomorrow. I promise 
Our scouting team took data on the number of tetras/balls actually scored by robots at the regionals we were attending. This may be more reflective of how teams are actually scoring. The only data I have access to at the moment is the data for the 2005 Chesapeake Bay Regional. So, out of 55 teams at the regional, 25 (45%) were numbered above 1000 (arbitrary number) with 9 (16.3%) being rookies. Looking at just the top 8 (scorers, not seeds), 2 (25%) were numbered over 1000, 1 (12.5%) was a rookie team. If I pick an arbitrary number again, like 2 tetras, 36% of teams that scored 2+ tetras were numbered over 1000 and 18% were rookies.
So I’m not sure if this data is in any way meaningful, but it may be more meaningful if someone keeps data on scoring, or something along those lines, to compare that to team number (in a more meaningful way than this) or to compare scoring to seed this year and last year (for example what % of the top 8 seeds were also in the top 8 scoring this years as compared to last year) to see if this algorithm is having an effect on where the highest scoring teams seed. (I can’t do that here because I don’t know the seeds for 2005 Chesapeake)
Or it might all be meaningless. But match results definitely do not equal scoring ability (the highest scoring robot in the Curie division at 2005 Championships was seeded 11th). Although, scoring ability definitely does not equal robot quality.
Because the controversial scheduling system is in place, all this shows that the desired effect of the system was achieved. Every match a low number team won, a low number team lost.
This data is meaningless.
Also, I am in no way supporting the existing scheduling algorithm.
I would not base anything off of this year. This year is just starting and the “veteran” teams have many competitions to go to. I would personally base any information off of the championships, for any year. If you are a good team then you will strut your stuff in the finals. All teams who have a good robot will have a good team. Not all veteran teams have good teamwork, but they do have the experience of doing this kind of thing over and over. I really want the rookie teams to shine this year which is what keeps the FIRST spirit strong.
At BAE, 58 scored the most game pieces (20) on Friday. 1276 scored the next most (18). Then came 1307 (16) and 1073 (13). Then there was 175 (12).
All of these teams are at least at the end of a high school cycle. 58 was a rookie just before '98. 1276 and 1307 were rookies in 2004. 1073 has at least five years experience (maybe 6), Buzz was a rookie in '96.
I don’t see a whole lot of correlation here, other than all of these teams have at least 4 years under their belt. BAE is pretty heavy with old teams, so this data may be weighted. I would say veteran teams should have an advantage, put after your third year, you’ve had the opportunity to learn as much as you need to be competative (with experienced mentors, this knowledge should come even quicker).