FIRSTStar(TM) Rating

[mpking]

Originally posted by Joel J.
Yea, about that page .. didn't you guys go to the NE regional?

Umm..... Yea.

Umm.. It there, honest. Just don't look at the file date.

Hehe. Darn topigraphical errors :-)

[Aaron Vernon]

I obviously have too much time on my hands, but here is some of the information that you asked for, Joe. I took the results from the FIRST website from 1998 through 2002. I didn't add anything before 1997 because the team numbers changed between '97 and '98, and I have yet to sit down and match old team names to current numbers. Either way, this is a snapshot of the last 5 years of FIRST.

The spreadsheet has 5 worksheets:
Totals - by Award - This lists all of the winners of every award from '98-'02. (Note that the most of any award that a single team has won is 6. Teams 47 & 254 have won 6 Regionals, and team 67 has won 6 Leadership in Controls Awards.)
Totals - by Event - A list of the winners of every award by event. This allows you to see how often a certain team has won an award at a specific event.
Totals - by Team - A list of every award that each team has won. Note that three teams are tied with 23 awards (teams 16, 47, and 67).
FIRSTStar rating - This is the pivot table with all of the information you could want. This totals up the number of awards each team has won. Also used to calculate the FIRSTStar rating (see below for equation).
FIRSTStar ranking - This ranks all of the teams who have won awards (excluding scholarships, Autodesk awards, and single-person awards) by their FIRSTStar rating.

FIRSTStar Equation
The FIRSTStar rating is a weighted average of a team's awards divided by the number of years that team has competed. Thus, a team who has 10 awards in 3 years would rank higher than a team with 10 awards in 5 years.
The key, however, is the way that I took the weighted average. Using some guidelines from how FIRST calculates eligibility for the Championship I came up with an equation.

FIRSTStar Raw Score = 5 * (Group 1 awards) + 4*(Group 2 awards) + 3*(Group 3 awards) + 2*(Group 4 awards) + 1*(Group 5 awards)

Group 1 awards = National Champion, Chairman's Award Winner
Group 2 awards = Regional Chairman's Winner, Chairman's Award Finalist
Group 3 awards = Regional/Division Champion, National Finalist
Group 4 awards = Regional/Division Finalist, Technical awards (Creativity, Quality, Controls, Tomorrow's tech, Ind. Design), #1 Seed, and Highest Rookie Seed
Group 5 awards = everything else

After you have the raw score, you divide that by the number of years that a team has competed. This gives you your FIRSTStar rating.

FIRSTStar Rating Top Ten
1. 16 (Even if Bomb Squad had competed last year and won nothing, they would still be in 1st place!)
2. 47
3. 67
4. 254
5. 71
6. 343
7. 175
8. 365
9. 111
10. 945

FIRSTStar Stars
I added a relative score from 1 to 5 stars to the spreadsheet. The calculations are as follows:
5 Stars- Greater than or equal to 5.0 FIRSTStar rating
4 Stars- Between 2.5 and 5.0 FIRSTStar rating
3 Stars- Between 1.0 and 2.5 FIRSTStar rating
2 Stars- Greater than 0 and less than 1.0 FIRSTSTar rating
1 Star - Zero FIRSTStar rating (i.e. no awards)

This should be a good start for discussion as to which teams are the most successful. We could use this as our own FIRST RPI poll to use in "handicapping" teams during the season. Luckily, we won't base who wins the Championship on what the computers say.

-Aaron Vernon
Team 224 - PSGA/Piscataway HS

[Jason Morrella]

Aaron,

That clearly took a lot of time and work - you did an excellent job! I can't believe that a first attempt could be so good. You included many things I would have overlooked. After looking at the criteria, there are only a couple things I could think of which might be worth discussion. Let me know what you think of the following observations/ideas.

Items I might add/change:

Should the average be further broken down by number of events attended in addition to years in the competition?

As the USA Today does with High School Sports Teams. Maybe have a National top 10 or 20, but also have it further broken down with a regional top 10 or 20 so that more teams can be recognized?

Personally, I think ALL awards should be included in group 4. I think rookie all star awards, sportsmanship, judges, spirit, etc... are of the same value as the technical awards and that those teams should be given the same credit. The qualities of those awards are very much responsible for making FIRST a "different" type of competition and really set FIRST apart - on a level unrivaled - from other programs.

I would also include the other other technical awards FIRST has and is starting to offer, such as the animation. These are important aspects of FIRST and the students on those sub teams learn valuable skills and put in just as much time as those building the robot.

Joe - great idea. All the sports teams at high schools around the country check the USA Today to see how they compare in regional and national rankings with other teams, now FIRST teams can do the same.

Aaron - great job putting these ideas into a working system.

[Joe Johnson]

A very good start.

Now we need the older data and to get some multidimensional metrics. I like the 5 star rating as a general idea but I have to think about the specifics of your forumula.

In addition to the overall metric, I would also like some "Chairman's oriented" metrics and some "robot oriented" metrics.

It may even be nice to have some goofball metrics like "Greatest looking -- non-winningest robot" metrics, or most "judges" award index. I don't know, we've got time to think about it.

Getting the data is a good part of it.

Joe J.

[P.J. Baker]

It would be a little more work, but I think that the data is there. Because the data is normalized to the number of years competing rather than the number of events, the rankings are a little skewed towards teams that have consistently competed in 3 or 4 events per year. Otherwise, a great job!

P.J.

[Jim Meyer]

What a great compilation of information!

Kudos to Aaron!

p.s.

I did notice one tiny error (not that anyone would expect such a large collection of information to be void of errors) Team 67 did not win the Rookie All Star award at the Great Lakes Regional in 2002, we were awarded the Engineering Inspiration Award.

[Aaron Vernon]

Thanks for the heads up about the mistake. I just realized - and it makes more sense that 903 got the Rookie All-Star and not 67.

Let me know if there are any other mistakes and I'll update the file. I'll also make it a bit easier to update (I was in a hurry so I didn't really perfect it).

Luckily, it didn't change anything (not that I could tell).

-Aaron

[Adam Krajewski]

A very interesting start.
I agree with basing it more on the number of competition events, rather than years in competition. Also, after season competitions could be interesting to add.

For my own personal amusement, I calculated my own personal FIRSTStar(TM) Rating of 3.75 based on the three teams I've been a part of the last four years. Could make an interesting CD Forum membership rating system (to give Brandon a little more work).

Adam

[Mark Hamilton]

Originally posted by Aaron Vernon
FIRSTStar Raw Score = 5 * (Group 1 awards) + 4*(Group 2 awards) + 3*(Group 3 awards) + 2*(Group 4 awards) + 1*(Group 5 awards)

Group 1 awards = National Champion, Chairman's Award Winner
Group 2 awards = Regional Chairman's Winner, Chairman's Award Finalist
Group 3 awards = Regional/Division Champion, National Finalist
Group 4 awards = Regional/Division Finalist, Technical awards (Creativity, Quality, Controls, Tomorrow's tech, Ind. Design), #1 Seed, and Highest Rookie Seed
Group 5 awards = everything else

After you have the raw score, you divide that by the number of years that a team has competed. This gives you your FIRSTStar rating.

I think all the nationals judged awards deserve more points.
Furthermore your data is missing the J+J sportsmanship award for 2000, which we (team 108) won.

[Curtis Williams]

Sounds like we need to make ourselves an SQL or Access database with all this info. Then people can run their own queries. I'm willing to help.

[Joe Johnson]

I like the idea of normalizing by the number of competitions.

How about also normalizing by the number of teams at a regional?

I am thinking that perhaps some metrics would have this option included and some would not.

So... Winning a 75 team regional would be weighted somewhat more than winning a 33 team regional. I am struggling with what the weighting should be though -- I don't think that I would make it worth (75/33) as much but it seems that it should be worth somewhat more.

I think the main thing is that we should get the data in a format that lets folks propose various metrics, then we could all compute that result of what is proposed and we could see if we agreed with the basic trend.

I have another tricky question. What about teams that sort of split up into several teams? Should the new teams get the credit for the old team's performance or should they start off fresh? I am leaning toward a "New Number, New Stats" policy. Any other ideas on how to handle this?

Joe J.

[JVN]

I don't think the "New Number, New Stats" policy would work. Some teams remained almost exactly the same, but they changed sponsors, and under the old FIRST rules, changed numbers. Is this right? If my understanding is correct, then these teams have quite a bit of history that would not be accounted for, which I thought was the entire point.

[mpking]

Originally posted by Mark Hamilton


I think all the nationals judged awards deserve more points.
Furthermore your data is missing the J+J sportsmanship award for 2000, which we (team 108) won.

I'm not singling out your post, but I think we should hold off trying to correct the data.

The reason I say this is that I found myself composing a very similar letter myself. (A few awards were missing from my team as well)

Before we get into the Data integrity stage, lets work the format out. I think this a good "test" set of data. Let's leave it as it is, and concentrate on getting a final form. I think the excel sheet is great, but for this to be truly viable, it would most likely need to be a database back end, serving web queries up on the fly. Although even as I write this, I think why does it have to be dynamic. Most of the content is static for 10 months of the year. It would only change during competitions. The only case I could make for dynamic data is comparing one team against another.

My only concern is this,
I really hope people don't become too enamored by this. My team is a good characterization of this. We've had good years, and we've had off years. Thankfully, the good years, (4 of them five if you count this year) out weigh the bad years (2).

[mpking]

Originally posted by Joe Johnson


I have another tricky question. What about teams that sort of split up into several teams? Should the new teams get the credit for the old team's performance or should they start off fresh? I am leaning toward a "New Number, New Stats" policy. Any other ideas on how to handle this?

Joe J.

Conversely, how about teams that have ceased to exists. (Plymouth North for example)

How do we take them into account. Do we delete them, or standby them? Keep in mind teams also resurface. (TigerBolt)

[Joe Johnson]

I say we keep teams that are no longer active in the FIRSTStar(tm) system.

Some day (a few decades from now) Chief Delphi may retire -- if they do, I want folks to have to a benchmark to hold themselves up to ;-)

Joe J.