FIRSTStar(TM) Rating

What a great compilation of information!

Kudos to Aaron!

p.s.

I did notice one tiny error (not that anyone would expect such a large collection of information to be void of errors) Team 67 did not win the Rookie All Star award at the Great Lakes Regional in 2002, we were awarded the Engineering Inspiration Award.

Thanks for the heads up about the mistake. I just realized - and it makes more sense that 903 got the Rookie All-Star and not 67.

Let me know if there are any other mistakes and I’ll update the file. I’ll also make it a bit easier to update (I was in a hurry so I didn’t really perfect it).

Luckily, it didn’t change anything (not that I could tell).

-Aaron

A very interesting start.
I agree with basing it more on the number of competition events, rather than years in competition. Also, after season competitions could be interesting to add.

For my own personal amusement, I calculated my own personal FIRSTStar™ Rating of 3.75 based on the three teams I’ve been a part of the last four years. Could make an interesting CD Forum membership rating system (to give Brandon a little more work).

Originally posted by Aaron Vernon *
FIRSTStar Raw Score = 5 * (Group 1 awards) + 4
(Group 2 awards) + 3*(Group 3 awards) + 2*(Group 4 awards) + 1*(Group 5 awards)

Group 1 awards = National Champion, Chairman’s Award Winner
Group 2 awards = Regional Chairman’s Winner, Chairman’s Award Finalist
Group 3 awards = Regional/Division Champion, National Finalist
Group 4 awards = Regional/Division Finalist, Technical awards (Creativity, Quality, Controls, Tomorrow’s tech, Ind. Design), #1 Seed, and Highest Rookie Seed
Group 5 awards = everything else

After you have the raw score, you divide that by the number of years that a team has competed. This gives you your FIRSTStar rating.

I think all the nationals judged awards deserve more points.
Furthermore your data is missing the J+J sportsmanship award for 2000, which we (team 108) won.

Sounds like we need to make ourselves an SQL or Access database with all this info. Then people can run their own queries. I’m willing to help.

I like the idea of normalizing by the number of competitions.

How about also normalizing by the number of teams at a regional?

I am thinking that perhaps some metrics would have this option included and some would not.

So… Winning a 75 team regional would be weighted somewhat more than winning a 33 team regional. I am struggling with what the weighting should be though – I don’t think that I would make it worth (75/33) as much but it seems that it should be worth somewhat more.

I think the main thing is that we should get the data in a format that lets folks propose various metrics, then we could all compute that result of what is proposed and we could see if we agreed with the basic trend.

I have another tricky question. What about teams that sort of split up into several teams? Should the new teams get the credit for the old team’s performance or should they start off fresh? I am leaning toward a “New Number, New Stats” policy. Any other ideas on how to handle this?

Joe J.

I don’t think the “New Number, New Stats” policy would work. Some teams remained almost exactly the same, but they changed sponsors, and under the old FIRST rules, changed numbers. Is this right? If my understanding is correct, then these teams have quite a bit of history that would not be accounted for, which I thought was the entire point.

*Originally posted by Mark Hamilton *
**

I think all the nationals judged awards deserve more points.
Furthermore your data is missing the J+J sportsmanship award for 2000, which we (team 108) won. **

I’m not singling out your post, but I think we should hold off trying to correct the data.

The reason I say this is that I found myself composing a very similar letter myself. (A few awards were missing from my team as well)

Before we get into the Data integrity stage, lets work the format out. I think this a good “test” set of data. Let’s leave it as it is, and concentrate on getting a final form. I think the excel sheet is great, but for this to be truly viable, it would most likely need to be a database back end, serving web queries up on the fly. Although even as I write this, I think why does it have to be dynamic. Most of the content is static for 10 months of the year. It would only change during competitions. The only case I could make for dynamic data is comparing one team against another.

My only concern is this,
I really hope people don’t become too enamored by this. My team is a good characterization of this. We’ve had good years, and we’ve had off years. Thankfully, the good years, (4 of them five if you count this year) out weigh the bad years (2).

*Originally posted by Joe Johnson *
**

I have another tricky question. What about teams that sort of split up into several teams? Should the new teams get the credit for the old team’s performance or should they start off fresh? I am leaning toward a “New Number, New Stats” policy. Any other ideas on how to handle this?

Joe J. **

Conversely, how about teams that have ceased to exists. (Plymouth North for example)

How do we take them into account. Do we delete them, or standby them? Keep in mind teams also resurface. (TigerBolt)

I say we keep teams that are no longer active in the FIRSTStar™ system.

Some day (a few decades from now) Chief Delphi may retire – if they do, I want folks to have to a benchmark to hold themselves up to

Joe J.

another thing for this database might be consistency, how many competitions has a team won in a row. Team las Guerrillas has won 5 so far in a row, and looking for a 6th (nationals)

Champions- Cocomo Indiana-2001

Champions-for sweet repeat 2001

Champions-chief delphi invitational 2001

Champions-buckeye regional 2002

Champions-great lakes regional 2002

-TheChosun

Now that the ratings are coming together very nicely. Could they be used by FIRST to influence the pairings during seeding and division groupings during the competition? I know past performance may not be indicative of future results but does anyone think 111, 47 or 16 are going to put out a crummy bot next year? We played some robots 2 and 3 times at KSC. Can you imagine how skewed the seeding results would be if some team got stuck playing Bomb Squad 3 out of 9 matches? If we have a past robot performance metric, then the sum of that metric for one alliance pair should be approximately equal to sum of their opponents in a given match. That way, if you are a medium performing robot and the computer randomly pairs you with wildstang in one match, they should have your alliance oppose an alliance with a medium performing robot and say 47. This way seeding would be much more performance based and not as much the luck of the draw. (BTW, I’m not saying all the high seeds didn’t perform well but you have to agree that for many teams there is a lot of luck involved).

This could also be used to even up the competitiveness of division. The summation of the FirstStar ratings in each division should be the same.

The only problem with this is your team gets labelled and may get stuck competing in a certain class for which you may not be suited that particular year. In other words, if you had stellar performance in the past but turned out a bad bot one year you would be competing with all the other bots that had a history of stellar performance. In other words you would get creamed. On the other hand if you had a rather poor history but turned out a great bot one year you would rise to the top like a rocket!

Hmmmmmmm.

James Jones
Engineer/Coach
Team 180 SPAM

*Originally posted by JamesJones *
**…

James Jones
Engineer/Coach
Team 180 SPAM **

I don’t think using the previous year’s results to set up matches is a wise idea. I’d rather go with random pairing. I don’t know about anyone else, but I wouldn’t want to get paired unfairly to other teams just because we did bad one year.

I definitely would not want to have pairings based on previous years ratings. It seems to me that this would place teams that will likely be strong (47, 111, 308) in much more difficult matches. The result would be that teams which have weaker records get easier matches, most likely more points, and thus higher rankings. Sure, this makes it nicer to rookie teams, but it makes the seeding/qualification ranking system somewhat obsolete, as you do not have a list of the best robots…do we really want to punish our strong veteran teams??? And do we want finals/eliminations to be full of teams with weaker schedules??? I like the random pairings, i agree that sometimes a team can get hit with a couple tough matches, but fixing the matches in this way, i feel, would only make the situation worse

I think a better use of data of this type would be to try to ensure that each team in a division had a relatively equal schedule. I know the system would not be perfect but ANY effort to eliminate some of the “luck of the draw” effect would be a step in the right direction.

Even something based on a teams years of experience would help. (every team would play with, and against roughly the same number of rookies, 2 year vets, 3 year vets etc.)

I think a system like this could go a long way towards having the best teams seed at the top. (We’ve all seen a few teams who seeded maily because they were lucky enough to have good partners.)

we were higher then i thought we would be on the list. and a good job putting that toghather must have took a lot of hard work, or a lot of free time;)

im sorry but the only way to be fair is to keep the matches random…i can think of a whole lot of complications that could arise by trying to match up teams by quality

first of all, the highest scoring rounds are the ones with the highest quality robots… and out of that match one of the great robots has to lose

even when a not so good team gets carried, most of the time they cant hold the position, and if they do, they usually dont last in the finals

besides isnt this what its all about? working with everyone no matter what