Seeding Calculations

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/28/99 8:23 PM MST

At the team forum someone suggested what I think is a great idea for calculating seeding points:

Tossing out the highest and lowest seeding point scores prior to calculating the average seeding points per match.

One slight variation on this theme would be to only toss out the lowest score in order to prevent a team from losing the benefit of a perfect score.

I LOVE these ideas.

Tossing out the highest and the lowest is the best method in my opinion (actually, I propose that an even better method would be to toss out the top and bottom 1/6th of matches, counting only the middle 2/3rds of matches in the seeding calculation).

My reasoning is that the middle scores are much more likely to reflect a team’s actual contribution to an alliance than atypically high or low score.

But, ‘Our team would have lost our best score!!!’ you say. Well yeah, but then you would lose your worst one as well.

Beyond this, If your robot has a lousy run, you have a chance to make it up.

To counter the idea that a perfect score wouldn’t count, I argue that it would ‘count’ because then your next highest score would count whereas without the perfect score that next highest score would have been tossed out. In addition to this I would say that a perfect score (or even a very high score) would not be wholly in vain in any case because it would get your team noticed. This notice would likely translate into an invitation to the elimination matches even if it did not mean you qualified outright.

It would be much more difficult for a team to have a single monster score put them into the elimination rounds. Equally good in my opinion, a single 0 score would not drop a team from the standings.

The rankings would be much more stable as a result.

Better seeding calculations would be better for FIRST.

What do others think about this idea?

Joe J.

Posted by Jon.

Engineer on team #190, Gompei, from Mass Academy of Math and Science and Worcester Polytechnic Institute.

Posted on 8/29/99 8:40 PM MST

In Reply to: Seeding Calculations posted by Joe Johnson on 8/28/99 8:23 PM MST:

yes! i like this idea… the 1/6ths part is still sinking in, but it’s like a statistical approach… you always take out the best and worst in order to get a more realistic number…
that’d be cool… true, if people had only one perfect round it would take it out, but i’d help pull the chaff from the wheat (is that how the expression goes?)

jonathan

Posted by colleen.   [PICTURE: SAME | NEW | HELP]

Other on team Actually, I’m a FIRST-aholic from an undecided team for the year 2000 formerly of Team #126 sponsored by Nypro and Clinton High School.

Posted on 8/29/99 10:05 PM MST

In Reply to: Re: Seeding Calculations posted by Jon on 8/29/99 8:40 PM MST:

i’m thinking hard on this one, and not really sure which way to vote for good reason to do with how my team ranked this year…

first, at regionals, we won all 8 of our qualifying matches yet only ranked 5th…after playing 6 matches, we weren’t even close to being in the top 8, and it was only after a perfect score and a 432 in our last matches that we made it… our scores had been all over the board, going as low as 16 points but still winning- we had no real consistency or average score because we weren’t consistent with our partners ability to perform… in the case of taking the average scores… we probably never would have made the top 8, and hey, we won that regional, chances are that wouldn’t have happened

the nationals: get this, we lost, i believe, 3 of our 6 matches (it may have only been four, but i’m pretty sure it was 3- don’t quote me) yet we ranked 2nd, and 2nd to TJ2 but not all that much… there we had a perfect score as well, but the rest of our scores were pretty balanced in the ranges from 100-300… had an average been taken there, i’m sure we still would have made the top 16, maybe not ranked as highly, but our scores were more consistent and it would’ve been a more fair ranking…

so maybe that’s it, the definition of fair… after driving our robots for 3 years, i feel this year we had the most efficient and consistent robots of any of our past years, little problems and constant performance of what we wanted to do… we were as consistent, scores weren’t, and that kind of scoring of seeding at the regionals would’ve killed us, and at nationals probably put us in an more appropriate seed in regards to the amount of matches we won…who knows…

what’s my point? i guess if you change the way you score, maybe something else should be done about the multipler for winning the match…although i liked the idea that you don’t have to win every match to rank high, it seems like there could be more of a balance between scores and matches in order to seed teams… i don’t know… that’s what i’m thinking…

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/30/99 5:16 PM MST

In Reply to: Agree and disagree… posted by colleen on 8/29/99 10:05 PM MST:

Colleen,

The situation you discussed is not necessarily a fair comparison.

Don’t forget that ALL teams would have had their top and bottom scores tossed out.

This means that you can’t say things like ‘…it was only after a perfect score and a 432 in our last matches that we made it…’

I would venture that you would have made it because there could not have been very many teams with two scores above the 400 mark.

But even that is conjecture.

Unless FIRST (or someone else with all the scores in a database somewhere) recalculates ALL the seeding using the new formula, it is very difficult to say.

I really believe that the rankings would have been much more reflective of each team’s ‘true rank’

If anyone has all the scores from the Nationals in an Excel spreadsheet or an Access database, I would love to post a round by round comparision of the top 30 teams.

It would be very instructive.

Joe J.

Posted by colleen.   [PICTURE: SAME | NEW | HELP]

Other on team Actually, I’m a FIRST-aholic from an undecided team for the year 2000 formerly of Team #126 sponsored by Nypro and Clinton High School.

Posted on 8/30/99 7:13 PM MST

In Reply to: not a fair comparison… posted by Joe Johnson on 8/30/99 5:16 PM MST:

i agree with you there- i understand all teams would have the top and bottom scores dropped… and yes, i agree that it would leave us all with a more fair seeding… i guess my disagreement with the whole thing lies in the reward of a multipler for winning the match, maybe that’s a little much in unbalancing the scores in way… i’m not sure…

i have to say, i liked the way seeding went better this year, that it didn’t totally rely on the amount of matches won, but something seemed a little odd in the way some of the seedings worked out, and maybe dropping the tops and bottoms would fix that, and/or doing what Jason said and adding 300 or so points instead of tripling… because when you win a match, with say 100 point and then another with 150pts., the difference isn’t that large, but when you triple it and it becomes 300 and 450, that’s quite a point gap… at least if you only added 300, the difference in your own scores would still only be 50points (cause it seems many teams may have scored in a close range when you drop the top and bottom, but when you triple those scores, it spreads it way out again)

anyway… confusing as it is… that’s what i’m thinking…

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/30/99 7:33 PM MST

In Reply to: agree w/ that posted by colleen on 8/30/99 7:13 PM MST:

I know that I am going to lose a lot of folks here, but let me give it a shot anyway.

I think that there is a wide spead feeling that defensive teams did not get enough credit in the seeding matches given the current qualifying point scoring system.

So…

I propose a system where high scoring teams and tough defensive teams have a more level playing field.

So here goes:

Let’s suppose that a teams qualifying points were comprised of 2 parts,

The first part would be your team’s (or alliance’s) score whether or not your alliance won or not

The second part would be a winning bonus based on either how statistically ‘outstanding’ your offense or your defense was that won the match.

Perhaps this bonus could be calculated as follows:

Teams would get a bonus equal to the average score scored in the seeding matches multiplied by:

1

  • the number of standard deviations your score was above the average score
    (no negative bonuses so enter 0 if you scored below the average)
  • the number of standard deviations your opponent’s score was below the average score
    (no negative bonuses so enter 0 if your opponent beat the average)

This formula would give high scoring teams lots of bonus points and also give great defensive teams lots of points.

An example (good offense):

Suppose after round 1 the average score is 100 points and the standard deviation (look it up if you don’t know what it is) for the scores is 33.3 points.

A team that wins a match 200 to 199 would get

200

+100 (the average score) X
(1
+3 (the number of standard deviations 200 is above the 100)
+0) (their opponent scored more than the average so they get a 0 for their defense)

600 points - not bad

Another example (good defense):

same average and standard deviation as above

A team that wins a match 20 to 0 would get

20

+100 X

(1
+0 (they failed to beat the average score)
+3) (they held their opponents to a score 3 standard deviations below the average)

420 points - pretty good compared to last year’s score of 60 points

A final example (good offense and defense):

same average and standard deviation as above

A team that wins a match 166 to 34 would get

166

+100 X

(1
+2 (winning score is 2 standard deviations above the average)
+2) (they held their opponents to a score 2 standard deviations below the average)

666 (a wicked score by all accounts)

Complex: yes, but not beyond the typical FIRST team to deal with

Fair: I guess that it is more fair than the current system because it rewards good offense and good defense (though offense still has an edge but not as big of one)

I think that it would make teams more conscience of where their team is scoring compared to the average. I think that that is a better measure.

Thoughts?

Joe J.

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 8/30/99 8:04 PM MST

In Reply to: statistical bonus points posted by Joe Johnson on 8/30/99 7:33 PM MST:

#1 Now this is what i like to see…a truly SIMPLE scoring system… :wink:
no…really…i like it…its got lots of good ideas…but…

#2 it’s nice and easy to look back and say…ok…the average at nationals was 112.3 pts with a standard deviation of
34.2 pts…and start calculating away…but…lets rewind…we’re sitting at nationals…you just completed your first match
and you won 112-36…you say great…we got…uhh…uhmmm…well…i guess we won’t know how many qp’s till the end of
all the matches on saturday…great…we have no idea where we rank among all teams…none at all…

i’d also like to repeat your request for a spreadsheet with allt he scores from nationals in it…heck…even if its not in a
spreadsheet…just anyone with all the scores…i tried to get them right after nationals and no one seemed to respond…
but if we had that we could do all kinds of possible scoring systems and see what teams come up where, etc…wanna
talk about some real math fun… :slight_smile:

my two cents
Tom

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/30/99 8:19 PM MST

In Reply to: well… posted by Tom Vanderslice on 8/30/99 8:04 PM MST:

Tom,

Yeah, I never said it was simple :frowning:

As to what teams are scoring, doubt that the average would change much after several rounds into a tourney (or if it did, it would change pretty gradually).

FIRST could recompute the scores with every match. Recomputing the new average score and the new point totals that all teams have.

As long as teams keep watching the leader board, I think that teams could deal with it.

Most teams leave the point calculations to FIRST anyway, so what difference would it make to a team how complex the FIRST spreadsheet was, as long as it was ‘fair’?

All that teams would have to focus on would be the fact that keeping your opponent’s score as low as possible is good and that scoring as much as possible is also good.

One thing I fear is that teams would be tempted to slaugher a weak team in order to maximize their points.

Can you imagine the carnage if teams had an even greater incentive to pull teams off the puck!?!? It might have been too brutal to watch.

Joe J.

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 8/30/99 9:10 PM MST

In Reply to: running averages… posted by Joe Johnson on 8/30/99 8:19 PM MST:

we really need last years scores…please…anyone…shouldn’t FIRST have them somewhere…even if its a computer stored
back somewhere that they have forgotten about…b/c with those…we could take the average after 2-3 rounds and just see
what happened…run a simulation under that scoring system or whatever…

but if you think about it…after say 100 matches…a score over or under the average by 100 pts would affect teh average a
full point…at 200 matches…half a point… (approx)…i don’t think there would be a problem with computing running averages
as long as all teams understood what it meant and it was fully explained to everyone…b/c i can see some teams getting
very confused…and also…FIRST couldn’t have their scoring system crash in the middle of the last round (or whenever that
was late inte h competition)…b/c the score is a lot more important than…well did this team that was ranked 14th win
they’re match…waht did they get…how bout the teams around them…but anyway…i think doing a running average has
potential…

tom

Posted by Dan.

Student on team #10, BSM, from Benilde-St. Margaret’s and Banner Engineering.

Posted on 8/31/99 2:24 PM MST

In Reply to: running averages posted by Tom Vanderslice on 8/30/99 9:10 PM MST:

Don’t get too wound up with a new scoring system, chances are things will change quite dramatically as Dean and Woodie refine the alliance idea (I’m assuming it will reappear in some form.) Imagine trying to develop scoring solutions two years ago with no idea that alliances would be formed the next year. And besides, any process of refining a scoring system until the teams you think should be in the top 10 are there doesn’t hold a lot of scientific merit.
I agree though, something needs to change.
:-Dan

Posted by colleen.   [PICTURE: SAME | NEW | HELP]

Other on team Actually, I’m a FIRST-aholic from an undecided team for the year 2000 formerly of Team #126 sponsored by Nypro and Clinton High School.

Posted on 8/30/99 9:03 PM MST

In Reply to: statistical bonus points posted by Joe Johnson on 8/30/99 7:33 PM MST:

Dr J

i’m definitely liking your idea. personally, i don’t think fair scoring is going to be simple unless you go back to your plain old score being the score, no multiplers, no bonuses, no nothing… and i think that it’s in general agreement that we want something…

and something like what you said would definitely be the answer… so what that it’s not too simple, i really think that could work… maybe after each match you couldn’t figure it out to a T, but you would have an idea based on how well you scored and how well or not your opponent scored, plus, like you said, whether we won or lost, you would never find me trying to figure out our new average, you would find my team watching the leader board waiting for FIRST to tell us…

one question though, which maybe you explained and i missed, even if we play with alliances, would the scores be individual? like the alliance score plus your deviation is what you get credit for, and your partner get’s credit for the alliance score and their deviation?

and, no promises, i think our team has all the scores from nationals, i’m working on finding out and will let you guys know once i do…

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/31/99 7:35 PM MST

In Reply to: liking it… posted by colleen on 8/30/99 9:03 PM MST:

As to alliances, I think that qualifying points would go to both (or all?) teams in the alliance, just as was the case in 99.

The difference would be that if we toss out the high and low scores they may or may not be counted in the average qualifying points per match calculation.

For example an alliance may score 400 points for their win but this may be the highest score for one team and thus not be counted in their total while it was not the highest score for the other team and therefore was included in their calculation.

As to teams keeping track of what they need to score or what they need to hold their opponent to in order to get lots of bonus points, I propose that FIRST periodically post the current average score and the current standard deviation.

In this way teams could get a clue as to what type of scores they need. Also, I would advise FIRST to try to post the qualifying point total that that the teams scored in addition to there match total (though perhaps on another screen – too much info already to digest).

Joe J.

Posted by Jon.

Engineer on team #190, Gompei, from Mass Academy of Math and Science and Worcester Polytechnic Institute.

Posted on 8/31/99 10:05 AM MST

In Reply to: statistical bonus points posted by Joe Johnson on 8/30/99 7:33 PM MST:

[poof!]
a: what was that noise?
b: paradigm shifting without a clutch

no disrespect to dr.j cause the idea is sound and makes sense, but whatever happened to scores that could be calculated quickly and don’t need a degree in Statistics and Probability to be understood?

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/31/99 7:51 PM MST

In Reply to: Re: statistical bonus points posted by Jon on 8/31/99 10:05 AM MST:

I agree with you that scores must be simple to calculate at the end of a match.

This is not the same as saying that the ranking system has to be simple.

Because we have a limited sample size and because individual team performances are confounded by factors (not the least of which is the strength or weakness of you alliance partner(s)), I think that it is fair to try to get the best predictor of ‘true rank’ that we can.

I am not a statistical wizard, but I am concerned about finding a formula that we can all live with (and perhaps even explain in broad brush terms to our grandmas) while at the same time giving the best results we can get from our confounded, limited sample size.

Those who are statistical wizards notice that I did not propose some sort of performance index combined with a statistical correlation algorithm, which by the way I think could be an even better predictor of ‘true rank’ but I don’t dare try to explain to my grandma :wink:

Joe J

Posted by P.J. Baker.

Engineer on team #177, Bobcat Robotics, from South Windsor High School and International Fuel Cells.

Posted on 8/31/99 10:43 AM MST

In Reply to: statistical bonus points posted by Joe Johnson on 8/30/99 7:33 PM MST:

Joe,

How many digits of precision on the # of std. devs.? With tenths ot hundredths, it would probably work o.k. Whole numbers only would create an artificial difference between teams that were just one side or the other of the standar dev.

I think that you are right on the money with your prediction of what would happen if there was an incentive to keep your opponents from scoring any points in the seeding rounds. Weak teams would be punished. There would be lots of broken bots in the seeding rounds, leading to many hurt feelings.

On this discussion in general, I would like to make the following comment:
The scoring system devised by FIRST, whatever it may be, is part of the design challenge that we are given. As long as the challenge is one that can be solved by almost everyone, not just the teams endowed with the most resources, the scoring system is inherently fair. We all made decisions about our robot designs after the rules of the game were released this year. Some of us decided that a defensive robot was the way to go, even if it was clear from the rules that this would make it difficult to seed high. Some of us used drive systems that were developed in the off season that were not easily adaptable to this year’s puck climbing requirement, and some built robots that could both score and play defense (Ace’s High comes to mind). Robots from all three classes did very well at the competitions this year. If we want to ask FIRST for a contest that is more balanced between offense and defense, that is one thing, but I think that as long as FIRST does not change the scoring rules after the design cycle is started, they should be judged as fair.

Posted by Dodd Stacy.

Engineer on team #95, Lebanon Robotics Team, from Lebanon High School and CRREL/CREARE.

Posted on 8/31/99 3:08 PM MST

In Reply to: Re: statistical bonus points posted by P.J. Baker on 8/31/99 10:43 AM MST:

I think PJ has it. The game’s the same for everyone, and we all know what the rules are. I hope FIRST will keep their focus on the larger objectives of the program, and that requires exponential growth for the foreseeable future. We’ve said it here before: that means media, spectators, and lots of local competition. I personally think that transparent scoring is essential to sustain excitement for all parties. Complex scoring rewards a team with a sharp scorekeeper/strategist when the general level of robot execution competance is primitive across the field. Those days are gone. The sophistication level of many machines today is amazing, and the teams who master them can ring up perfect multiplied or exponential scores. Random elements of the game though can punish severely, too, leading to cries of unfair.

I would urge FIRST to design games that have transparent scoring and balance the incentives for robot offense and defense, while keeping the human player influence from dominating the game. I would also urge FIRST to design games and rules suitable for playing on a basketball court, without tearing it up. This would mean games with NO tipping over, and NO marking the floor - different, but just a new set of rules for everyone. With the benefit that local school on school matches could leverage off the basketball season - instant spectators and student body identification with the school’s FIRST team, which I suspect is as lacking at other schools as it is at ours.

Finally, at the risk of irritating a lot of you, I think this angst over seeding is short sighted. FIRST is straining at the seams to give all the teams, especially at the Nationals, an adequate chance to play yet still manage the length of the tournament. Everybody got to play their 6 qualification matches this year, nobody got bumped for good by one match loss or two. Then the field gets narrowed for the Eliminations, with the alliance choosing correcting the process imperfections a bit. We discuss perfecting the process 4 months later, but this is a moving target. The issue doesn’t change. How will FIRST adapt the competition to exponential growth in the number of teams? The answer this year, with 200+ teams in Orlando, was 4 teams on the court and downselecting for a single elimination tournament via qualiers. Can we tweak the scoring system to make it work next year with 300 teams? 400? FIRST is on the knee of its exponential. What does it look like with 2000 teams?

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/31/99 7:58 PM MST

In Reply to: First Principles posted by Dodd Stacy on 8/31/99 3:08 PM MST:

Dodd,

Well said as usual.

While I am not really terribly unhappy about last year’s method of seeding, I am always trying to push for a better method.

I have to agree with you that the real challenges FIRST faces in the coming years have very little to do with seeding calculations.

Yet, I would like for the seeding to be better. In that spirit I have made some proposals for consideration.

Joe J.

Posted by Bethany Dunning.   [PICTURE: SAME | NEW | HELP]

Coach on team #163, Quantum Mechanics, from International Academy and Quantum Consultants/EATON/ITT Industries.

Posted on 9/1/99 2:45 PM MST

In Reply to: well said posted by Joe Johnson on 8/31/99 7:58 PM MST:

I like the general concept, except for one thing: at both competitions I went to last year,
each team only got 6 qualifying matches. Not enough. And when you drop the highest
and the lowest, you are down to 4 matches. I’d be a little apprehensive about having all
that ride on 4 matches.

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]

Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 9/1/99 6:56 PM MST

In Reply to: Re: well said posted by Bethany Dunning on 9/1/99 2:45 PM MST:

But it isn’t just a random 4 matches from your matches it is the middle 4.

By dropping your highest and lowest you loose both extremes, both the time you were very lucky (when your partner was great or your opponents we lame) AND the time you were very unlucky (your robot broke, your alliance parter was a box with wheels, you played Chief Delphi ;-).

The middle is much more likely to represent your team’s true ranking, whether there are 6 seeding matches or 60.

Joe J.

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 8/31/99 6:51 PM MST

In Reply to: Re: statistical bonus points posted by P.J. Baker on 8/31/99 10:43 AM MST:

I’ve been having fun messing with the scoring system (see posts below)…but i couldn’t agree more…no scoring system is ‘fair’ or ‘unfair’ as long as everyone knows exactly how it is scored before they start adn it doesn’t change in teh middle…that’s all part of the design strategy and building…but i think the goal of making up a new one is to create a system that doesn’t so blatantly favor offense or defense…

Tom