Seeding Calculations

[archiver]

Posted by Dodd Stacy.

Engineer on team #95, Lebanon Robotics Team, from Lebanon High School and CRREL/CREARE.

Posted on 8/31/99 3:08 PM MST


In Reply to: Re: statistical bonus points posted by P.J. Baker on 8/31/99 10:43 AM MST:



I think PJ has it. The game's the same for everyone, and we all know what the rules are. I hope FIRST will keep their focus on the larger objectives of the program, and that requires exponential growth for the foreseeable future. We've said it here before: that means media, spectators, and lots of local competition. I personally think that transparent scoring is essential to sustain excitement for all parties. Complex scoring rewards a team with a sharp scorekeeper/strategist when the general level of robot execution competance is primitive across the field. Those days are gone. The sophistication level of many machines today is amazing, and the teams who master them can ring up perfect multiplied or exponential scores. Random elements of the game though can punish severely, too, leading to cries of unfair.

I would urge FIRST to design games that have transparent scoring and balance the incentives for robot offense and defense, while keeping the human player influence from dominating the game. I would also urge FIRST to design games and rules suitable for playing on a basketball court, without tearing it up. This would mean games with NO tipping over, and NO marking the floor - different, but just a new set of rules for everyone. With the benefit that local school on school matches could leverage off the basketball season - instant spectators and student body identification with the school's FIRST team, which I suspect is as lacking at other schools as it is at ours.

Finally, at the risk of irritating a lot of you, I think this angst over seeding is short sighted. FIRST is straining at the seams to give all the teams, especially at the Nationals, an adequate chance to play yet still manage the length of the tournament. Everybody got to play their 6 qualification matches this year, nobody got bumped for good by one match loss or two. Then the field gets narrowed for the Eliminations, with the alliance choosing correcting the process imperfections a bit. We discuss perfecting the process 4 months later, but this is a moving target. The issue doesn't change. How will FIRST adapt the competition to exponential growth in the number of teams? The answer this year, with 200+ teams in Orlando, was 4 teams on the court and downselecting for a single elimination tournament via qualiers. Can we tweak the scoring system to make it work next year with 300 teams? 400? FIRST is on the knee of its exponential. What does it look like with 2000 teams?

[archiver]

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]


Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 8/31/99 7:58 PM MST


In Reply to: First Principles posted by Dodd Stacy on 8/31/99 3:08 PM MST:



Dodd,

Well said as usual.

While I am not really terribly unhappy about last year's method of seeding, I am always trying to push for a better method.

I have to agree with you that the real challenges FIRST faces in the coming years have very little to do with seeding calculations.

Yet, I would like for the seeding to be better. In that spirit I have made some proposals for consideration.

Joe J.

[archiver]

Posted by Bethany Dunning.   [PICTURE: SAME | NEW | HELP]


Coach on team #163, Quantum Mechanics, from International Academy and Quantum Consultants/EATON/ITT Industries.

Posted on 9/1/99 2:45 PM MST


In Reply to: well said posted by Joe Johnson on 8/31/99 7:58 PM MST:



I like the general concept, except for one thing: at both competitions I went to last year,
each team only got 6 qualifying matches. Not enough. And when you drop the highest
and the lowest, you are down to 4 matches. I'd be a little apprehensive about having all
that ride on 4 matches.

[archiver]

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]


Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 9/1/99 6:56 PM MST


In Reply to: Re: well said posted by Bethany Dunning on 9/1/99 2:45 PM MST:



But it isn't just a random 4 matches from your matches it is the middle 4.

By dropping your highest and lowest you loose both extremes, both the time you were very lucky (when your partner was great or your opponents we lame) AND the time you were very unlucky (your robot broke, your alliance parter was a box with wheels, you played Chief Delphi ;-).

The middle is much more likely to represent your team's true ranking, whether there are 6 seeding matches or 60.

Joe J.

[archiver]

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 8/31/99 6:51 PM MST


In Reply to: Re: statistical bonus points posted by P.J. Baker on 8/31/99 10:43 AM MST:



I've been having fun messing with the scoring system (see posts below)...but i couldn't agree more...no scoring system is 'fair' or 'unfair' as long as everyone knows exactly how it is scored before they start adn it doesn't change in teh middle...that's all part of the design strategy and building...but i think the goal of making up a new one is to create a system that doesn't so blatantly favor offense or defense....

Tom

[archiver]

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 9/1/99 8:51 PM MST


In Reply to: statistical bonus points posted by Joe Johnson on 8/30/99 7:33 PM MST:



After realizing...hey..i've got the scores from 2 regionals from last year...i went and jsut did some quick stats on them:

chicago:
avg = 56
st. dev = 73

detroit (that is what i have it labeled as...but i must have meant michigan...i dunno..its a big list of scores thouhg...
avg = 83
st. dev = 120


so just based on this data...you can see that it would be impossible to accomplish the goal (defense gets as much credit
as offense) just b/c ther real numbers don't work out well...it would be impossible to be a full st. dev below the avg...
now i didn't run any #'s to see whether this would really benefit offense 'less'...but i think its a moot point anyway...

Tom

[archiver]

Posted by P.J. Baker.

Engineer on team #177, Bobcat Robotics, from South Windsor High School and International Fuel Cells.

Posted on 9/1/99 9:38 PM MST


In Reply to: A Hitch posted by Tom Vanderslice on 9/1/99 8:51 PM MST:



Please excuse my pun.


Because of all the doubling, tripling, etc. of the raw score (0-10), the distribution of scores will not be normal. Therefore you can not use the 'standard' formula for the calculation of the standard deviation. I would guess that a professional statistician could figure all of this out for us, but it would not be simple to explain to your Granny - unless she's a professional statistician.

P.J.

[archiver]

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]


Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 9/2/99 5:47 AM MST


In Reply to: That doesn't sound normal posted by P.J. Baker on 9/1/99 9:38 PM MST:



I was afraid of that.

I was pretty sure that it was going to be easier to get N standard deviations above the average than below the average (I was thinking more about the scores being clipped at the zero point rather than the non-normal distribution caused by the doubling/tripling -- but it is the same problem)

Ah well.

Perhaps we live with it and just say that it is a step in the direction toward rewarding defense or we keep looking.

How about a system where winning is worth a bonus of 2X the average score?

Jeo J

[archiver]

Posted by P.J. Baker.

Engineer on team #177, Bobcat Robotics, from South Windsor High School and International Fuel Cells.

Posted on 9/2/99 11:13 AM MST


In Reply to: I was afraid of that... posted by Joe Johnson on 9/2/99 5:47 AM MST:



Winners QP Formula: 200 + 100*(% of winning scores below yours + % of losing scores above your opponents)

Losers QP Formula: 100*(%of losing scores below yours + %of winning scores above your opponents)


Rewards Won/Loss record, offense, and defense. What do you think?

[archiver]

Posted by Joe Johnson.   [PICTURE: SAME | NEW | HELP]


Engineer on team #47, Chief Delphi, from Pontiac Central High School and Delphi Automotive Systems.

Posted on 9/2/99 7:33 PM MST


In Reply to: How about this? posted by P.J. Baker on 9/2/99 11:13 AM MST:



Very good. I like it a lot.

It has some of the problems of not being a fixed number that folks can get ahold until the end, but it could be made to work.

FIRST would have to 'publish' the current running average of the winning and losing scores along with corresponding standard devaitions so that teams could at least estimate what sort of scores teams they would have to shoot for in the later rounds in order to make the finals (this may actually give a lot of folks a pretty good understanding of a lot of statistical ideas

One thing I like about your formula is that there is no shame in losing to a great offensive team 512 to 500 only the extra 200 points are at stake. Also, there is no shame in losing to a great defensive team by the score of 1 to 2 again, only the 200 points. In each case the score ends up substantially 300 for the winner and 100 for the loser.

What about the case where two teams battle to (the median winning score) to (the median losing score)? Again, this would yield a score of 300 for the winner and 100 for the loser.

What about a blowout? Highest score to zero. 400 points for the winner zero points for the loser.

I am a little bit afraid of this case. I think that too many strong teams would be compelled to embarass a week team if they got the chance.

So...

How about this in stead:

Winners QP Formula: 100 + 100*MAX(% of winning scores below yours, % of losing scores above your opponents)

Losers QP Formula: 100*MAX(%of losing scores below yours , %of winning scores above your opponents)

This formula rewards a team for high scoring OR for great defense but not both at the same time. It also reduces the winning bonus so that it is on par with outstanding defense or outstanding offense

In this way a team either makes the choice to go for a high score or great defense but their is no advantage to doing BOTH.

This puts the incentives where they belong: Winning matters a lot. Outstanding performance matters a lot -- whether offense or defense.

Thoughts?

Joe J.

P.S. And one more thing that is good about your idea (and is preserved in my slight modification) is that teams have an incentive to keep playing even if they are going to loose. Even if you can't win, you still gain QP by having your opponent score less or by scoring more yourself.

[archiver]

Posted by Tom Vanderslice.

Student on team #275, ORHS/AST/Hitachi, from Academy of Science and Technology and Hitachi.

Posted on 9/2/99 9:14 PM MST


In Reply to: not bad... not bad... posted by Joe Johnson on 9/2/99 7:33 PM MST:



Ok...doing the math/some serious excel work on the Chicago Scores

Team # Real Seed PJ Seed Joe Seed
16 1 1 1
239 2 9 9
79 3 12 8
312 4 20 11
148 5 2 3
180 6 7 5
159 7 3 2
179 8 16 20
122 9 8 10
212 10 4 4
10 11 5 7
15 12 18 14
233 13 15 18
278 14 6 6
21 15 13 17
57 16 10 12
132 17 11 13
282 18 14 16
290 19 19 15
267 20 17 22
234 21 27 26
147 22 22 21
218 23 25 25
98 24 23 23
170 25 24 24
285 26 21 19
168 27 26 27

ok...real seed would be how they were seeded at teh tourney...pj seed would be under pj baker's system...joe seed would be
under joe johnson's system...

i just did this so you can look at your team (if you were there) and say...'gee...i think we should use joe's b/c it gave us a
higher seed'...or 'gee..i like hte original way...b/c we did really bad in the toher ones'...or stuff like taht...

plus...i'd rather mess with these numbers than do homework..

So have fun playing with your scores/scoring system...

[archiver]

Posted by P.J. Baker.

Engineer on team #177, Bobcat Robotics, from South Windsor High School and International Fuel Cells.

Posted on 9/5/99 12:31 PM MST


In Reply to: not bad... not bad... posted by Joe Johnson on 9/2/99 7:33 PM MST:



Joe,


I still think that I prefer mine. I think that score differential should count. That way teams are encouraged to be both offense and defense minded, rather than just picking one (probably defense). That's just my opinion though.

[archiver]

Posted by Bethany Dunning.   [PICTURE: SAME | NEW | HELP]


Coach on team #163, working on a new name!, from International Academy and Ford Motor Company.

Posted on 9/5/99 9:38 PM MST


In Reply to: not bad... not bad... posted by Joe Johnson on 9/2/99 7:33 PM MST:



One of the things that has been mentioned in here is that the game last year was too complicated,
and scoring couldn't be done until the end. I'm sure we've all had people ask us to explain the game,
seeding, etc, to them, and we replied with 'sure, got 20 minutes?' if you want to open the 'sport'
of FIRST up to the masses, draw the crowds, get the following, we can't have a seeding system that
is this incredibly complicated. yeah, it makes sense mathematically. but i can't imagine trying to
explain to my grandfather, who attended regionals to see my team, that our seeding is based on a
standard deviation blah blah blah.

[archiver]

Posted by Raul.

Engineer on team #111, Wildstang, from Rolling Meadows & Wheeling HS and Motorola.

Posted on 9/5/99 10:57 PM MST


In Reply to: Re: not bad... not bad... posted by Bethany Dunning on 9/5/99 9:38 PM MST:



As long as we're throwing out ideas, I can contribute to the absurdity:

How about having specialty rounds?
Round 1: Pure offense - team with most points get most bonus points
Round 2: Pure defense - team allowing least points gets most bonus points
Round 3: Anti-defense - team that allows the most points but still wins
Round 4: Anti-offense - team that score the least points but still wins
Round 5: Largest point difference - as it says
Round 6: Smallest point difference - as it says
etc.

Just think of the variations in strategy required to do well.
For instance, in round 3 above you may need to help your opponent score points but still have more so you could win.

But of course, in all cases these would just be for tie breakers because I will always believe that
NUMBER OF WINS SHOULD BE THE MOST IMPORTANT IN ANY SCORING SYSTEM.

There is no way that a team with more wins should be ranked below a team with less wins.
If this would have been the case at the nationals, we would have had much different strategies for our matches.
We would have tried to win instead of risking losing by trying to get a big score.

It is sad that last year's scoring could rank a team with 1 win and 2 loses higher than a team with 3 wins.
Here is an example:
Team #1: lose with 18 points, lose with 24 points, win with 540 points = 1662 Q-points = 554 ave QP's
Team #2: win with 90 points, win with 150 points, win with 120 points = 1080 Q-points = 360 ave QP's

Now I'm no expert statistician, but I believe that given all the variables with alliances that it was easier for a team to get lucky and get 2 very high scores (almost ensuring them a top 16 seed at nationals) then it was for a team to get lucky and win all 6 rounds!!!

So, there it is. I tried to resist complaining about scoring systems, but I could not resist.

Raul

[archiver]

Posted by P.J. Baker.

Engineer on team #177, Bobcat Robotics, from South Windsor High School and International Fuel Cells.

Posted on 9/7/99 5:50 AM MST


In Reply to: If you think that was complicated. try this... posted by Raul on 9/5/99 10:57 PM MST:



Raul,


With a 6 match schedule, I can't quite agree that won lost record should be the most important factor in seeding. I do think though, that this year it was not important enough. It probably was possible for a 2-6 team to seed ahead of a 6-0 team. That's not right.

I'm perfectly comfortable with a 5-1 team, or even maybe a 4-2 team seeding ahead of a 6-0 team. A team that should have been 4-2 is just as likely to get 2 lucky wins and go to 6-0 as a team that should have been 6-0 is to get 2 unlucky losses and slip to 4-2.

This is why I think that it is important to also look at the scoring and defensive capabilities of the team. If we played lots more matches, I'd be with you 100%. With only 6 though, I think that the seeding should be based on as many measureable factors as possible (won/loss, ave. offense, ave. defense). What do you think?

P.J.