Comparison of DIV MAX OPRs

[ATannahill]

Originally Posted by Joe Johnson

Here are the DCMP's compared to the Divisions at Worlds (note, the zero line is equal to the associated percentile of all teams CMP this year). I have also added an non-normalized chart. Harder to read but you can see raw numbers.

I have to say, I am surprised and I was wrong, very wrong. The DCMP's are not generally better than the Divisions at Worlds.

A wise man's mind changes, a fool's changes not...

Dr. Joe J.

P.S. Thanks for the data (several sources).

<images snipped>

Did you intentionally leave out PNW, NC and Chesapeake? I know they were week 6.

[FiMFanatic]

If I read the chart correctly, the 50% scoring indicates that the Michigan Championship has the highest average quality.

Of course top 5% at World's will be higher than the top 5% of any district championship.....so 95% level somewhat irrelevant.

[Caleb Sykes]

Originally Posted by FiMFanatic

Of course top 5% at World's will be higher than the top 5% of any district championship.....so 95% level somewhat irrelevant.

Except for New England.....so your point is somewhat irrelevant.

[Ether]

Originally Posted by rtfgnow

Did you intentionally leave out PNW, NC and Chesapeake? I know they were week 6.

This is not the same as Joe's chart, but it compares all 7 district CMPs.

[howellroy]

Ok, so I am looking things over and it is true Newton seems to be the toughest but there seems to be some balance. It doesn't mean things are completely unbalanced although there is a gap between Galileo and Newton but that is going to happen when trying to set these things up. I think the greater variable is the number of matches each team played. Those who go to Regional play less matches than those who are part of a District. Regional matches do cost more to go to and there is a bit more travel over longer distances related to it. That give district matches more practice and development of a strategy that best fits their robot. Parity is a matter of every team having exactly the same resources and access money. I'm just saying.

Galileo 35.68
Tesla 35.95
Curie 36.28
Archimedes 36.93
Carver 37.85
Carson 37.96
Hopper 38.15
Newton 40.11

[Joe Johnson]

Originally Posted by rtfgnow

Did you intentionally leave out PNW, NC and Chesapeake? I know they were week 6.

No I didn't. I was in a hurry to post this with my wife basically turning out the lights in the living room as I was crunching the data (think pits at closing time ;-) and I completely forgot about the week 6 districts.

Sorry. Maybe tonight if I get time...

Dr. Joe J.

[Jared Russell]

Another trivia question for you stat-gurus out there to chew on...

Given random assignment of teams to 8 divisions, what is the probability that any division's [mean OPR, 90th %ile OPR, 75th %ile OPR, 50th %ile OPR, etc.] is as strong as Newton's?

[Joe Johnson]

Originally Posted by Jared Russell

Another trivia question for you stat-gurus out there to chew on...

Given random assignment of teams to 8 divisions, what is the probability that any division's [mean OPR, 90th %ile OPR, 75th %ile OPR, 50th %ile OPR, etc.] is as strong as Newton's?

Zing!

I don't know but I'll try to model this tonight.

Dr. Joe J

[Ether]

FWIW

Ten million 75-team samples randomly drawn from 600-team CMP population; 2.5% had average OPR greater than Newton's.


Interestingly:

mean of max OPR of 600 CMP teams = 37.331

std dev of max OPR of 600 CMP teams = 12.287

predicted std dev of the means of the 75-team samples = 12.287/sqrt(75) = 1.4187

mean OPR of Newton = 40.1149

predicted Zscore of Newton = (40.1149-37.331)/1.4187 = 1.9623

area under normal curve between mean and 1.9623 = .4570

predicted probability = 0.5 - 0.4750 = 0.025 = 2.5%

[Ether]

Originally Posted by Ether

Ten million 75-team samples randomly drawn from 600-team CMP population...

A different simulation:

1) shuffle the 600-element vector of CMP max OPR scores

2) divide the vector into octants and compute the average OPR for each octant

3) if one or more of the octants have average OPR >= Newton's, increment a counter.

4) repeat steps 1 thru 3 ten million times.

14.4% of the time, at least one octant has average OPR >= Newton's

[Jim Zondag]

Being a former signals guy, I prefer to look at the world like this graph shows.
By comparing the average OPR against the Signal to Noise Ratio of the same data, you can get a singular view of both the overall strength of an event and its competitive balance.
We try to get our District System to produce a DCMP event which as far up and to the right as possible. As high performing and balanced as we can make it.
This makes for an exciting event (and exciting TV )
The CMP divisions always under-perform the DCMP events on balance, because there are simply more weak teams at the CMP due to the FIRST regional promotion strategies as compared to the district promotion system.
This graph gets pretty cluttered in the center these days with 150 events (i have a little wizard plugin which helps with viewing this)
The CMP divisions are in RED. These points are of course predictive, actual results may vary next weekend.
From this you can see that the Newton division is a pretty big outlier, so like Waterloo, the top teams from Newton will have a real battle to win their division. It should be fun to watch.

[Joe Johnson]

Originally Posted by Jim Zondag

Being a former signals guy, I prefer to look at the world like this graph shows.
By comparing the average OPR against the Signal to Noise Ratio of the same data, you can get a singular view of both the overall strength of an event and its competitive balance.
We try to get our District System to produce a DCMP event which as far up and to the right as possible. As high performing and balanced as we can make it.
This makes for an exciting event (and exciting TV )
The CMP divisions always under-perform the DCMP events on balance, because there are simply more weak teams at the CMP due to the FIRST regional promotion strategies as compared to the district promotion system.
This graph gets pretty cluttered in the center these days with 150 events (i have a little wizard plugin which helps with viewing this)
The CMP divisions are in RED. These points are of course predictive, actual results may vary next weekend.
From this you can see that the Newton division is a pretty big outlier, so like Waterloo, the top teams from Newton will have a real battle to win their division. It should be fun to watch.

Jim,
I like the chart but I am always suspicious of averages over medians/percentiles. Can you redo the data with each "point" becoming a bar between two points (50%tile & 90%tile say).

Can you explain how each you generate an SNR for each group of teams? I suppose it is the (average/stdev) but perhaps you have another idea. If you are using average/stdev, I would suggest that the two points for each competition group would be (50%tile/stdev, 50%tile) & (90%tile/stdev, 90%tile). I think this bar would better represent the team groupings. FWIW.

Dr. Joe J.

P.S. What happened to Newton this year is really a fruit of the same seed that produces a lot of the frustration I have with FIRST (I am not saying FIRST is nothing but frustration. I love them. They are doing a lot of great things. But when I get frustrated with them, it is often from the same root cause). Namely, they dither between being a "everyone should to the right thing because it's the right thing and we all agree that right is right and who can be against being right?" and being a robotic sports activity.

Because part of them doesn't want to be a robotic sport, they are completely blind to problems that participants in the robotic sports activities they organize are painfully aware of. Like having 1 division out of 8 be stacked with capable robot teams while others are sparely populated.

[Big Ideas]

Originally Posted by Jim Zondag

Being a former signals guy, I prefer to look at the world like this graph shows.
By comparing the average OPR against the Signal to Noise Ratio of the same data, you can get a singular view of both the overall strength of an event and its competitive balance.
We try to get our District System to produce a DCMP event which as far up and to the right as possible. As high performing and balanced as we can make it.
This makes for an exciting event (and exciting TV )
The CMP divisions always under-perform the DCMP events on balance, because there are simply more weak teams at the CMP due to the FIRST regional promotion strategies as compared to the district promotion system.
This graph gets pretty cluttered in the center these days with 150 events (i have a little wizard plugin which helps with viewing this)
The CMP divisions are in RED. These points are of course predictive, actual results may vary next weekend.
From this you can see that the Newton division is a pretty big outlier, so like Waterloo, the top teams from Newton will have a real battle to win their division. It should be fun to watch.

I like this view BUT; I would love to see a time axis WK1-CMP, colors would work. Since San Diego (week 1) shows as less competitive then Sacramento (week 4 ) But my perception was opposite. So, how much does competitiveness slide as the season progresses?

[Joe Johnson]

Originally Posted by Jared Russell

Another trivia question for you stat-gurus out there to chew on...

Given random assignment of teams to 8 divisions, what is the probability that any division's [mean OPR, 90th %ile OPR, 75th %ile OPR, 50th %ile OPR, etc.] is as strong as Newton's?

As I suggested above, I looked into this very question.

TL;DR: I estimate that as a competitor you would expect to be in a division as stacked as Newton (or more stacked) once every 250-500 years. Further as a spectator, you would would expect to attend a Worlds with a division as stacked as Newton (or more stacked) once every 40-70 years.


A lot depends on the method you use to define what we mean when we say Newton is stacked. I decided that I would plot the MAX OPR for the Nth Percentile team for a division and compare that to the same Nth Percentile for the CMP Population as a whole.

This chart helps you see what I mean.




This chart is better because I do the subtraction (DIV OPR %tile - CMP OPR %tile).




So NOW I can propose my metric. Let's integrate the area under the curve for the above chart in some range and then normalize for the width of the range the area.

But what range do we want to use?

Here are two charts for proposed ranges.

The first is my "I think this would be a fair metric" range (55%tile to 95%tile). My thinking is that it eliminates the very top of the range because, Hey, whenever a handful of powerhouse teams end up in a division, the top is off the charts, that's just normal.




My second is "Let's tailor the metric as much as we can to be favorable to Newton" That is, pick the range so that it measures where Newton opens up a big gap compared to the CMP as a whole. Coincidentally that also is the top 24 Teams in a 75 Division so, if those teams make up the Playoff teams they'd have a pretty amazing set of 8 Alliances.



Now I can do a simulated Division Assignment. I did it (almost) like FIRST is rumored to do it. I divided the rookies teams and the non-rookies randomly but separately. For Rookies, the rule should be that teams with numbers 5726 and higher but that gave me a number of rookies that wasn't evenly divisible by 8. For reasons that are hard to explain, it was easier for my simulation if I had the number of rookies be a divisible by 8 so I added 3 teams to my "rookie group" (5686, 5690, & 5712) I don't think this changed my results significantly.

After running 2,500 simulated CMP Division Assignments (that is 20,000 Divisions), I get this when you look at the distribution of Division Stacked Metric (only showing chart for Method 1, but Method 2 was similar).




Looking at the data, it is easy to see that 2,500 CMP's are plenty when it comes to characterizing the variation involved.

Long story short.

Based on Method #1:
I estimate that as a competitor you would expect to be in a division as stacked as Newton (or more stacked) once every 250 years. Further as a spectator, you would would expect to attend a Worlds with a division as stacked as Newton (or more stacked) once every 40 years.

Based on Method #2:
I estimate that as a competitor you would expect to be in a division as stacked as Newton (or more stacked) once every 500 years. Further as a spectator, you would would expect to attend a Worlds with a division as stacked as Newton (or more stacked) once every 67 years.

So... there you have it. Any way you cut it, Newton is one wacky division this year.

Also, FIRST should probably think about how they make the divisions a bit more to avoid this kind of think in the future.

Comments welcome.

Dr. Joe J.

[Ether]

Originally Posted by Ether

14.4% of the time, at least one octant has average OPR >= Newton's

Yet a another perspective:

75 (= 1/8 or 12.5%) of the 600 CMP teams have max OPR greater than 50.98

Newton has 18 (or about 24%) teams greater than 50.98, roughly twice the CMP population percentage.

I re-ran the simulation mentioned in the quote above, but this time counted the number of times at least one division had 18 or more teams >= 50.98. The result was 1.87% of the time.