OPR after Week Five Events

[MikeE]

Thanks Ed & Ether for this great resource.

However I do need to quibble about how OPR/CCWM is being discussed, as exemplified by several earlier posts. Picking one of these:

Originally Posted by efoote868

Summary of global OPR and CCWM match win/loss predictions: 81.72% and 82.47% respectively.

Those numbers are misleading since OPR/CCWM are calculated from the same data which are being used to test their predictive power, i.e. the training & test sets are the same.

It's analogous to (although not as extreme as) stating that final qualification ranking is a good predictor of the performance in earlier qualifying matches. Whereas obviously qualification ranking is a consequence of performance in earlier matches.

Good practice would use disjoint training and testing sets. I'm sure this analysis has been performed in previous seasons but I didn't see it from a brief search of CD.

Interestingly the simple baseline heuristic of "alliance with lower team numbers" has 59.1% predictive power for qualification matches this season. I'm assume that OPR and CCWM are better than that, but not as good as the ~82% claimed above.

[Ether]

Originally Posted by MikeE

Those numbers are misleading since OPR/CCWM are calculated from the same data which are being used to test their predictive power, i.e. the training & test sets are the same.

That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes. That would take a little bit more work, since Week5 may have teams which are competing for the first time, so they would have no OPR values. Those matches would have to be omitted from the analysis.

[efoote868]

Originally Posted by MikeE

Good practice would use disjoint training and testing sets. I'm sure this analysis has been performed in previous seasons but I didn't see it from a brief search of CD.

Interestingly the simple baseline heuristic of "alliance with lower team numbers" has 59.1% predictive power for qualification matches this season. I'm assume that OPR and CCWM are better than that, but not as good as the ~82% claimed above.

At Boilermaker Regional I calculated the OPR of teams using their Friday match results (about 8 or 9 matches), and tracked the qualification results on Saturday. OPR predictions were 20 for 24.

I'm guessing the easiest place to track predictive power would be the Championship event.

Originally Posted by Ether

That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes. That would take a little bit more work, since Week5 may have teams which are competing for the first time, so they would have no OPR values. Those matches would have to be omitted from the analysis.

I'd be interested in substituting unknown values with world averages. I'll probably do this during Crossroads until each team has played 5 matches.

[George1902]

OPR and CCWM are calculated using only qualification match results, right?

So, one could test their predictive value using the elimination results of that event.

[Ether]

Originally Posted by Ether

That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes.

OK, here's qual matches of Week5 events being predicted by OPR World rank based on weeks 1 thru 4 data only. 72% correct

This is a pessimistic estimate, since data gathered during the Week5 events for already-played qual matches could be used to supplement the weeks 1 thru 4 data to predict future qual matches in the Week5 events.

Based on the earlier post by efoote868, future qual matches at any given event are best predicted by OPR of already-played matches at that event, once a sufficient number of matches has been played.

Forgot to mention: columns V thru AA list the teams for each match for which no week 1 thru 4 OPR data is available.

[faust1706]

Originally Posted by I-DOG

It's bittersweet that we have the 5th highest OPR in the world and didn't qualify for Championship...

we're in the same boat. 4th.

[DELurker]

Originally Posted by Ether

Does not predict 100%. But its correlation with match win/loss outcomes is unarguably statistically significant (witness earlier posts in this thread), and it is one among many useful scouting tools.

True. Assuming that nothing changes from previous form, the OPR and CCWM can be very good indicators of probable results. However, there is an implicit error function on the numbers (otherwise, they would be 100% in the absence of meaningful changes). In addition, any team that knows they are on the short end of the statistics and does not try for a change in their favor is doing themselves a disservice.

I suppose that what I am trying to lead to is that the OPR and CCWM numbers are very useful tools (we use them a lot), but they only indicate based on the assumption that nothing will change from previous matches.

[Racer26]

So I was just toying with this spreadsheet out of interest.

I added a field to the OPR results page for each team's state/province, and filtered it based on that field, because I wanted to support my intuitive feeling that Ontario's second tier is much stronger because of the influence of 1114 and 2056.

Calculating average OPR's for different regions gives me the following results:

Avg Max OPR FIRST-Wide: 16.3

Avg Max OPR in Ontario: 23.3
Avg Max OPR in Michigan: 22.8
Avg Max OPR in Ontario (not including 1114 and 2056): 21.1
Avg Max OPR in New Jersey: 18.8
Avg Max OPR in Ohio: 18.6
Avg Max OPR in Pennsylvania: 18.5
Avg Max OPR in Indiana: 17.4
Avg Max OPR in New York: 16.8
Avg Max OPR in Quebec: 16.7
Avg Max OPR in Massachusetts: 16.1
Avg Max OPR in New Hampshire: 14.0
Avg Max OPR in California: 14.0
Avg Max OPR in Texas: 13.8
Avg Max OPR in Minnesota: 13.0

[Nemo]

Thanks for posting this information - I have fun with it. Also... congratulations on your team's win over the weekend.

[DELurker]

Originally Posted by Racer26

So I was just toying with this spreadsheet out of interest.

...

Calculating average OPR's for different regions gives me the following results:

Avg Max OPR FIRST-Wide: 16.3

Avg Max OPR in Ontario: 23.3
Avg Max OPR in Michigan: 22.8
Avg Max OPR in Ontario (not including 1114 and 2056): 21.1
Avg Max OPR in New Jersey: 18.8
...

(Edited for brevity)

What about Delaware? 27.8 Avg Max OPR. Of course, it's not as though we have enough teams to form a full alliance, much less a full match, but still...

[Racer26]

Originally Posted by DELurker

(Edited for brevity)

What about Delaware? 27.8 Avg Max OPR. Of course, it's not as though we have enough teams to form a full alliance, much less a full match, but still...

Two data points does not a trend make

[DELurker]

Originally Posted by Racer26

Two data points does not a trend make

True enough... but it's good enough for a little bit of bragging... at least until Iowa, Brazil (RS), and Nevada get in on the act.

Of the states/provinces/areas that can field their own matches, Mississippi, Hawaii, and Ontario are 1-2-3.

*sigh* One day, maybe we'll grow up to be a bigger state...

[DELurker]

Originally Posted by Racer26

Two data points does not a trend make

Actually, isn't the definition of a trend the point at which you can draw a line through two points? Not only is it a trend, but it has 100% correlation.

[Basel A]

Trend implies some sort of change over time, so really none of them are trends. They're all just statistical distributions.

[Racer26]

Originally Posted by DELurker

Of the states/provinces/areas that can field their own matches, Mississippi, Hawaii, and Ontario are 1-2-3.

Field their own matches with teams that have played so far in 2013. HI has more than 6 teams, just only 6 of them play OUTSIDE of HI, since the Hawaii Regional hasn't happened yet.

And of those 3, only Ontario has had enough to hold a regional.