Incorporating Opposing Alliance Information in CCWM Calculations

[AGPapa]

Originally Posted by Ed Law

This is a very cool way of looking at it. By putting it this way, EPR seems to be half way between OPR and WMPR.
Again, I like it because it is one number instead of two numbers. I like it because it has a better chance to predict outcome regardless of the game, rather than OPR being good for some games and WMPR being good for some other games.

WMPR (with a mean of the average score/3) is also just one number instead of two. And what game is WMPR bad for? Recycle Rush seems like it would be the worst game for WMPR, but it's comparable to OPR in predicting outcomes, if not slightly better.

And from my testing, the order of predictiveness goes WMPR>EPR>OPR. The only improvement EPR has over OPR is that it's half WMPR! Why not just go all the way and stick with WMPR?


Again, this is with using the training data as the testing data, if EPR is shown to be better when these are separate then perhaps we should use it instead.

[Ed Law]

Originally Posted by AGPapa

WMPR (with a mean of the average score/3) is also just one number instead of two. And what game is WMPR bad for? Recycle Rush seems like it would be the worst game for WMPR, but it's comparable to OPR in predicting outcomes, if not slightly better.

And from my testing, the order of predictiveness goes WMPR>EPR>OPR. The only improvement EPR has over OPR is that it's half WMPR! Why not just go all the way and stick with WMPR?


Again, this is with using the training data as the testing data, if EPR is shown to be better when these are separate then perhaps we should use it instead.

The reason I said two numbers is that in the past I look at OPR and CCWM. I am considering WMPR as a replacement of CCWM, which is why I will be looking at OPR and WMPR.
When we have more data, multiple years and multiple events that support WMPR as the best predictor for match outcome, then I will stop looking at OPR. But sometimes in alliance selection for first round pick, without any scouting data and you want somebody for pure offense, OPR is still a good indicator.

[wgardner]

[Edit: The data has been updated to reflect an error in the previous code. Previously, the data was reported for the scaled down versions of the metrics in the TESTING DATA section. Now, the data is reported for the unscaled metrics (though the last table for each tournament shows the benefits of scaling them, which is substantial!)]

Here's the data for the four 2014 tournaments starting with "A". My thoughts will be in a subsequent post:

2014: archi
Teams = 100, Matches = 167, Matches Per Team = 1.670

TRAINING DATA
Stdev of winning margin prediction residual
OPR :  51.3.	 66.9% of outcome variance predicted.
CCWM:  57.0.	 59.2% of outcome variance predicted.
WMPR:  36.1.	 83.6% of outcome variance predicted.

Match prediction outcomes
OPR : 142 of 166  (85.5 %)
CCWM: 146 of 166  (88.0 %)
WMPR: 154 of 166  (92.8 %)

TESTING DATA
Stdev of winning margin prediction residual
OPR :  72.1.	 34.8% of outcome variance predicted.
CCWM:  85.2.	  8.8% of outcome variance predicted.
WMPR:  89.3.	 -0.1% of outcome variance predicted.

Match prediction outcomes
OPR : 127 of 166  (76.5 %)
CCWM: 124 of 166  (74.7 %)
WMPR: 123 of 166  (74.1 %)

Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight:	  1.0	  0.9	  0.8	  0.7	  0.6	  0.5
OPR:	 72.1	 70.8	 70.2	 70.3	 71.2	 72.8
CCWM:	 85.2	 80.3	 76.3	 73.5	 71.9	 71.7
WMPR:	 89.3	 84.3	 80.3	 77.3	 75.4	 74.7


2014: abca
Teams = 35, Matches = 76, Matches Per Team = 2.171

TRAINING DATA
Stdev of winning margin prediction residual
OPR :  59.8.	 65.1% of outcome variance predicted.
CCWM:  62.9.	 61.2% of outcome variance predicted.
WMPR:  51.5.	 74.1% of outcome variance predicted.

Match prediction outcomes
OPR :  63 of  76  (82.9 %)
CCWM:  60 of  76  (78.9 %)
WMPR:  65 of  76  (85.5 %)

TESTING DATA
Stdev of winning margin prediction residual
OPR :  78.9.	 39.1% of outcome variance predicted.
CCWM:  93.6.	 14.4% of outcome variance predicted.
WMPR:  92.5.	 16.3% of outcome variance predicted.

Match prediction outcomes
OPR :  56 of  76  (73.7 %)
CCWM:  55 of  76  (72.4 %)
WMPR:  55 of  76  (72.4 %)

Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight:	  1.0	  0.9	  0.8	  0.7	  0.6	  0.5
OPR:	 78.9	 77.9	 77.6	 78.2	 79.6	 81.6
CCWM:	 93.6	 89.5	 86.4	 84.3	 83.4	 83.7
WMPR:	 92.5	 88.8	 86.1	 84.3	 83.6	 84.1


2014: arfa
Teams = 39, Matches = 78, Matches Per Team = 2.000

TRAINING DATA
Stdev of winning margin prediction residual
OPR :  45.8.	 61.4% of outcome variance predicted.
CCWM:  46.6.	 60.1% of outcome variance predicted.
WMPR:  38.2.	 73.1% of outcome variance predicted.

Match prediction outcomes
OPR :  59 of  78  (75.6 %)
CCWM:  66 of  78  (84.6 %)
WMPR:  64 of  78  (82.1 %)

TESTING DATA
Stdev of winning margin prediction residual
OPR :  61.8.	 29.8% of outcome variance predicted.
CCWM:  71.7.	  5.6% of outcome variance predicted.
WMPR:  75.4.	 -4.5% of outcome variance predicted.

Match prediction outcomes
OPR :  55 of  78  (70.5 %)
CCWM:  53 of  78  (67.9 %)
WMPR:  49 of  78  (62.8 %)

Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight:	  1.0	  0.9	  0.8	  0.7	  0.6	  0.5
OPR:	 61.8	 61.0	 60.6	 60.8	 61.4	 62.5
CCWM:	 71.7	 68.4	 65.9	 64.1	 63.1	 62.9
WMPR:	 75.4	 71.9	 69.1	 66.9	 65.5	 64.9


2014: azch
Teams = 49, Matches = 82, Matches Per Team = 1.673

TRAINING DATA
Stdev of winning margin prediction residual
OPR :  36.3.	 78.2% of outcome variance predicted.
CCWM:  37.8.	 76.4% of outcome variance predicted.
WMPR:  25.5.	 89.2% of outcome variance predicted.

Match prediction outcomes
OPR :  66 of  79  (83.5 %)
CCWM:  68 of  79  (86.1 %)
WMPR:  73 of  79  (92.4 %)

TESTING DATA
Stdev of winning margin prediction residual
OPR :  52.1.	 54.9% of outcome variance predicted.
CCWM:  67.5.	 24.6% of outcome variance predicted.
WMPR:  63.0.	 34.3% of outcome variance predicted.

Match prediction outcomes
OPR :  59 of  79  (74.7 %)
CCWM:  56 of  79  (70.9 %)
WMPR:  66 of  79  (83.5 %)

Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight:	  1.0	  0.9	  0.8	  0.7	  0.6	  0.5
OPR:	 52.1	 52.1	 52.8	 54.2	 56.2	 58.7
CCWM:	 67.5	 65.7	 64.6	 64.1	 64.2	 65.0
WMPR:	 63.0	 59.6	 57.3	 56.1	 56.1	 57.3

[wgardner]

[Edit: my previously posted results had mistakenly reported the values for the scaled versions of OPR, CCWM, and WMPR as the unscaled values (!). Conclusions are somewhat changed as noted below.]

So my summary of the previous data:

WMPR always results in the smallest training data winning margin prediction residual standard deviation. (Whew, try saying that 5 times fast.)

WMPR is also very good at predicting training data match outcomes. For some reason, CCWM beats it in 1 tournament but otherwise WMPR is best in the other 3.

But on the testing data, things go haywire. There are significant drops in performance in predicting winning margins for all 3 stats, showing that all 3 stats are substantially overfit. Frequently, all 3 stats give better performance at predicting winning margins by using scaled down versions of the stats. The WMPR in particular is substantially overfit (look for a later post with a discussion of this).

BTW, it seems like some folks are most interested in predicting match outcomes rather than match statistics. If that's really what folks are interested in, there are probably better ways of doing that (e.g., with linear models but where the error measure better correlates with match outcomes, or with non-linear models). I'm going to ponder that for a while...

[Citrus Dad]

I've been watching this thread because I'm really interested in a more useful statistic for scouting--a true DPR. I think this path may be a fruitful way to arrive at that point.

Currently the DPR doesn't measure how a team's defensive performance causes the opposing alliance to deviate from its predicted OPR. The current DPR calculation simply assumes that the OPRs of the opposing alliances are randomly distributed in a manner that those OPRs are most likely to converge on the tournament average. Unfortunately that's only true if a team plays a very large number of matches that capture potential alliance combinations. Instead we're working with a small sample set that is highly influenced by the individual teams included in each alliance.

Running the DPR separately across the opposing alliances becomes a two-stage estimation problem in which 1) the OPRs are estimated for the opposing alliance and 2) the DPR is estimated against the predicted OPRs. The statistical properties become interesting and the matrix quite large.

I'll be interested to see how this comes out. Maybe you can report the DPRs as well.

[AGPapa]

I tested how well EPR predicted match outcomes in the four events in 2014 beginning with "a". These tests excluded the match being tested from the training data and recomputed the EPR.

EPR:
ABCA: 59 out of 76 (78%)
ARFA: 50 out of 78 (64%)
AZCH: 63 out of 79 (78%)
ARCHI: 123 out of 166 (74%)

And as a reminder, here's how OPR did (as found by wgardner)
OPR:
ABCA: 56 out of 76 (74%)
ARFA: 55 out of 78 (71%)
AZCH: 59 out of 79 (75%)
ARCHI: 127 out of 166 (77%)

So over these four events OPR successfully predicted 297 matches and EPR successfully predicted 295.

[wgardner]

On the Overfitting of OPR and WMPR

I'm working on studying exactly what's going on here with respect to the overfitting of the various stats. Look for more info in a day or two hopefully.

However, I thought I'd share this data point as a good example of what the underlying problem is.

I'm looking at the 2014 casa tournament structure (# of teams=54 which is a multiple of 6 and the # of matches is twice the # of teams, so it fits in well with some of the studies I'm doing).

As one data point, I'm replacing the match scores with completely random, normally distributed data for every match (i.e., there is absolutely no relationship between the match scores and which teams played!). Stdev of each match score is 1.0, so the winning margin is the difference between 2 and has variance of 2.0 and stdev of 1.414.

I get the following result on one run (which is pretty typical).

2014 Sim: casa
Teams = 54, Matches = 108, Matches Per Team = 2.000
SIMULATED MATCH SCORES THAT ARE 100% RANDOM NOISE!

TRAINING DATA
Stdev of winning margin prediction residual
OPR :   1.3.	 26.5% of outcome variance predicted.
WMPR:   1.1.	 47.3% of outcome variance predicted.

Match prediction outcomes
OPR :  78 of 108  (72.2 %)
WMPR:  87 of 108  (80.6 %)

TESTING DATA
Stdev of winning margin prediction residual
OPR :   1.7.	-31.3% of outcome variance predicted.
WMPR:   2.1.	-105.0% of outcome variance predicted.

Match prediction outcomes
OPR :  58 of 108  (53.7 %)
WMPR:  56 of 108  (51.9 %)

Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight:	  1.0	  0.9	  0.8	  0.7	  0.6	  0.5
OPR:	  1.7	  1.7	  1.6	  1.6	  1.5	  1.5
WMPR:	  2.1	  2.0	  1.9	  1.8	  1.8	  1.7

For random match scores, the OPR can still "predict out" 26% of the "training data" winning margin variance and the WMPR can still "predict out" 47% of the "training data" winning margin variance! And they can correctly predict 72% and 81% of the match results of the training set, respectively.

This is what I mean by overfitting: the metrics are modeling the match noise even when the underlying OPRs and WMPRs should all be zero. And this is why the final table shows that scaling down the OPRs and WMPRs (e.g., replace the actual OPRs by 0.9*OPRs, or 0.8*OPRs, etc.) results in a lower standard deviation in the predicted Testing data residual, because that reduces the amount of overfitting by decreasing the variance of the predicted outputs. In this case, the best weighting should be zero, as it's better to predict the testing data with 0*OPR or 0*WMPR than it is to predict with completely bogus OPRs and WMPRs.

And WMPR seems to suffer from this more because there are fewer data points to average out (OPR uses 216 equations to solve for 54 values, whereas WMPR uses 108 equations to solve for 54 values).

More to come...

[wgardner]

For posterity, the follow up work I did on this is reported and discussed the paper in this thread.

[nuclearnerd]

I just want to say how awesome you people are. My linear algebra skills are weak, but this thread has moved me a lot closer to a working understanding of the scouting stats. Thank you all for sharing your work.