On the Overfitting of OPR and WMPR
I'm working on studying exactly what's going on here with respect to the overfitting of the various stats. Look for more info in a day or two hopefully.
However, I thought I'd share this data point as a good example of what the underlying problem is.
I'm looking at the 2014 casa tournament structure (# of teams=54 which is a multiple of 6 and the # of matches is twice the # of teams, so it fits in well with some of the studies I'm doing).
As one data point, I'm replacing the match scores with completely random, normally distributed data for every match (i.e., there is absolutely no relationship between the match scores and which teams played!). Stdev of each match score is 1.0, so the winning margin is the difference between 2 and has variance of 2.0 and stdev of 1.414.
I get the following result on one run (which is pretty typical).
Code:
2014 Sim: casa
Teams = 54, Matches = 108, Matches Per Team = 2.000
SIMULATED MATCH SCORES THAT ARE 100% RANDOM NOISE!
TRAINING DATA
Stdev of winning margin prediction residual
OPR : 1.3. 26.5% of outcome variance predicted.
WMPR: 1.1. 47.3% of outcome variance predicted.
Match prediction outcomes
OPR : 78 of 108 (72.2 %)
WMPR: 87 of 108 (80.6 %)
TESTING DATA
Stdev of winning margin prediction residual
OPR : 1.7. -31.3% of outcome variance predicted.
WMPR: 2.1. -105.0% of outcome variance predicted.
Match prediction outcomes
OPR : 58 of 108 (53.7 %)
WMPR: 56 of 108 (51.9 %)
Stdev of testing data winning margin prediction residual with scaled versions of the metrics
Weight: 1.0 0.9 0.8 0.7 0.6 0.5
OPR: 1.7 1.7 1.6 1.6 1.5 1.5
WMPR: 2.1 2.0 1.9 1.8 1.8 1.7
For random match scores, the OPR can still "predict out" 26% of the "training data" winning margin variance and the WMPR can still "predict out" 47% of the "training data" winning margin variance! And they can correctly predict 72% and 81% of the match results of the training set, respectively.
This is what I mean by overfitting: the metrics are modeling the match noise even when the underlying OPRs and WMPRs should all be zero. And this is why the final table shows that scaling down the OPRs and WMPRs (e.g., replace the actual OPRs by 0.9*OPRs, or 0.8*OPRs, etc.) results in a lower standard deviation in the predicted Testing data residual, because that reduces the amount of overfitting by decreasing the variance of the predicted outputs. In this case, the best weighting should be zero, as it's better to predict the testing data with 0*OPR or 0*WMPR than it is to predict with completely bogus OPRs and WMPRs.
And WMPR seems to suffer from this more because there are fewer data points to average out (OPR uses 216 equations to solve for 54 values, whereas WMPR uses 108 equations to solve for 54 values).
More to come...