Incorporating Opposing Alliance Information in CCWM Calculations

[AGPapa]

Originally Posted by wgardner

Attached.
Col1 is WMPR based WM prediction.
Col2 is CCWM based WM prediction.
Col3 is OPR based WM prediction.
Col4 is actual match WM.

I'm using my sim to compute the WMPR values, which I earlier verified matched Ether's values (at least the min and max were identical).

Thanks, it turns out that we were using different WMPR values. I redownloaded Ether's attachment and it contains different values. Maybe the initial download was corrupted? I'm baffled. Anyways, I can confirm that after redownloading the correct WMPR values that I get your results.

[Ether]

Originally Posted by wgardner

The following results are from my scilab sim for the 2015 MISJO tournament

Attached are WMPR A and b for all 117 2015 events.

OPR A and b are posted here.

[wgardner]

And now for the data where the single testing point was removed from the training data, then the model was computed, then the single testing point was evaluated, and this was repeated 80 times. So the results below are for separate training and testing data.

Stdev of prediction residual of the winning margins:
OPR: 34.6
CCWM: 30.5
WMPR: 30.4

(note that on the testing data from a few posts ago, WMPR had a Stdev of 15.9, so this is an argument that WMPR is "overfitting" the small amount of data available and that it could benefit from having more matches per team)

# of matches predicted correctly (out of 79 possible)
OPR: 63
CCWM: 55
WMPR: 58

So here, the WM-based measures are both better at predicting the winning margin itself but not at predicting match outcomes. CCWM and WMPR have almost identical prediction residual standard deviations but WMPR is slightly better at match outcome prediction in this particular example for some reason.

Again, it would be great to test this on some 2014 data where there was more defense.

[Ether]

Originally Posted by wgardner

Again, it would be great to test this on some 2014 data where there was more defense.

When I get a chance I'll write a script for TBA API to grab their 2014 data and convert it to CSV. It may be a while.

Otherwise, if someone can provide the the 2014 qual match data in CSV format, I can quickly generate all the A and b for WMPR, OPR, and CCWM and post it here.

[wgardner]

Originally Posted by wgardner

Stdev of prediction residual of the winning margins:
OPR: 34.6
CCWM: 30.5
WMPR: 30.4

(note that on the testing data from a few posts ago, WMPR had a Stdev of 15.9, so this is an argument that WMPR is "overfitting" the small amount of data available and that it could benefit from having more matches per team)

For data that is "overfit" you can sometimes improve the prediction performance on the testing data by simply scaling down the solution.

For fun, I computed the standard deviation of the prediction residual of the testing data not in the training set using the WMPR solution, 0.9*WMPR, 0.8*WMPR, etc. The standard deviation of the prediction residual of the winning margin for the test data for this particular tournament was minimized by 0.7*WMPR, and that standard deviation was down to 28.4 from 30.4 for the unscaled WMPR. So again, more evidence that the WMPR is overfit and could benefit from additional data.

This doesn't change the match outcome prediction that some folks are interested in, since scaling all of the WMPRs down doesn't change the sign of the predicted winning margin which is all the match outcome prediction is.

[Ether]

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

[wgardner]

Originally Posted by Ether

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

How do you interpret these new EPR values? They look closer to an OPR than a WM-measure as #2 and #3 both only factor in offense. How will you measure its performance? Perhaps by comparing the overall residual of all 3 combined vs. other ways of predicting all 3 (e.g., using WMPR for #1 and standard OPR for #2 and #3)?

[Ed Law]

Originally Posted by Ether

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

Hi Ether,

I like this idea of solving all 3 equations simultaneously. WMPR is a good improvement over CCWM because winning margin depends on who you are playing against (except in 2015), but I think EPR is even better. I will adopt it after you guys finish your validation on how well it predicts outcome.

I like this EPR because it is one number instead of two. It can replace both OPR and WMPR. The problem with OPR is similar to the problem with CCWM. It does not take into account of who the opponents were. If you play against stronger opponents, you may not be able to score as many points, especially in those years with limited game pieces. Equation 1 will take care of that. It will improve on the line fitting. To me, I would interpret EPR as how many points a team will be able to score with typical opponents on the field. This eliminates the error of match schedule strength due to luck of the draw. A team may have higher than normal score because they faced weaker opponents more often. That would skew the OPR numbers. I think EPR would be more accurate in predicting match scores. Would somebody like to test it out?

Another reason I like EPR is that it is easier to compute without all that
SVD stuff. I would prefer high school students to be able to understand and implement this on their own.

[saikiranra]

Originally Posted by Ether

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

If I'm understanding this properly, are we setting square matrix for that system as the following?

| 2 2 2 -1 -1 -1 | |r1| = |2RS - BS|
| 2 2 2 -1 -1 -1 | |r2| = |2RS - BS|
| 2 2 2 -1 -1 -1 | |r3| = |2RS - BS|
| -1 -1 -1 2 2 2 | |b1| = |2BS - RS|
| -1 -1 -1 2 2 2 | |b2| = |2BS - RS|
| -1 -1 -1 2 2 2 | |b3| = |2BS - RS|

[efoote868]

Originally Posted by Ether

Also introducing a funky new metric, EPR,

Ether Power Rating?

[Ether]

Originally Posted by Ether

...the following two computational methods yield virtually identical results for min L2 norm of b-Ax:


Method 1

1a) [U,S,V] = svd(A)

1b) Replace the diagonal elements of S with their reciprocals, except when abs(Sjj)<threshold (I used 1e-4 for threshold), in which case make Sjj zero.

1c) compute x = V*S*(U'*b)


Method 2

2a) N = A'*A

2b) d= A'*b

2c) compute x = N\d ..... (Octave mldivide notation)

2d) compute m = mean(x)

2e) subtract m from each element of x


Notice Method 1 factors A, not A'A, resulting in less rounding error.

There's a simpler way to do Method#1 above if you are using Matlab or Octave (hat tip to AGPapa):

x = pinv(A,tol)*b;

pinv() is explained in detail here:

http://www.mathworks.com/help/matlab/ref/pinv.html

(well worth reading; explains the interesting difference between x₁=pinv(A) and x₂=A\b)