Incorporating Opposing Alliance Information in CCWM Calculations

[wgardner]

The following results are from my scilab sim for the 2015 MISJO tournament. Again, this should be a BAD tournament year for both CCWM and WMPR as there was little defense involved.

For OPR, the winning margin was predicted by computing the prediction for the offensive score of the red alliance and subtracting the prediction for the offensive score of the blue alliance from it.

For CCWM, the winning margin was predicted by computing the prediction of the winning margin of the red alliance and subtracting the prediction for the winning margin of the blue alliance from it.

For WMPR, the winning margin was computed the same was as in CCWM, but using the values computed using the WMPR derivation instead of the CCWM derivation.

Standard deviations of the prediction residuals of the winning margins:
OPR: 25.6
CCWM: 21.1
WMPR: 15.9

(interesting that CCWM and WMPR both do better than OPR, even in a game with "no defense." Perhaps the race to get the center cans acts like defense in that a team that does well at that may cause the opposing alliance to have a lower score? Or litter?)

The tournament had 80 matches but one match appeared to be a tie, so there were only 79 matches where the match outcome could be predicted.

# of match outcomes predicted correctly:
OPR: 67
CCWM: 66
WMPR: 68

(This is all on the training data (!). I'm not using data not in the training set yet.)

[AGPapa]

Originally Posted by wgardner

# of match outcomes predicted correctly:
OPR: 67
CCWM: 66
WMPR: 68

Can you please attach match-by-match predictions?

I'm getting:

OPR: 67
CCWM: 66
WMPR: 53


Are you using the same numbers in the MISJO_GPR.CSV file that Ether attached? A difference with numbers we're using seems to be the only explanation for this difference since our OPR and CCWM predictions match up.


In the previously attached spreadsheet I erroneously awarded the blue alliance a victory in match 69, it should have been a tie.

[wgardner]

Originally Posted by AGPapa

Can you please attach match-by-match predictions?

I'm getting:

OPR: 67
CCWM: 66
WMPR: 53

Are you using the same numbers in the MISJO_GPR.CSV file that Ether attached?

Attached.
Col1 is WMPR based WM prediction.
Col2 is CCWM based WM prediction.
Col3 is OPR based WM prediction.
Col4 is actual match WM.

I'm using my sim to compute the WMPR values, which I earlier verified matched Ether's values (at least the min and max were identical).

[AGPapa]

Originally Posted by wgardner

Attached.
Col1 is WMPR based WM prediction.
Col2 is CCWM based WM prediction.
Col3 is OPR based WM prediction.
Col4 is actual match WM.

I'm using my sim to compute the WMPR values, which I earlier verified matched Ether's values (at least the min and max were identical).

Thanks, it turns out that we were using different WMPR values. I redownloaded Ether's attachment and it contains different values. Maybe the initial download was corrupted? I'm baffled. Anyways, I can confirm that after redownloading the correct WMPR values that I get your results.

[Ether]

Originally Posted by wgardner

The following results are from my scilab sim for the 2015 MISJO tournament

Attached are WMPR A and b for all 117 2015 events.

OPR A and b are posted here.

[wgardner]

And now for the data where the single testing point was removed from the training data, then the model was computed, then the single testing point was evaluated, and this was repeated 80 times. So the results below are for separate training and testing data.

Stdev of prediction residual of the winning margins:
OPR: 34.6
CCWM: 30.5
WMPR: 30.4

(note that on the testing data from a few posts ago, WMPR had a Stdev of 15.9, so this is an argument that WMPR is "overfitting" the small amount of data available and that it could benefit from having more matches per team)

# of matches predicted correctly (out of 79 possible)
OPR: 63
CCWM: 55
WMPR: 58

So here, the WM-based measures are both better at predicting the winning margin itself but not at predicting match outcomes. CCWM and WMPR have almost identical prediction residual standard deviations but WMPR is slightly better at match outcome prediction in this particular example for some reason.

Again, it would be great to test this on some 2014 data where there was more defense.

[Ether]

Originally Posted by wgardner

Again, it would be great to test this on some 2014 data where there was more defense.

When I get a chance I'll write a script for TBA API to grab their 2014 data and convert it to CSV. It may be a while.

Otherwise, if someone can provide the the 2014 qual match data in CSV format, I can quickly generate all the A and b for WMPR, OPR, and CCWM and post it here.

[Foster]

Originally Posted by Caleb Sykes

I'm having trouble understanding this sentence. Could you please clarify?

Sorry, I was trying to factor in the "strength of schedule" matches into what you are doing. Lots of us have kicked the dirt going "Yes 1640 is in the top, they played all their matches with 3 digit team alliances against all rookie alliances."

But I spent an hour and did some "what IF runs" through the data and the result is pretty low. While low scoring opposing alliances do make a difference, about match 8,9,10 things swing the other way. So while we all hate the "random" selections, it seems to work out in the end.

I only did a small segment, with the full season available, feel free to run your own numbers.

[AGPapa]

I ran the numbers on the 2014 St. Joseph event. I checked that my calculations for 2015 St. Joseph match Ether's, so I'm fairly confident that everything is correct.

Here's how each stat did at "predicting" the winner of each match.

OPR: 87.2%
CCWM: 83.3%
WMPR: 91.0%


I've attached my analysis, WMPR values, A and b matrices, along with the qual schedules for both the 2014 and 2015 St. Joe event.

[saikiranra]

Originally Posted by Ether

When I get a chance I'll write a script for TBA API to grab their 2014 data and convert it to CSV. It may be a while.

Otherwise, if someone can provide the the 2014 qual match data in CSV format, I can quickly generate all the A and b for WMPR, OPR, and CCWM and post it here.

Attached are the A b and t CSV files from all official events from 2014 (Except for 2014waell, because some match scores are missing). The only difference, other than file name, is that the matches are not in sequential order within the A and b files, although they still correspond. The t files were generated by team number and the matches are in what ever order TBA decides to publish their JSON data (which I think is match number, alphabetically sorted).

Also attached are CSV files that have the OPR, CCWM, and GPR (if we are still calling it that) of each team at each event.

I can easily generate these files for any year if anyone would like.

[Ether]

Originally Posted by Ether

...the following two computational methods yield virtually identical results for min L2 norm of b-Ax:


Method 1

1a) [U,S,V] = svd(A)

1b) Replace the diagonal elements of S with their reciprocals, except when abs(Sjj)<threshold (I used 1e-4 for threshold), in which case make Sjj zero.

1c) compute x = V*S*(U'*b)


Method 2

2a) N = A'*A

2b) d= A'*b

2c) compute x = N\d ..... (Octave mldivide notation)

2d) compute m = mean(x)

2e) subtract m from each element of x


Notice Method 1 factors A, not A'A, resulting in less rounding error.

There's a simpler way to do Method#1 above if you are using Matlab or Octave (hat tip to AGPapa):

x = pinv(A,tol)*b;

pinv() is explained in detail here:

http://www.mathworks.com/help/matlab/ref/pinv.html

(well worth reading; explains the interesting difference between x₁=pinv(A) and x₂=A\b)

[wgardner]

Originally Posted by saikiranra

Attached are the A b and t CSV files from all official events from 2014 (Except for 2014waell, because some match scores are missing). The only difference, other than file name, is that the matches are not in sequential order within the A and b files, although they still correspond. The t files were generated by team number and the matches are in what ever order TBA decides to publish their JSON data (which I think is match number, alphabetically sorted).

Thanks!

The files appear to have the "new" A and "new" b. Could you by any chance also generate the "old" A and "old" b? I can generate the old A from the new A by stripping out the +1s and -1s from each row, but I can't generate the old b from the new b since the new b only has match winning margins and I can't get red and blue scores from that. Having both the old and new versions allows for quick and direct comparisons between OPR (which requires the old b), and CCWM and WMPR (which require the new b that can also be derived from the old one).

Maybe we can just call the old A and b by their names, and the new A and b something like Awm and bwm?

Originally Posted by saikiranra

Also attached are CSV files that have the OPR, CCWM, and GPR (if we are still calling it that) of each team at each event.

It looks like the proposal of calling it WMPR instead of GPR may be sticking. As the "G" in GPR, I would prefer to not have the metric be name-based. (Though I did work at Qualcomm for a long time and saw what "The Viterbi Algorithm" did for Dr. Viterbi's fame, I doubt there's as much money in obscure robotics statistical computations.)

[wgardner]

Originally Posted by wgardner

Stdev of prediction residual of the winning margins:
OPR: 34.6
CCWM: 30.5
WMPR: 30.4

(note that on the testing data from a few posts ago, WMPR had a Stdev of 15.9, so this is an argument that WMPR is "overfitting" the small amount of data available and that it could benefit from having more matches per team)

For data that is "overfit" you can sometimes improve the prediction performance on the testing data by simply scaling down the solution.

For fun, I computed the standard deviation of the prediction residual of the testing data not in the training set using the WMPR solution, 0.9*WMPR, 0.8*WMPR, etc. The standard deviation of the prediction residual of the winning margin for the test data for this particular tournament was minimized by 0.7*WMPR, and that standard deviation was down to 28.4 from 30.4 for the unscaled WMPR. So again, more evidence that the WMPR is overfit and could benefit from additional data.

This doesn't change the match outcome prediction that some folks are interested in, since scaling all of the WMPRs down doesn't change the sign of the predicted winning margin which is all the match outcome prediction is.

[Ether]

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

[wgardner]

Originally Posted by Ether

Attached are A, b, and T for 2015 OPR, CCWM, and WMPR.

Also introducing a funky new metric, EPR, which uses 3 simultaneous equations for each match:

1) r1+r2+r3-b1-b2-b3 = RS-BS
2) r1+r2+r3=RS
3) b1+b2+b3=BS

... and solves all the equations simultaneously.

How do you interpret these new EPR values? They look closer to an OPR than a WM-measure as #2 and #3 both only factor in offense. How will you measure its performance? Perhaps by comparing the overall residual of all 3 combined vs. other ways of predicting all 3 (e.g., using WMPR for #1 and standard OPR for #2 and #3)?