[wgardner]

Another thread for stat nerds. Again, if you don't know or don't care about OPR and CCWM and how they're calculated, this thread will probably not interest you.
----------------------------------------------------------------

Based on the recent thread on stats, I did way too much study and simulation of what the different stats do in different situations. I've attached a very long paper with all of the findings.

Many thanks to Ether who helped a lot with behind-the-scenes comments and data generation. I think he's off working on some related ideas that I suspect we'll all hear about soon.

The Overview and Conclusions of the paper are included below.

-----------------------------------------------------------------

Overview:

This paper presents and analyzes a wide range of statistical techniques that can be applied to FIRST Robotics Competition (FRC) and FIRST Tech Challenge (FTC) tournaments to rate the performance of teams and robots competing in the tournament.

The well-known Offensive Power Rating (OPR), Combined Contribution to Winning Margin (CCWM), and Defensive Power Rating (DPR) measures are discussed and analyzed.

New measures which incorporate knowledge of the opposing alliance members are discussed and analyzed. These include the Winning Margin Power Rating (WMPR), the Combined Power Rating (CPR), and the mixture-based Ether Power Rating (EPR).

New methods are introduced to simultaneously estimate separate offensive and defensive contributions of teams. These methods lead to new, related simultaneous metrics called sOPR, sDPR, sWMPR, and sCPR.

New MMSE estimation techniques are introduced. MMSE techniques reduce overfitting problems that occur when Least Squares (LS) parameter estimation techniques are used to estimate parameters on a relatively small data set. The performance of LS and MMSE techniques is compared over a range of scenarios.

All of the techniques are analyzed over a wide range of simulated and actual FRC tournament data, using results from the 2013, 2014, and 2015 FRC seasons.

-----------------------------------------------------------------

Conclusions

New improved techniques for incorporating defense into FRC and FTC tournament statistics have been introduced.

New MMSE techniques for estimating model parameters have been introduced.

Most FRC tournaments do suffer from a small data size, causing Least Squares estimates to be overfit to the noisy tournament data which degrades their performance in predicting match outcomes not in the Training set.

MMSE techniques appear to provide limited but significant and consistent improvements in match score and winning margin prediction compared to similar Least Squares techniques.

While incorporating defense into the statistics using MMSE estimation techniques does not result in any decrease in the statistical prediction performance, the advantages in doing so are usually quite small and may make it not worth the effort to do so unless a given FRC season is expected to have substantial defensive components. Occasionally incorporating defense can result in around an 8-12% further reduction in winning margin prediction error (e.g., 2014 casb, 2015 incmp, 2015 micmp tournaments), but this is rare.

MMSE based estimation of the sOPR, sDPR, and sCPR parameters results in the smallest squared prediction error for match scores and match winning margins across all of the studied parameters. MMSE based estimation of OPR parameters often produces results that are quite close.

Least Squares estimates of OPR, CCWM, and DPR using FRC tournament data probably overestimate the relative differences in ability of the teams. MMSE estimates probably underestimate the relative differences.

The small amount of data created in FRC tournaments results in noisy estimates of statistics. Testing set match outcomes from 2013-2015 often had very significant random components to them that could not be predicted by the best linear prediction methods, most likely due to purely random issues that occur in FRC matches.