[wgardner]

Check out the attached image. It has a lot of data in it. Here's the explanation:

I simulated a tournament using the 2014 casa structure with the underlying O components having mean 25 and stdev 10, as in most of my sims. I set the D component to 0 (no defense) and the match noise component N to 3 (so random variability roughly equals the variability due to the 3 offensive teams).

The top left plot shows the LS OPRs after each team has played 1-12 matches. X=13 corresponds to the actual, underlying O values.

Across the top row is the standard MMSE OPRs with Var(N)/Var(O) estimated at 1, 3, and 5. You can see that the OPRs start progressively tighter at X=1 and then branch out, with 3 and 5 branching out more slowly.

So the top row is basically the same stuff from the paper, just MMSE OPR estimation. It's assuming an apriori OPR distribution with Oave mean and varying standard deviation for all of the OPRs.

Now, the plots on the 2nd and 3rd rows are MMSE estimation but where we have additional guesses about what the specific OPRs should be before the tournament starts (!!). For example, this could happen if we try to estimate the OPRs before the tournament from previous tournament results.

The 2nd row assumes that we have an additional estimate for each team's OPR with the same standard deviation of all of the OPRs. So for example, the underlying Os are chosen with mean 25 and stdev 10. Row 2 assumes that we have additional guesses of the real underlying O values, but the guesses have random noise added to them with a stdev of 10 also. So they're noisy guesses, similar to what we might have if we predicted the championship OPRs from the teams' OPRs in previous tournaments.

The 3rd row is the same, but the extra guess has a standard deviation of only 0.3 of the stdev of the actual underlying O. In this example, we have another guess of the real O values with standard deviation of only 3 before the tournament starts.

The left center plot shows how well the techniques do at estimating the true, underlying O values over time. You can see the clumping on the left based on how much knowledge was known ahead of time.

The left bottom plot shows how well the techniques do at estimating the match scores. Note that with Var(N)/Var(O)=3, the best we should be able to do is 50% so the fact that we're just a bit under 50% is an artifact of this being only 1 simulation run. Again, you can see the clustering on the left of the plot based on how much apriori info was known.

For the most part, the results are pretty similar at the end of the tournament, but you can clearly see the advantage of the apriori inforomation at the start of the tournament.

If all you ever use OPRs for is for alliance selection at the end of the tournament, then there's not much advantage to going with MMSE over LS. But if you would use live stat information to plan out your strategy in upcoming matches during a tournament, then MMSE could provide you with benefits.

So, the question is: how good of an estimate can we get for the OPRs before a tournament starts based on previous tournament data?