Re: "standard error" of OPR values
 

OPR is not computed per match, it is computed per tournament (or at least based on a large number of matches).

We use OPR to estimate the score of an alliance in a match. Or to be even more precise, we compute the OPR values as the ones that result in the best linear prediction of the match results.

If we have an alliance run the same match over and over, we will see a variability in the match results and a variability in the prediction error we get when we subtract the actual match results from the OPR-based prediction. We can compute the standard error of this prediction error. This SE tells us the probability range that we would expect the match result to fall in, but doesn't tell us anything about the range that we would expect OPR estimates to fall in over a full tournament.

I'm confused by this sentence: "So if we run the same match over and over, we would expect to see a similar OPR." ???

Re: "standard error" of OPR values
 

BTW, I'm not trying to dodge the question. I can't say if I agree with the post until you answer my bigger picture question about what exactly you're computing the standard error of.

Re: "standard error" of OPR values
 

OK, so can we agree that you're looking for the standard error of the OPR estimate (not the standard error of the match predictions or match results)?

Can we agree that, if we have multiple measurements of a value and make fundamental assumptions that these multiple measurements are representative of the underlying distribution, then we can model the underlying distribution and look at the standard error of the estimates assuming they are computed from the underlying distribution?

If you're willing to accept this, then I humbly suggest that my long derivation from this morning is what you're looking for.

One topic of confusion is this statement: "The OPR is essentially an estimate of the average point contribution across all of the matches in the tournament."

I will agree that if you truly computed the average contribution across all matches by just computing the average of the match results, then you could simply compute the standard error because you have multiple estimates of the average: the individual match results.

But in fact OPR is not computed by averaging the match results. It's a single, simultaneous joint optimization of ALL OPRs for ALL teams at the same time. That's why, for example, if a new match is played and new match results are provided, ALL of the OPRs change, not just the OPRs for the teams in that match.

We don't actually have a bunch of OPR estimates that we just average together to compute our final estimate. That's the rub. If we did, we could compute the standard error of these separate estimates. But in fact, we don't have them: only the single estimate computed from the whole tournament.

Re: "standard error" of OPR values
 

I know we're beating a dead horse here. Here's my attempt at trying to create a simpler example of my confusion.

Let's say we have a single bag of 10 apples. We compute the average weight of the 10 apples. Let's say we want to know the confidence we have in the estimate of the average weight of the apples.

I claim there are a few ways to do this.

Ideally we'd get some more bags of apples, compute the average weights of the apples in each bag, and compute the standard error of these different measurements.

Or, we could compute the average weight of the 10 apples we know, compute the standard deviation of the weights of the 10 apples, then assume that the true distribution of the weights of all apples has this average weight and standard deviation. If we buy into this assumed distribution, then we can look at the standard error of all estimates of the average weight of 10 apples as if each set of 10 apples was actually taken from this modeled distribution.

Does this make sense? Are there other ways y'all are thinking about this?

To relate this to OPRs, I'd claim that the "get lots of bags of apples" approach is like the "get lots of results from different tournaments and see what the standard error in those OPR estimates is".

I'd claim that the "model the underlying distribution and then look at how the estimates will vary if the data is truly taken from this distribution" approach is like what I derived this morning.

Re: "standard error" of OPR values
 

Do you agree with everything in the post except for the sentence

"Citrus Dad wants to know "the standard error" of each element in [OPR]"

?

Re: "standard error" of OPR values
 

I posted a simple example here. Please compute your standard error for that example.

I think that would help greatly to clear the confusion about what you mean.

Re: "standard error" of OPR values
 

Yep. The rest of the post describes how the OPR is computed for the data set you're using, and I have no qualms about that.

Re: "standard error" of OPR values
 

Yeah, I can do this but it might take me a day or two as I don't have the environment set up to do the matrix operations quickly.

I do compute OPRs and match predictions in Java in my android apps available on Google play (Watch FTC Tournament and FTC Online) but it may take me a few days to find the time to either translate the new equations from this morning's derivation into the Java or to bring the code up in Octave, Scilab, or something similar as I haven't had to do that for a while.

Re: "standard error" of OPR values
 

Well I would argue the sentence "Citrus Dad wants to know "the standard error" of each element in [OPR]" is true, but neither you nor I know what he means (yet).

I am hoping he will do the specific computation for the example I posted; I think that will make things clear.

Re: "standard error" of OPR values
 

Fair enough. I didn't want to agree to the statement because it would seem to imply that I understood what you meant by the standard error of each element in [OPR], which I don't. :)

Re: "standard error" of OPR values
 

That's where the similarity ends though.

Your proof starts with N, squares it, finds the least-squares minimum, and shows that it's the solution to the normal equations.

The proof I posted starts with the solution to the normal equations, and shows that it minimizes (in the least-squares sense) the residuals of the associated overdetermined system.

Re: "standard error" of OPR values
 

Perhaps it would have been better had I repeated the Citrus Dad quote from post #1 verbatim, instead of paraphrasing:

Re: "standard error" of OPR values
 

What is "the matrix" (no, not trying to make a Keanu Reeves reference) and if the standard error is a primary output of any statistical package, what is/are the values for the Archimedes data you've provided?

Re: "standard error" of OPR values
 

I don't know either. Hopefully Citrus Dad will answer that question when he responds to this post.

That is the question I am asking here and here and here.

Re: "standard error" of OPR values
 

Scilab code is in the attachment.

Note that there is a very real chance that there's a bug in the code, so please check it over before you trust anything I say below. :)

----------------------

Findings:

stdev(M)=47 (var = 2209) Match scores have a standard deviation of 47 points.

stdev(M - A O)=32.5 (var = 1060) OPR prediction residuals have a standard deviation of about 32 points.

So OPR linear prediction can account for about 1/2 the variance in match outcomes (1-1060/2209).


What is the standard error for the OPR estimates (assuming the modeled distribution is valid) after the full tournament?

about 11.4 per team. Some teams have a bit more or a bit less, but the standard deviation of this was only 0.1 so all teams were pretty close to 11.4.

To be as clear as I can about this: This says that if we compute the OPRs based on the full data set, compute the match prediction residuals based on the full data set, then run lots of different tournaments with match results generated by adding the OPRs for the teams in the match and random match noise with the same match noise variance, and then compute the OPR estimates for all of these different randomly generated tournaments, we would expect to see the OPR estimates themselves have a standard deviation around 11.4.

If you choose to accept these assumptions, you might be willing to say that the OPR estimates have a 1 std-deviation confidence of +/- 11.4 points.


How does the standard error of the OPR (assuming the modeled distribution is valid) decrease as the number of matches increases?

I ran simulations through only the first 3 full matches per team up to 10 full matches per team, or with match totals of:
76, 102, 128, 152, 178, 204, 228, 254


sig^2 (the variance of the per-match residual prediction error) from 3 matches per team to 10 matches per team was
0.0, 19.7, 26.1, 29.3, 30.2, 30.8, 32.5, 32.5

(With only 3 matches played per team, the "least squares" solution can perfectly fit the data as we only have 76 unknowns and 76 parameters. With 4 or 5 matches per team, the model is still a bit "overfit" as we have 102 or 128 unknowns being predicted by 76 parameters.)


mean (StdErr of OPR) from 3 matches per team to 10 is
0.0, 16.5, 16.3, 15.2, 13.6, 12.8, 12.2, 11.4

(so the uncertainty in the OPR estimates decreases as the number of matches increases, as expected)

stdev (StdErr of OPR) from 3 matches per team to 10 is
0.0, 1.3, 0.6, 0.4, 0.3, 0.2, 0.1, 0.1

(so there isn't much variability in team-to-team uncertainty in the OPR measurements, though the uncertainty does drop as the number of matches increases. There could be more variability if we only ran a number of matches where, say, 1/2 the teams played 5 matches and 1/2 played 4?)


And for the record, sqrt(sig^2/matchesPerTeam) was
0.0, 9.9, 12.0, 12.0, 11.4, 11.1, 10.8, 10.3

(compare this with "mean (StdErr of OPR)" above. As the number of matches per team grows, the OPR will eventually approach the simple average match score per team/3 and then these two values should approach each other. They're in the same general range but still apart by 1.1 (or 11.4 - 10.3) with 10 matches played per team.)