Re: "standard error" of OPR values
 

Kind of reminds me of a joke I heard this past weekend that was accidentally butchered:

A physicist, engineer and a statistician are out hunting. Suddenly, a deer appears 50 yards away.

The physicist does some basic ballistic calculations, assuming a vacuum, lifts his rifle to a specific angle, and shoots. The bullet lands 5 yards short.

The engineer adds a fudge factor for air resistance, lifts his rifle slightly higher, and shoots. The bullet lands 5 yards long.

The statistician yells "We got him!"
************************************************** ********

A really interesting read into "what is important" from stats in basketball:
http://www.nytimes.com/2009/02/15/ma...ewanted=1&_r=0

+/- system is probably the most similar "stat" to OPR utilized in basketball. It is figured a different way, but is a good way of estimating impact from a player vs. just using points/rebounds and....

The article does a really good job of doing some comparison to a metric like that to more typical event driven stats to actual impactful details of a particularly difficult to scout player.

I really enjoy the line where it discusses trying to find undervalued mid pack players. Often with scouting, this is exactly what you too are trying to do. Rank the #16-#24 team at an event as accurately as possible in order to help foster your alliances best chance at advancing.

If you enjoy this topic, enjoy the article, and have not read Moneyball, it is well worth the read. I enjoyed the movie, but the book is so much better about the details.

Re: "standard error" of OPR values
 

Of course! I'm absolutely successful everytime I go hunting!

There's an equivalent economists' joke in which trying to feed a group on a desert island ends with "assume a can opener!":D
************************************************** ********

Wholly endorse Moneyball to anyone reading this thread. It's what FRC scouting is all about. We call our system "MoneyBot."

In baseball, this use of statistics is called "sabremetrics." Bill James is the originator of this method.

Re: "standard error" of OPR values
 

Getting back to the original question:


So for those of you who answered "yes":

Pick an authoritative (within the field of statistics) definition for standard error, and compute that "standard error" for each Team's OPR for the attached example.

Re: "standard error" of OPR values
 

... and for those of you who think the answer is "no", explain why none of the well-defined "standard errors" (within the field of statistics) can be meaningfully applied to the example data (provided in the linked post) in a statistically valid way.

Re: "standard error" of OPR values
 

Here's a poor-man's approach to approximating the error of the OPR value calculation (as opposed to the prediction error aka regression error):

1. Collect all of a team's match results.

2. Compute the normal OPR.

3. Then, re-compute the OPR but excluding the result from the first match.

4. Repeat this process by removing the results from only the 2nd match, then only the 3rd, etc. This will give you a set of OPR values computed by excluding a single match. So for example, if a team played 6 matches, there would be the original OPR plus 6 additional "OPR-" values.

5. Compute the standard deviation of the set of OPR- values. This should give you some idea of how much variability a particular match contributes to the team's OPR. Note that this will even vary team-by-team.

Thoughts?

Re: "standard error" of OPR values
 

This is interesting but not what I'm looking for.

The question is this thread is how (or if) a standard, textbook, widely-used, statistically valid "standard error" (as mention by Citrus Dad and quoted in the original post in this thread) can be computed for OPR from official FRC qual match results data unsupplemented by manual scouting data or any other data.

Re: "standard error" of OPR values
 

Using Ether's data, I just did essentially this, where I randomly* selected 200 (I just chose this because it excludes enough matches to ensure variation in OPRs, but should include enough matches to keep the system sufficiently over-determined) of the 254 alliance scores to use for the OPR calculation. I ran this 200 times and got the following:
Original OPR is just copied straight from Ether's OPR.csv; Mean OPR is just the mean OPR that a team received across the 200 iterations; Standard Deviation is the Standard Deviation of all the OPRs a team recieved and the final column is just the standard deviation divided by the Mean OPR. The data is sorted by Mean OPR.

In terms of whether this is a valid way of looking at it, I'm not sure--the results seem to have some meaning, but I'm not sure how much of it is just that only looking at 200 scores is even worse than just 254, or if there is something more meaningful going on.

*Using python's random.sample() function. This means that I did nothing to prevent duplicate runs (which are extremely unlikely; 254 choose 200 is ~7.2 * 10^55) and nothing to ensure that a team didn't "play" <3 times in the selection of 200 scores.

Re: "standard error" of OPR values
 

I guess I'm not sure how you're defining "standard error." I assume you're trying to get some confidence on the OPR value itself (not in how well the OPR can predict match results, which is the other error I referred to previously).

The method I propose above gives a standard deviation measure on how much a single match changes a team's OPR. I would think this is something like what you want. If not, can you define what you're looking for more precisely?

Also, rather than taking 200 of 254 matches and looking at the standard deviation of all OPRs, I suggest just removing a single match (e.g., compute OPR based on 253 of the 254 matches) and looking at how that removal affects only the OPRs of the teams involved in the removed match.

So if you had 254 matches in a tournament, you'd compute 254 different sets of OPRs (1 for each possible match removal) and then look at the variability of the OPRs only for the teams involved in each specific removed match.

This only uses the actual qualification match results, no scouting or other data as you want.

Re: "standard error" of OPR values
 

And just to make sure I'm being clear (because I fear that I may not be):

Let's say that team 1234 played in a tournament and was involved in matches 5, 16, 28, 39, 51, and 70.

You compute team 1234's OPR using all matches except match 5. Say it's 55.
Then you compute team 1234's OPR using all matches except match 16. Say it's 60.
Keep repeating this, removing each of that team's matches, which will give you 6 different OPR numbers. Let's say that they're 55, 60, 50, 44, 61, and 53. Then you can compute the standard deviation of those 6 numbers to give you a confidence on what team 1234's OPR is.

Of course, you can do this for every team in the tournament and get team-specific OPR standard deviations and an overall tournament OPR standard deviation.

Team 1234 may have a large standard deviation (because maybe 1/3 of the time they always knock over a stack in the last second) while team 5678 may have a small standard deviation (because they always contribute the exactly same point value to their alliance's final score).

And hopefully the standard deviations will be lower in tournaments with more matches per team because you have more data points to average.

Re: "standard error" of OPR values
 

I am not defining "standard error".

I am asking you (or anyone who cares to weigh in) to pick a definition from an authoritative source and use that definition to compute said standard errors of the OPRs (or state why not):

Re: "standard error" of OPR values
 

No. I am not trying to do this. I will try to be clearer:

Citrus Dad asked why no-one ever reports "the" standard error for the OPRs.

"Standard Error" is a concept within the field of statistics. There are several well-defined meanings depending on the context.

So what am trying to do is this: have a discussion about what "the" standard error might mean in the context of OPR.

Re: "standard error" of OPR values
 

No, your original post was quite clear. And interesting. But it's not what I am asking about.

Re: "standard error" of OPR values
 

@ Citrus Dad:

If you are reading this thread, would you please weigh in here and reveal what you mean by "the standard errors" of the OPRs, and how you would compute them, using only the data in the example I posted?

Also, what assumptions do you have to make about the data and the model in order for the computed standard errors to be statistically valid/relevant/meaningful, and what is the statistical meaning of those computed errors?

Re: "standard error" of OPR values
 

Let us assume that the model OPR uses is a good description of FRC match performance - that is, match scores are given by a linear sum of team performance values, and each team's performance in a given match is described by a random variable whose distribution is identical between matches.

OPR should then yield an estimate of the mean of this distribution. An estimate of the standard deviation can be obtained, as mentioned, by taking the RMS of the residuals.

To approximate the standard deviation of the mean (which is what is usually meant by "standard error" of these sorts of measurements), one would then divide this by sqrt(n) (for those interested in a proof of this, simply consider the fact that when summing random variables, variances add), where n is the number of matches used in the team's OPR calculation.

This, of course, fails if the assumptions we made at the outset aren't good (e.g. OPR is not a good model of team performance). Moreover, even if the assumptions hold, if the distribution of the random variable describing a team's performance in a given match is sufficiently wonky that the distribution of the mean is not particularly Gaussian then one is fairly limited in the conclusions they can draw from the standard deviation, anyway.

Re: "standard error" of OPR values
 

I am interested in a proof of this, because I don't think the normal assumptions hold. Can you explain this more in the full context of how OPR is computed? [Edit: I spent more time trying to derive this whole thing. See my next posts for an attempt at the derivation].

What you say holds if one is taking a number of independent, noisy measurements of a value and computing the mean of the measurements as the estimate of the underlying value. So that would work if OPR was computed by simply averaging the match scores for a team (and dividing by 3 to accommodate for 1/3 of the match score being due to each team's contribution).

But that's not the way OPR is computed at all. It's computed using linear regressions and all of the OPRs for all of the teams are computed simultaneously in one big matrix operation.

For example, it isn't clear to me what n should be. You say "n is the number of matches used in the team's OPR calculation." But all OPRs are computed at the same time using all of the available match data. Does n count matches that a team didn't play in, but that are still used in the computation? Is n the number of matches a team has played? Or the total matches? OPR can be computed based on whatever matches have already occurred at any time. So if some teams have played 4 matches and some have played 5, it would seem like the OPRs for the teams that have played fewer matches should have more uncertainty than the OPRs for the teams that have played more. And the fact that the computation is all intertwined and that the OPRs for different teams are not independent (e.g., if one alliance has a huge score in one match, that affects 3 OPRs directly and the rest of them indirectly through the computation) seems to make the standard assumptions and arguments suspect.

Thoughts?