Is OPR an accurate measurement system?

[KrazyCarl92]

OPR is a least squares solution to an over constrained matrix.

If you've ever done statistics at school, you can think of it sort of like a linear regression, but with more than two variables. If you've got 3 points that form a triangle on a scatter plot, you can't make a single line go through them all. So, you do a "best fit line" knowing there will be some error in your regression.

When there is a strong correlation between OPR and actual contribution like in this example:
http://www.mrholloman.net/SCP/Notes/...9/image006.png
OPR is very well suited to assess a team's point contribution in a match. We are most likely to see a strong correlation between OPR and actual point contribution in years when scoring is linear and non-excludable. For example, in 2013 if you scored a Frisbee in the high goal it was 3 points...no matter what. 2 Frisbees? 6 points. 10 Frisbees? 30 points. Additionally, one team scoring Frisbees usually did not prevent their partner from scoring Frisbees (except for some cases with FCS draining all discs from the Human Player Stations).

However, sometimes it is a weaker correlation, more like this:
http://surveyanalysis.org/images/thu...orrelation.png
This is usually observed when there is some non-linearity in scoring or excludability between partners. In this years game, defenses are non-linear (only count the first 2 times they are crossed) and excludable among partners (i.e. one team crossing the low bar twice excludes their partner from scoring points for doing so). Excludability, diminishing marginal returns, and plateaus for scoring are generally bad news for using OPR to predict scoring contribution. It gets more muddled when things like the incentives from the ranking system, the random pairing of alliances, etc. come into play. We have a lot of that this year.

In 2015, OPR was more useful because the limit of 3-7 Recycling Containers (depending on canburglarring) was less commonly hit than a breach is this year, especially in qualifying matches. Additionally, your sole ranking incentive was scoring as many points as possible. Thus there weren't really reasons to deviate from scoring as many points as you could all the time.

Bottom line is understand what OPR generally is before you use it. It IS a useful tool for somewhat understanding a team's relative contribution at an event (within some margin of error). It IS NOT a reasonable justification for picking a team with an OPR of 30 instead of another team with an OPR of 29. If you're comparing a team with an OPR of 40 to one with an OPR of 5 and there's a reasonable sample size? Sure, there's probably a good reason for the discrepancy.

[CJ_Elliott]

Originally Posted by KrazyCarl92

OPR is a least squares solution to an over constrained matrix.

If you've ever done statistics at school, you can think of it sort of like a linear regression, but with more than two variables. If you've got 3 points that form a triangle on a scatter plot, you can't make a single line go through them all. So, you do a "best fit line" knowing there will be some error in your regression.

When there is a strong correlation between OPR and actual contribution like in this example:
http://www.mrholloman.net/SCP/Notes/...9/image006.png
OPR is very well suited to assess a team's point contribution in a match. We are most likely to see a strong correlation between OPR and actual point contribution in years when scoring is linear and non-excludable. For example, in 2013 if you scored a Frisbee in the high goal it was 3 points...no matter what. 2 Frisbees? 6 points. 10 Frisbees? 30 points. Additionally, one team scoring Frisbees usually did not prevent their partner from scoring Frisbees (except for some cases with FCS draining all discs from the Human Player Stations).

However, sometimes it is a weaker correlation, more like this:
http://surveyanalysis.org/images/thu...orrelation.png
This is usually observed when there is some non-linearity in scoring or excludability between partners. In this years game, defenses are non-linear (only count the first 2 times they are crossed) and excludable among partners (i.e. one team crossing the low bar twice excludes their partner from scoring points for doing so). Excludability, diminishing marginal returns, and plateaus for scoring are generally bad news for using OPR to predict scoring contribution. It gets more muddled when things like the incentives from the ranking system, the random pairing of alliances, etc. come into play. We have a lot of that this year.

In 2015, OPR was more useful because the limit of 3-7 Recycling Containers (depending on canburglarring) was less commonly hit than a breach is this year, especially in qualifying matches. Additionally, your sole ranking incentive was scoring as many points as possible. Thus there weren't really reasons to deviate from scoring as many points as you could all the time.

Bottom line is understand what OPR generally is before you use it. It IS a useful tool for somewhat understanding a team's relative contribution at an event (within some margin of error). It IS NOT a reasonable justification for picking a team with an OPR of 30 instead of another team with an OPR of 29. If you're comparing a team with an OPR of 40 to one with an OPR of 5 and there's a reasonable sample size? Sure, there's probably a good reason for the discrepancy.

Ok. This actually makes some sense. I have just been looking at the top 15 at Northern Lights where I had been talking to our scouters constantly and I just didn't understand the OPR's on TBA were not what I saw or was told. But the linear regression idea makes a lot more since.

[throwaway]

Simple answer: It's not a great measurement this year but it's certainly better than the rankings

[saikiranra]

OPR using match scores can be misleading.

Finding component OPR numbers can be useful depending on what you are looking for.

[gblake]

To get the right answer, you first have to ask the right question ....

And

"... all models are wrong, but some are useful."

George Box

OPR is what is it is; and the OPR equations compute OPRs 100% accurately.

You need to ask/determine whether OPR is a useful tool for your purpose (or ask what things OPR is useful for).

I personally think that Chairman's Award submissions are a better (but still imperfect) tool to use than OPR is, if I'm (quoting the OP) searching for "... the best teams in FRC."

Blake

[Caleb Sykes]

Originally Posted by saikiranra

Finding component OPR numbers can be useful depending on what you are looking for.

Is anyone actually going to be computing component OPRs this year? I believe Ed Law is not doing that this year, so I need to find a new master scouting database to reference.

[Spoam]

Originally Posted by Caleb Sykes

Is anyone actually going to be computing component OPRs this year? I believe Ed Law is not doing that this year, so I need to find a new master scouting database to reference.

If you want this, I could calculate them. I'd just have to learn how the scoring system works this year (alumnus problems).

Originally Posted by Chris is me

If you're actually scouting matches, you have basically no use for OPR. Actual objective data will beat it every time.

QFT.

As far as how it works, the simplest explanation is that OPR is the assumption that every robot always contributes the same amount of points. This assumption is obviously false, but it's often close enough that it can still provide useful data.

Learning how this calculation is done is honestly not that hard, and it can provide some useful insight into the limitations and capabilities of OPR.

[Caleb Sykes]

Originally Posted by Spoam

If you want this, I could calculate them. I'd just have to learn how the scoring system works this year (alumnus problems).

Pretty pretty please do. I might do it myself if no one else does, but the earliest that would happen would be late next week when I am on spring break

[Caleb Sykes]

Originally Posted by Caleb Sykes

Is anyone actually going to be computing component OPRs this year? I believe Ed Law is not doing that this year, so I need to find a new master scouting database to reference.

It seems that 2834 is continuing to publish their scouting database without Ed Law. This has almost everything that I want. It would still be nice if there was a complete package that contained analysis of all of the information in the API (breach, capture, auto defense points, auto reach points, etc...)

[Ether]

Originally Posted by Caleb Sykes

It would still be nice if there was a complete package that contained analysis of all of the information in the API (breach, capture, auto defense points, auto reach points, etc...)

What kind of analysis are you looking for? Do you just mean component OPRs?

[Caleb Sykes]

Originally Posted by Ether

What kind of analysis are you looking for? Do you just mean component OPRs?

Yes.

[themccannman]

This comes up pretty much every year, the best answer is "it's okay", but the most important part is to understand how it's derived which Jared explained very well. If you understand what it actually means then it's a very useful tool.

[Ether]

Originally Posted by themccannman

the most important part is to understand how it's derived

The most important part is how much you can learn by studying how it's derived.

[Foster]

Originally Posted by CalTran

Karthik's views on OPR. YMMV.

"Baaaa"

I took a class this summer on Sabermetrics and the baseball world is just nuts over different models over different data sets. They are tracking temps, wind speeds, humidity, etc on top of all the other data that they gather.

I'd consider OPR to be a poor cousin to Sabermetrics WAR.

[Caleb Sykes]

Originally Posted by throwaway

Simple answer: It's not a great measurement this year but it's certainly better than the rankings

This. Too many people bash on OPR, and not enough people bash on the rankings.