Is OPR an accurate measurement system?

[Caleb Sykes]

Originally Posted by BrennanB

Because no ranking system is perfect, and people (usually) don't go around praising rankings like they are the god of all numbers.

People do that with OPR

I am aware that there are people who praise OPR much more than is justified. However, for every time I hear "OPR is the best thing ever," I hear twice as many things like "we were ranked 12th and didn't make elims, every other team must have horrible scouts because we clearly deserved that spot" or "I can't believe they picked the 40th ranked team as the second overall pick, I hope they know what they are doing." This tells me that far too many people don't realize just how badly the rankings generally reflect ability.

[Ether]

Originally Posted by themccannman

the most important part is to understand how it's derived

The most important part is how much you can learn by studying how it's derived.

[Foster]

Originally Posted by CalTran

Karthik's views on OPR. YMMV.

"Baaaa"

I took a class this summer on Sabermetrics and the baseball world is just nuts over different models over different data sets. They are tracking temps, wind speeds, humidity, etc on top of all the other data that they gather.

I'd consider OPR to be a poor cousin to Sabermetrics WAR.

[dmorewood]

Personally I don't think it is, this weekend we beat an alliance that had the 2 highest OPR's at the competition with robots that had the 12th and 15th OPR's.

[BrennanB]

Originally Posted by Caleb Sykes

I am aware that there are people who praise OPR much more than is justified. However, for every time I hear "OPR is the best thing ever," I hear twice as many things like "we were ranked 12th and didn't make elims, every other team must have horrible scouts because we clearly deserved that spot" or "I can't believe they picked the 40th ranked team as the second overall pick, I hope they know what they are doing." This tells me that far too many people don't realize just how badly the rankings generally reflect ability.

How many times have one of the first two picks at a regional been a sub 40/very low ranked team? I bet you couldn't find 5 examples in the last 10 years of that happening.

Sure lots of people do the "We were ranked high but didn't get picked" card, which yes it's true. However people don't use rankings to the same degree as OPR numbers for global comparison. People (several examples even today within the last few hours) on chief compare teams based on OPR values that didn't attend the same event. People never say "I was rank 5 at xxx regional you were rank 6 at yyyy regional, I must be better" However that happens with OPR all the time.

Secondly it's all good and fine to complain about things, but when you don't present a better solution to the problem what is the point? The problem with OPR is some people consider it as the law, and don't understand at all where it comes from or how it is calculated or what it's limitations are. The solution is to use it as a guidline, and watch the actual matches before you go around making conclusions.

So your problem is the ranking system isn't good enough to your satisfaction, and your solution is...?

That's all just things to think about the two differences between rankings and OPR and why people are more vocal about OPR. There isn't a solution to the rankings problem (aside from playing an infinite number of matches) that will properly sort the teams based on ability. There may be a better solution, but nothing will be perfect.

[Spoam]

Originally Posted by Caleb Sykes

Is anyone actually going to be computing component OPRs this year? I believe Ed Law is not doing that this year, so I need to find a new master scouting database to reference.

If you want this, I could calculate them. I'd just have to learn how the scoring system works this year (alumnus problems).

Originally Posted by Chris is me

If you're actually scouting matches, you have basically no use for OPR. Actual objective data will beat it every time.

QFT.

As far as how it works, the simplest explanation is that OPR is the assumption that every robot always contributes the same amount of points. This assumption is obviously false, but it's often close enough that it can still provide useful data.

Learning how this calculation is done is honestly not that hard, and it can provide some useful insight into the limitations and capabilities of OPR.

[Caleb Sykes]

Originally Posted by BrennanB

How many times have one of the first two picks at a regional been a sub 40/very low ranked team? I bet you couldn't find 5 examples in the last 10 years of that happening.

Defining "very low ranked" to be in the bottom third of teams at the event:
418 at Arkansas 2014
973 at Central Valley 2014
1796 at New York City 2014
1287 at North Carolina 2014
3986 at Montreal 2014

This is really beside the point though, I never claimed this to be a common occurrence. I made up an example to prove a point. I knew this was not a common occurence, but was just providing a statement which indicated lack of knowledge of the drawbacks of the ranking system.

Sure lots of people do the "We were ranked high but didn't get picked" card, which yes it's true. However people don't use rankings to the same degree as OPR numbers for global comparison. People (several examples even today within the last few hours) on chief compare teams based on OPR values that didn't attend the same event. People never say "I was rank 5 at xxx regional you were rank 6 at yyyy regional, I must be better" However that happens with OPR all the time.

Awesome, comparing OPRs between events is a useful thing to do. I don't really understand your point here. If you are saying that teams should be comparing ranks instead of OPRs that is silly, OPRs clearly trend better with future competitive success than ranks do, and I can prove it if I need to do so, but I thought that this was common knowledge. If you are saying that the statement "My OPR was 32 and yours was 31 therefore my robot must be better than yours" is silly, you are correct. That is a silly thing to say because absolute statements are always false. Statements like that don't make OPR any less valuable though.

Secondly it's all good and fine to complain about things, but when you don't present a better solution to the problem what is the point? The problem with OPR is some people consider it as the law, and don't understand at all where it comes from or how it is calculated or what it's limitations are. The solution is to use it as a guidline, and watch the actual matches before you go around making conclusions.

My issue really isn't with the ranking systems (well, I do have issues with FIRST's ranking systems, see below), my issue is that lots of people assume that the ranking does a good job of sorting teams. Just like you seem to have issue with people who use OPR without understanding its limitations, I have issue with people who use rankings less than 16** for things other than for the alliance selection process, and people who use rankings over 15 at all**. If someone really wants to know how they stack up to other teams at their event, and don't have a scouting database handy, they shouldn't even think about using rank instead of OPR.

So your problem is the ranking system isn't good enough to your satisfaction, and your solution is...?

This is getting off from the point a bit, but I'll answer anyway. The best solution is to get more matches in with fewer teams at an event. The next big improvement would be for FIRST to quit using screwy ranking algorithms* and just do WLT with cumulative auto points as the second sort. Finally, as a competitor, I would love it if FIRST used a modified Elo rating system to determine ranks. However, an optimized system would likely be too confusing to easily explain to outsiders, so I wouldn't want it.

That's all just things to think about the two differences between rankings and OPR and why people are more vocal about OPR. There isn't a solution to the rankings problem (aside from playing an infinite number of matches) that will properly sort the teams based on ability. There may be a better solution, but nothing will be perfect.

People should be more vocal about OPR than rank because OPR is a much better metric than rank.

Look, I don't want to be the defender of OPR. OPR has some very clear limitations that many people do not realize. OPR is just a tool, it doesn't fit every situation, but it certainly has uses. If some people don't want OPR in their metaphorical toolbox, that is fine, there are many other tools. However, if I see anyone that has rank in their toolbox and not OPR, I will try to convince them to use OPR. Hopefully someday every match at every event will be archived and teams will publicly share their scouting databases, and OPR will become obsolete. Until then, I will always use OPR to compare performance at events.


*2015 was a bit of an exception. Although it does fall under the "screwy ranking algorithms" umbrella, the average score system did a much better job than WLT for sorting teams, but then we had to have coopertition also, which just ruined it.

**For anything other than calculating points in a district system.

[Caleb Sykes]

Originally Posted by Spoam

If you want this, I could calculate them. I'd just have to learn how the scoring system works this year (alumnus problems).

Pretty pretty please do. I might do it myself if no one else does, but the earliest that would happen would be late next week when I am on spring break

[Crazyaimer]

Originally Posted by dmorewood

Personally I don't think it is, this weekend we beat an alliance that had the 2 highest OPR's at the competition with robots that had the 12th and 15th OPR's.

I think this is what CJ is really hinting at. Our robot was part of the winning alliance at northern lights and the double deccer, but none of the robots in our alliance really had that high of an OPR.

[Lil' Lavery]

Originally Posted by Chris is me

In games where the scoring actions of different teammates are more separable, like in 2010 or 2013, OPR is more accurate. In games where scoring actions are less separable, like 2014, OPR is much less accurate.

In terms of the sample sizes we're given, I absolutely agree.

However, over an infinitely large sample I'd love to see what OPR could do in a game like 2014. Jared made reference to "secondary effects" in his post, referring to freeing up resources on your alliance that could be spent elsewhere. That's the type of thing, in theory, OPR could be better at tracking than manual data entry. It's easy to manually track how teams complete objectives and directly impact the scoresheet, it's much tougher to determine how they impact the match in less obvious ways. The most obvious example is defense, which is very hard to quantify accurately (and "DPR" has rarely done a good job at it). In a game like 2014, where so much of the match is spent playing "away from the ball" (playing defense, positioning for the next cycle, blocking for teammates, etc), it can be really hard to determine how effective some teams are at impacting the score sheet. This is even true in professional sports, where broadcasters and analysts frequently talk about "intangibles" and how players impact the game in ways other than scoring (think good defensemen in hockey or offensive linemen in football, for instance). Sports have also turned to more advanced metrics to try and solve this, ranging from the sabremetrics movements in baseball and hockey to the motion tracking in basketball and soccer. That's the type of area where OPR/DPR/CCWM could potentially have significant value. However, a 12 match sample size (with random alliance partners/opponents) is nowhere near enough data to iron out the noise.

[Louisiana Jones]

Originally Posted by Crazyaimer

I think this is what CJ is really hinting at. Our robot was part of the winning alliance at northern lights and the double deccer, but none of the robots in our alliance really had that high of an OPR.

The captain of your alliance, 2883, had the third highest OPR at the event. The captain of the 1st seeded alliance had the 4th highest OPR at the event.

[Citrus Dad]

The spread on this year's OPRs appear to be quite compressed across the events so far (vs. last year which had the biggest spread ever). As a result, upsets are much more likely--the OPRs have error terms that are probably bigger than the apparent differences.

[CJ_Elliott]

Originally Posted by Louisiana Jones

The captain of your alliance, 2883, had the third highest OPR at the event. The captain of the 1st seeded alliance had the 4th highest OPR at the event.

Counter to this. If you take the highest absolute OPR that TBA could have our third alliance partner at; our alliance would only be scoring 97 points if you round up on all three teams given 5232 had a 28.26 which is .01 behind us.... we had a match of 170....

[mjc49]

A factor in this is that the game is played and scored differently in the elimination rounds. OPR is a measurement that may be considered when choosing alliance partners though for example, shooting is much more important in elimination rounds than it has been in qualifications to date. I would hope that the alliance captain has a plan to build the best alliance and make selections based on the way they believe and elimination round should be played knowing the game is scored and played differently.

[Caleb Sykes]

Originally Posted by CJ_Elliott

.... we had a match of 170....

No, you had a match of 125 points and also scored 45 points (capture, breach) that OPR doesn't consider.