Is OPR an accurate measurement system?
 

First off, I would like to start this with a warning. I am NOT trying to degrade any teams or try to say that they are not as good as they were, and i fully understand that teams do their best in this competition and I am proud to be able to associate myself with these teams.

Now. I don't know exactly how OPR is equated. But right now I am seeing a lot of people looking at OPR more than anything for finding the best teams in FRC.

Im looking at Northern Lights specifically because my team was there. But TBA has teams with OPR's that are significantly less than they should be. 5232 for example, was breaching their defenses with about half of the match left. Now. The lowest OPR out of the 15 is 28.27 points... now. The largest miss in my mind is 5232 (Talons). Lets say 5232 was challenging (which they were), and breaking 3 defenses (which they were). Even outside of elims where you get 20 points for the breach, their contribution to their alliance is 35... 5232 isn't even in the top 15. Now i get that there can be problems with a system but in Northern Lights alone I can come up with 5 or 6 teams off the top of my head that should be ranked higher.

So if someone could explain this or help me understand how OPR is this amazing system for ranking teams when I am not seeing it accurately representing teams, i question it's validity. (And I highly urge people to watch Northern Lights and see once the videos come out)

Re: Is OPR an accurate measurement system?
 

OPR is not an amazing system for ranking teams. It is a statistical calculation that generally relates the performance level of teams, but it should never be trusted for accuracy.

Re: Is OPR an accurate measurement system?
 

Ed law in the past has created some amazing data sheets. He along with these data sheets also made http://file:///home/chronos/u-8e3fa3...ation_2014.pdf
this cool powerpoint explaining opr and ccwm

Re: Is OPR an accurate measurement system?
 

Robot A is a robot that crosses 10 defenses per match and can therefore score (let's ignore auto for now) 50 points on their own per match.

Let's say that Robot A is far and away the best defense crosser at the event - every other team there can only cross 3 defenses per match on average.

In the matches with Robot A and two other robots, the alliance crosses 10 defenses with tons of time left over, and scores 50 points (plus whatever else during auto, from balls, and from endgame).

In the matches without Robot A, three average robots cross 9 defenses (3 each), and scores 45 points (plus whatever else during auto, from balls, and from endgame).

What are the OPRs of the robots at this event with respect to defenses? If we play infinite matches (and assume there are a lot of teams), we will eventually find that the "average" robot's defense OPR is ~1/3 of their average alliance score, so just north of 15 points (since the score is a bit higher in any matches with Robot A). Robot A, the world's best defense crossing robot, has an OPR of just under 20 (they account for one extra crossing per match)...<5 points higher than the OPR of a robot that is less than 1/3 as capable at this aspect of the game.

This is obviously an oversimplification, but it goes to show that because of the finite number of crossings that can be rewarded per match, excelling at this aspect of the game does not actually get that well rewarded on the scoreboard (and it will be even less rewarded as the season goes on and drivetrains have their kinks ironed out). This of course does not factor in second-order benefits like an exceptional crosser freeing up teammates to score balls, etc.

Re: Is OPR an accurate measurement system?
 

You linked to a local file on your computer, I think you meant this.

Re: Is OPR an accurate measurement system?
 

What OPR / CCWM always excelled at wasn't in being an absolute ranking of teams. It is a better sort than average score or ranking.

In games where the scoring actions of different teammates are more separable, like in 2010 or 2013, OPR is more accurate. In games where scoring actions are less separable, like 2014, OPR is much less accurate.

OPR never will be better than actual data at ranking the quality of teams, and a team's OPR will never exactly match the team's actual scoring point. It's just a rough starting point that is a better place to start than other methods in the absence of actual match data.

Re: Is OPR an accurate measurement system?
 

But my question still stands. How valid is OPR if there are teams that can be tossed out of rank just because of a poor schedule, their teammates being broken, etc?

Re: Is OPR an accurate measurement system?
 

Karthik's views on OPR. YMMV.

Re: Is OPR an accurate measurement system?
 

Yep you're right. I always forget chromebooks do stupid things. It seems like they randomly generated a link for that pdf that I downloaded since chromebooks do hardly anything offline

Re: Is OPR an accurate measurement system?
 

Since OPR is calculated under the implication that every team is playing at their normal ability every match, any situation where a team is playing below (or above) their ability is going to mess up OPR calculations not only for them but for other teams in their matches. Same goes for DPR (which essentially calculates how many points a team allows their opponents to score per match).

Re: Is OPR an accurate measurement system?
 

If you assume that every robot makes the same contribution to the score in every match regardless of their teammates, it is a perfect model.

The less that assumption holds, the less the model is perfect. It's usually accurate for gross estimation of team ability (top quartile vs. bottom quartile, etc.) and for finding outliers (the rare team that is several standard deviations better than the mean), but I wouldn't trust it too much beyond that, especially early in the season (where match-to-match contributions tend to vary a lot).

Re: Is OPR an accurate measurement system?
 

OPR is a least squares solution to an over constrained matrix.

If you've ever done statistics at school, you can think of it sort of like a linear regression, but with more than two variables. If you've got 3 points that form a triangle on a scatter plot, you can't make a single line go through them all. So, you do a "best fit line" knowing there will be some error in your regression.

When there is a strong correlation between OPR and actual contribution like in this example:
http://www.mrholloman.net/SCP/Notes/...9/image006.png
OPR is very well suited to assess a team's point contribution in a match. We are most likely to see a strong correlation between OPR and actual point contribution in years when scoring is linear and non-excludable. For example, in 2013 if you scored a Frisbee in the high goal it was 3 points...no matter what. 2 Frisbees? 6 points. 10 Frisbees? 30 points. Additionally, one team scoring Frisbees usually did not prevent their partner from scoring Frisbees (except for some cases with FCS draining all discs from the Human Player Stations).

However, sometimes it is a weaker correlation, more like this:
http://surveyanalysis.org/images/thu...orrelation.png
This is usually observed when there is some non-linearity in scoring or excludability between partners. In this years game, defenses are non-linear (only count the first 2 times they are crossed) and excludable among partners (i.e. one team crossing the low bar twice excludes their partner from scoring points for doing so). Excludability, diminishing marginal returns, and plateaus for scoring are generally bad news for using OPR to predict scoring contribution. It gets more muddled when things like the incentives from the ranking system, the random pairing of alliances, etc. come into play. We have a lot of that this year.

In 2015, OPR was more useful because the limit of 3-7 Recycling Containers (depending on canburglarring) was less commonly hit than a breach is this year, especially in qualifying matches. Additionally, your sole ranking incentive was scoring as many points as possible. Thus there weren't really reasons to deviate from scoring as many points as you could all the time.

Bottom line is understand what OPR generally is before you use it. It IS a useful tool for somewhat understanding a team's relative contribution at an event (within some margin of error). It IS NOT a reasonable justification for picking a team with an OPR of 30 instead of another team with an OPR of 29. If you're comparing a team with an OPR of 40 to one with an OPR of 5 and there's a reasonable sample size? Sure, there's probably a good reason for the discrepancy.

Re: Is OPR an accurate measurement system?
 

Ok. This actually makes some sense. I have just been looking at the top 15 at Northern Lights where I had been talking to our scouters constantly and I just didn't understand the OPR's on TBA were not what I saw or was told. But the linear regression idea makes a lot more since.

Re: Is OPR an accurate measurement system?
 

Simple answer: It's not a great measurement this year but it's certainly better than the rankings

Re: Is OPR an accurate measurement system?
 

OPR using match scores can be misleading.

Finding component OPR numbers can be useful depending on what you are looking for.

Re: Is OPR an accurate measurement system?
 

Would you like to learn? I can post some links to discussion threads here on CD that are written at an accessible level.

... and if you have any questions I -- and others I'm sure -- would be glad to answer them.

Re: Is OPR an accurate measurement system?
 

To get the right answer, you first have to ask the right question ....

And

"... all models are wrong, but some are useful."

George Box

OPR is what is it is; and the OPR equations compute OPRs 100% accurately.

You need to ask/determine whether OPR is a useful tool for your purpose (or ask what things OPR is useful for).

I personally think that Chairman's Award submissions are a better (but still imperfect) tool to use than OPR is, if I'm (quoting the OP) searching for "... the best teams in FRC." ;)

Blake

Re: Is OPR an accurate measurement system?
 

Some teams have systems where scouters track irregularities like broken robots/ penalties and include that data in the OPR calculations to make the calculation more accurately reflect the performance of robots.

Re: Is OPR an accurate measurement system?
 

This. Too many people bash on OPR, and not enough people bash on the rankings.

Re: Is OPR an accurate measurement system?
 

Is anyone actually going to be computing component OPRs this year? I believe Ed Law is not doing that this year, so I need to find a new master scouting database to reference.

Re: Is OPR an accurate measurement system?
 

With the data that FIRST provides through the FRC Event API, we can certainly do much better than your typical OPR. For example, I can pull down that data and know exactly which defenses were on the field and which of those were crossed and damaged in every match any team played in. I can know exactly how many balls were scored in which goals, how many robots challenged, and how many robots climbed. Proper statistical analysis (think OPR, but for each individual category instead of just overall score) can get you much more detailed and specific data. It won't be the whole story, but I would be willing to bet it would be more accurate than just the overall OPR. And more useful in assembling an eliminations alliance with the capabilities you want.

Re: Is OPR an accurate measurement system?
 

Taking Northern Lights as an example the first robot that was picked had the Highest OPR, but was ranked 16th.

While 5232 helped their alliance win and probably was a good bargain for when they were picked, it appears that most teams at the event didn't think they were a top 15 team, otherwise they would have been picked sooner.

Re: Is OPR an accurate measurement system?
 

But they probably had the best challenge we will see all year in the double DECCer :). That was insane

Re: Is OPR an accurate measurement system?
 

OPR is no worse or no better than how they rank teams in sports.
Any given Saturday or Sunday?
NCAA March Madness tournament? 5 vs. 12 seeds? etc...

Re: Is OPR an accurate measurement system?
 

Because no ranking system is perfect, and people (usually) don't go around praising rankings like they are the god of all numbers.

People do that with OPR

Re: Is OPR an accurate measurement system?
 

If you're actually scouting matches, you have basically no use for OPR. Actual objective data will beat it every time.

Re: Is OPR an accurate measurement system?
 

My favorite statement on OPR will always be this comment from Karthik's 2012 Effective FIRST Strategies presentation.

Take OPR with a couple grains of salt and understand, within that year's game, why it might not be accurate. (Scroll back to ~50min in that video if you want to hear the whole spiel. Better yet, watch the whole thing.)

Re: Is OPR an accurate measurement system?
 

Or you could just go directly to 57:44 :rolleyes: .

Re: Is OPR an accurate measurement system?
 

Its easy for a single scout to track dead robots or any other measure throughout a competition and apply it into an OPR calculator. While this isn't as good as having experienced scouts watching every match some teams might not have enough experienced students to scout every match.

I have found that actual objective data collected and used by our students to make decisions sometimes yields what some would consider questionable decisions.

Sometimes teams don't know what data should be collected or overlook a key feature of the game. OPR can help account for a team's contribution even if they aren't the robot actually scoring the points or they do something significant that isn't accounted for in the collected data.

Re: Is OPR an accurate measurement system?
 

This comes up pretty much every year, the best answer is "it's okay", but the most important part is to understand how it's derived which Jared explained very well. If you understand what it actually means then it's a very useful tool.

Re: Is OPR an accurate measurement system?
 

I am aware that there are people who praise OPR much more than is justified. However, for every time I hear "OPR is the best thing ever," I hear twice as many things like "we were ranked 12th and didn't make elims, every other team must have horrible scouts because we clearly deserved that spot" or "I can't believe they picked the 40th ranked team as the second overall pick, I hope they know what they are doing." This tells me that far too many people don't realize just how badly the rankings generally reflect ability.

Re: Is OPR an accurate measurement system?
 

The most important part is how much you can learn by studying how it's derived.

Re: Is OPR an accurate measurement system?
 

"Baaaa"

I took a class this summer on Sabermetrics and the baseball world is just nuts over different models over different data sets. They are tracking temps, wind speeds, humidity, etc on top of all the other data that they gather.

I'd consider OPR to be a poor cousin to Sabermetrics WAR.

Re: Is OPR an accurate measurement system?
 

Personally I don't think it is, this weekend we beat an alliance that had the 2 highest OPR's at the competition with robots that had the 12th and 15th OPR's.

Re: Is OPR an accurate measurement system?
 

How many times have one of the first two picks at a regional been a sub 40/very low ranked team? I bet you couldn't find 5 examples in the last 10 years of that happening.

Sure lots of people do the "We were ranked high but didn't get picked" card, which yes it's true. However people don't use rankings to the same degree as OPR numbers for global comparison. People (several examples even today within the last few hours) on chief compare teams based on OPR values that didn't attend the same event. People never say "I was rank 5 at xxx regional you were rank 6 at yyyy regional, I must be better" However that happens with OPR all the time.

Secondly it's all good and fine to complain about things, but when you don't present a better solution to the problem what is the point? The problem with OPR is some people consider it as the law, and don't understand at all where it comes from or how it is calculated or what it's limitations are. The solution is to use it as a guidline, and watch the actual matches before you go around making conclusions.

So your problem is the ranking system isn't good enough to your satisfaction, and your solution is...?

That's all just things to think about the two differences between rankings and OPR and why people are more vocal about OPR. There isn't a solution to the rankings problem (aside from playing an infinite number of matches) that will properly sort the teams based on ability. There may be a better solution, but nothing will be perfect.

Re: Is OPR an accurate measurement system?
 

If you want this, I could calculate them. I'd just have to learn how the scoring system works this year (alumnus problems).

QFT.

As far as how it works, the simplest explanation is that OPR is the assumption that every robot always contributes the same amount of points. This assumption is obviously false, but it's often close enough that it can still provide useful data.

Learning how this calculation is done is honestly not that hard, and it can provide some useful insight into the limitations and capabilities of OPR.

Re: Is OPR an accurate measurement system?
 

Defining "very low ranked" to be in the bottom third of teams at the event:
418 at Arkansas 2014
973 at Central Valley 2014
1796 at New York City 2014
1287 at North Carolina 2014
3986 at Montreal 2014

This is really beside the point though, I never claimed this to be a common occurrence. I made up an example to prove a point. I knew this was not a common occurence, but was just providing a statement which indicated lack of knowledge of the drawbacks of the ranking system.

Awesome, comparing OPRs between events is a useful thing to do. I don't really understand your point here. If you are saying that teams should be comparing ranks instead of OPRs that is silly, OPRs clearly trend better with future competitive success than ranks do, and I can prove it if I need to do so, but I thought that this was common knowledge. If you are saying that the statement "My OPR was 32 and yours was 31 therefore my robot must be better than yours" is silly, you are correct. That is a silly thing to say because absolute statements are always false. Statements like that don't make OPR any less valuable though.

My issue really isn't with the ranking systems (well, I do have issues with FIRST's ranking systems, see below), my issue is that lots of people assume that the ranking does a good job of sorting teams. Just like you seem to have issue with people who use OPR without understanding its limitations, I have issue with people who use rankings less than 16** for things other than for the alliance selection process, and people who use rankings over 15 at all**. If someone really wants to know how they stack up to other teams at their event, and don't have a scouting database handy, they shouldn't even think about using rank instead of OPR.

This is getting off from the point a bit, but I'll answer anyway. The best solution is to get more matches in with fewer teams at an event. The next big improvement would be for FIRST to quit using screwy ranking algorithms* and just do WLT with cumulative auto points as the second sort. Finally, as a competitor, I would love it if FIRST used a modified Elo rating system to determine ranks. However, an optimized system would likely be too confusing to easily explain to outsiders, so I wouldn't want it.

People should be more vocal about OPR than rank because OPR is a much better metric than rank.

Look, I don't want to be the defender of OPR. OPR has some very clear limitations that many people do not realize. OPR is just a tool, it doesn't fit every situation, but it certainly has uses. If some people don't want OPR in their metaphorical toolbox, that is fine, there are many other tools. However, if I see anyone that has rank in their toolbox and not OPR, I will try to convince them to use OPR. Hopefully someday every match at every event will be archived and teams will publicly share their scouting databases, and OPR will become obsolete. Until then, I will always use OPR to compare performance at events.


*2015 was a bit of an exception. Although it does fall under the "screwy ranking algorithms" umbrella, the average score system did a much better job than WLT for sorting teams, but then we had to have coopertition also, which just ruined it.

**For anything other than calculating points in a district system.

Re: Is OPR an accurate measurement system?
 

Pretty pretty please do. I might do it myself if no one else does, but the earliest that would happen would be late next week when I am on spring break

Re: Is OPR an accurate measurement system?
 

I think this is what CJ is really hinting at. Our robot was part of the winning alliance at northern lights and the double deccer, but none of the robots in our alliance really had that high of an OPR.

Re: Is OPR an accurate measurement system?
 

In terms of the sample sizes we're given, I absolutely agree.

However, over an infinitely large sample I'd love to see what OPR could do in a game like 2014. Jared made reference to "secondary effects" in his post, referring to freeing up resources on your alliance that could be spent elsewhere. That's the type of thing, in theory, OPR could be better at tracking than manual data entry. It's easy to manually track how teams complete objectives and directly impact the scoresheet, it's much tougher to determine how they impact the match in less obvious ways. The most obvious example is defense, which is very hard to quantify accurately (and "DPR" has rarely done a good job at it). In a game like 2014, where so much of the match is spent playing "away from the ball" (playing defense, positioning for the next cycle, blocking for teammates, etc), it can be really hard to determine how effective some teams are at impacting the score sheet. This is even true in professional sports, where broadcasters and analysts frequently talk about "intangibles" and how players impact the game in ways other than scoring (think good defensemen in hockey or offensive linemen in football, for instance). Sports have also turned to more advanced metrics to try and solve this, ranging from the sabremetrics movements in baseball and hockey to the motion tracking in basketball and soccer. That's the type of area where OPR/DPR/CCWM could potentially have significant value. However, a 12 match sample size (with random alliance partners/opponents) is nowhere near enough data to iron out the noise.

Re: Is OPR an accurate measurement system?
 

The captain of your alliance, 2883, had the third highest OPR at the event. The captain of the 1st seeded alliance had the 4th highest OPR at the event.

Re: Is OPR an accurate measurement system?
 

The spread on this year's OPRs appear to be quite compressed across the events so far (vs. last year which had the biggest spread ever). As a result, upsets are much more likely--the OPRs have error terms that are probably bigger than the apparent differences.

Re: Is OPR an accurate measurement system?
 

Counter to this. If you take the highest absolute OPR that TBA could have our third alliance partner at; our alliance would only be scoring 97 points if you round up on all three teams given 5232 had a 28.26 which is .01 behind us.... we had a match of 170....

Re: Is OPR an accurate measurement system?
 

A factor in this is that the game is played and scored differently in the elimination rounds. OPR is a measurement that may be considered when choosing alliance partners though for example, shooting is much more important in elimination rounds than it has been in qualifications to date. I would hope that the alliance captain has a plan to build the best alliance and make selections based on the way they believe and elimination round should be played knowing the game is scored and played differently.

Re: Is OPR an accurate measurement system?
 

No, you had a match of 125 points and also scored 45 points (capture, breach) that OPR doesn't consider.

Re: Is OPR an accurate measurement system?
 

What language (Python perhaps) do you use to access the FRC API?

Re: Is OPR an accurate measurement system?
 

It seems that 2834 is continuing to publish their scouting database without Ed Law. This has almost everything that I want. It would still be nice if there was a complete package that contained analysis of all of the information in the API (breach, capture, auto defense points, auto reach points, etc...)

Re: Is OPR an accurate measurement system?
 

What kind of analysis are you looking for? Do you just mean component OPRs?

Re: Is OPR an accurate measurement system?
 

Yes.