Is OPR an accurate measurement system?

[CJ_Elliott]

Originally Posted by Jared Russell

Robot A is a robot that crosses 10 defenses per match and can therefore score (let's ignore auto for now) 50 points on their own per match.

Let's say that Robot A is far and away the best defense crosser at the event - every other team there can only cross 3 defenses per match on average.

In the matches with Robot A and two other robots, the alliance crosses 10 defenses with tons of time left over, and scores 50 points (plus whatever else during auto, from balls, and from endgame).

In the matches without Robot A, three average robots cross 9 defenses (3 each), and scores 45 points (plus whatever else during auto, from balls, and from endgame).

What are the OPRs of the robots at this event with respect to defenses? If we play infinite matches (and assume there are a lot of teams), we will eventually find that the "average" robot's defense OPR is ~1/3 of their average alliance score, so just north of 15 points (since the score is a bit higher in any matches with Robot A). Robot A, the world's best defense crossing robot, has an OPR of just under 20 (they account for one extra crossing per match)...<5 points higher than the OPR of a robot that is less than 1/3 as capable at this aspect of the game.

But my question still stands. How valid is OPR if there are teams that can be tossed out of rank just because of a poor schedule, their teammates being broken, etc?

[tindleroot]

Originally Posted by CJ_Elliott

But my question still stands. How valid is OPR if there are teams that can be tossed out of rank just because of a poor schedule, their teammates being broken, etc?

Since OPR is calculated under the implication that every team is playing at their normal ability every match, any situation where a team is playing below (or above) their ability is going to mess up OPR calculations not only for them but for other teams in their matches. Same goes for DPR (which essentially calculates how many points a team allows their opponents to score per match).

[Louisiana Jones]

Originally Posted by tindleroot

Since OPR is calculated under the implication that every team is playing at their normal ability every match, any situation where a team is playing below (or above) their ability is going to mess up OPR calculations not only for them but for other teams in their matches.

Some teams have systems where scouters track irregularities like broken robots/ penalties and include that data in the OPR calculations to make the calculation more accurately reflect the performance of robots.

[Chris is me]

Originally Posted by Louisiana Jones

Some teams have systems where scouters track irregularities like broken robots/ penalties and include that data in the OPR calculations to make the calculation more accurately reflect the performance of robots.

If you're actually scouting matches, you have basically no use for OPR. Actual objective data will beat it every time.

[Louisiana Jones]

Originally Posted by Chris is me

If you're actually scouting matches, you have basically no use for OPR. Actual objective data will beat it every time.

Its easy for a single scout to track dead robots or any other measure throughout a competition and apply it into an OPR calculator. While this isn't as good as having experienced scouts watching every match some teams might not have enough experienced students to scout every match.

I have found that actual objective data collected and used by our students to make decisions sometimes yields what some would consider questionable decisions.

Sometimes teams don't know what data should be collected or overlook a key feature of the game. OPR can help account for a team's contribution even if they aren't the robot actually scoring the points or they do something significant that isn't accounted for in the collected data.

[Jared Russell]

Originally Posted by CJ_Elliott

But my question still stands. How valid is OPR if there are teams that can be tossed out of rank just because of a poor schedule, their teammates being broken, etc?

If you assume that every robot makes the same contribution to the score in every match regardless of their teammates, it is a perfect model.

The less that assumption holds, the less the model is perfect. It's usually accurate for gross estimation of team ability (top quartile vs. bottom quartile, etc.) and for finding outliers (the rare team that is several standard deviations better than the mean), but I wouldn't trust it too much beyond that, especially early in the season (where match-to-match contributions tend to vary a lot).