Error bar of OPR in Stronghold

[Whatever]

Stronghold was my first year as a mentor in FRC. I have been interested in the OPR statistic. One thing I have noticed is it commonly gets quoted to 2 decimal places which seems like way too much accuracy. So the question I have been asking myself is what is the error bar on OPR (at least for Stronghold)?


As a starting point I looked at the Einstein alliances. My thought process being those alliances have a bunch of matches together and their qual OPR would have the least oddities. I looked at division and Einstein matches where the 2nd pick of the alliance was on field and there were no red cards in the match (either side). I also assumed the alliance would breach and capture so the expected score for an alliance was the sum of the 3 OPRs plus 45.

The standard deviations of the actual scores averaged 24.6 for those alliances (min 18.2, max 34.3). The swing (max score-min score) of scores for each alliance averaged 77.9 points (min 55, max 107). 7 of the 8 alliances had an average score higher than their expected. 2 of the 8 alliances had an average score more than 30 points greater than their expected.


For a thought experiment, I created a spreadsheet that would calculate the OPRs of a small scale event. The event has 6 teams and each would play 10 matches. This meant every possible lineup is included. The match scores were the sum of 3 alliance OPRs plus a random +-25 points (roughly the average stdev of scores from the Einstein alliances). The true OPRs of the 6 teams was fixed at 60,50,40,30,20,10. After 126 events, the average OPRs for each team was under 1 away from their true OPR. The max error was just a shade over 17. Out of the 126 events, OPR had all 6 teams in the correct order 59 times. In 4 cases a team was above the team 2 places above them in true OPR.

I did sanity check the +-25 against the early week regional winning alliances. Backing out the breach/capture points, those alliances showed swings of 50 for those alliances over 6 to 8 matches. It should be noted, I used the random function in excel so the distribution of the randoms is flat and not gaussian.

So I would estimate the error bar on OPR (for Stronghold) at around 15.

Does that seem to align to other people's thoughts?
Has anyone tried to do an analysis on this?

[SpaceBiz]

The margin of error (or error bar as you put it) would also vary depending on how high the average OPR for the event you are looking at is. Let's say Team 9001 that did not contribute any points to their alliances in quals has a low "true" OPR of 0. 9001 can have a negative OPR, but this does not reflect their "true OPR" of 0. Plus or minus the some of the OPRs below zero gives you a good idea of the margin of error for lower levels of play.

When an event averages more points, there is normally a larger margin in total score, and inherently makes for a larger margin of error. These higher levels of play are unlikely to have teams with "real" OPRs below zero, so you can't assume the margin of error is "plus or minus the lowest OPR below zero". Because the exact margin of error is not really important, I would just understand the margin of error at higher levels of play is a bit larger than the margin of error at lower levels of play.

However, I am making some assumptions that are not great if we wanted to calculate the margin of error to a more exact number.

In my opinion, the best way to determine the margin of error for OPR would be to look at DPR for an event where almost no one tried defense in quals. Off the top of my head, I don't know where this happened.

My guess is the margin of error for most events is plus or minus 7 at most with OPR only meaning anything compared to other teams if it is larger by a full point. Two decimal places definitely imply that OPR is much more accurate than it truly is.

[Brian Maher]

Relevant thread from last year.

Originally Posted by SpaceBiz

Plus or minus the some of the OPRs below zero gives you a good idea of the margin of error for lower levels of play.

I would be cautious about using this as a means of assessing error. It shouldn't be assumed that negative OPR indicates error. There are valid reasons for having a negative OPR: it means that, on average, that team causes their alliance partners to score worse than they otherwise have. This is especially likely during low levels of play, while drive teams are more clumsy and still learning how to stay out of partners' ways.

The consequence of negative OPR was more obvious last year, when it possible to undo any teleop scoring attempt up until the last second of the game by knocking over stacks, but it is mathematically sound for it to happen this year as well. Teams can end up with a negative OPR this year by being in the way of their partners' scoring attempts without scoring much themselves.

For example, say there's a team that doesn't score much and has a propensity for getting stuck under the low bar. Many teams use the low bar to quickly cycle and score goals. This team may prevent their partners from doing so, and from finishing damaging the low goal. If they score very little and this happens often enough with enough low bar cycler partners, there OPR will become a (deserved) negative number, because they contribute negatively to their scores.

[Whatever]

I should define "error bar". In this context I mean error bar to mean when comparing two robots, Robot A with an OPR of X and Robot B with an OPR of X+Y - what does Y have to be to reasonable confident that B will be a better robot in it's next match (assuming both robots run).


Thanks for the pointer to lasts years thread. Now I have some reading to do.

[MooreteP]

Originally Posted by Whatever

Stronghold was my first year as a mentor in FRC. I have been interested in the OPR statistic. One thing I have noticed is it commonly gets quoted to 2 decimal places which seems like way too much accuracy.

Thank you for your post, glad to have you involved in FIRST

SigFigs, or significant digits are maligned by students and professionals.
By definition: "Significant Figures are used in measurements and calculations. They indicate the numbers that have meaning.
The limits of precision of the measurement tool limit the number of digits that have meaning in any calculated result.
A calculated result cannot be more precise than the least precise measurement."
Excessive digits are hyperbole.
Like curse words, they are sometimes used to imply more weight to ones opinions or answers than is justified by the underlying data.
I can at times surmise a students confidence in their calculated answers by the number of digits it contains. (BTW, inverse)

Your error bar analysis and scenarios is a good exercise.
It would be interesting to see more analyses that may tease out variabilities.

One issue with "Obsessive Performance" Ratio is the sample size.
Another wildcard is the design iterations that occur over a season.
And there is always the strategy of picking complementary designs for an alliance.

This is where scouting becomes an intellectual exercise. There are many sources of data, OPR is a good quantitative metric, but dynamic performance at an immediate competition (a Robot getting much better through "piteration" and drive time, a Robot experiencing cascading crucial component failures, bad schedule, or a Robot that doesn't score a lot but can assist well), can reveal dark horses and lame ducks.

Took the summer off from the team, presented today with our new captains for Freshman Orientation.
Mentor meeting tonight.
2017 starts on 9/12.
The juices are starting to flow again.

137 days to Kickoff.

[Ether]

Here's a thought-provoking paper about sig figs.

[MooreteP]

Originally Posted by Ether

Here's a thought-provoking paper about sig figs.

This is great. It sums up what I tell my students. Only use 3 digits (with scientific Notation where appropriate)
Anymore, and they have to justify the precision.

[SpaceBiz]

Originally Posted by BMSOTM

Relevant thread from last year.



I would be cautious about using this as a means of assessing error. It shouldn't be assumed that negative OPR indicates error. There are valid reasons for having a negative OPR...

I definitely agree there are valid reasons for having OPRs below 0. I don't think most teams with OPRs below 0 are actually contributing negatively to their alliance. When I said plus or minus some of the OPRs below zero, I meant one of but not the lowest OPR below zero in a larger data set. When I said I was making some not so great assumptions, this was one of them. My bad for not clarifying.

[Caleb Sykes]

Originally Posted by Whatever

Stronghold was my first year as a mentor in FRC. I have been interested in the OPR statistic. One thing I have noticed is it commonly gets quoted to 2 decimal places which seems like way too much accuracy.

I quote OPR to 2 decimal places in my scouting database, and I agree that this is generally far too high of precision. The reason I do this is for consistency. Depending on the category, higher precision may be required to present any useful information. For example, the "teleop Tower Captured" metric exclusively had values between -1 and +1. Without any decimal precision, this metric would be nearly worthless. With 1 decimal point precision, there is still in my opinion useful information being left out. So I decided to use 2 decimal point precision. Since I wanted consistency over all metrics, I just used 2 decimal point precision over all metrics, including raw OPR.

The other consideration we should have is that the "correct" level of precision for OPR varies with the game. In a year like this, where scores were frequently in the 100s, no decimal point precision is needed. However, in 2010, where scores were frequently less than 10, the first decimal point value was essential, and 2 decimal point precision was also arguably useful.

I would tend to agree though that any precision beyond the decimal point for raw OPR from the past 5 years is not meaningful.

[Ether]

R and Octave stats for 2014 MISJO quals OPR

Residuals Quartiles: Min 1Q Median 3Q Max
Coefficients: Std. Error, t value, Pr(>|t|), Signif. codes
Residual standard error, degrees of freedom,
Multiple R-squared, Adjusted R-squared, F-statistic, p-value

R2, R2adj, mean_abs_dev, median_abs_dev, Ybar,
std(Y), std(residuals), OPRse, SStot, SSreg, SSres, RSSres, RASres,
residual_variance, residual_standard_error,
std (standard_error_of_coefficients), Quantiles
[Y, Y - Ye, Y - Ybar, Ye - Ybar],
[coefficients, standard_error_of_coefficients, t_values_of_coefficients, p_tail]