An improvement to OPR

[Citrus Dad]

Note: I was reviewing the 2834 database and think I found that the Championship OPRs are in error. The sums of the individual components often do not add up to the Total. (3824's in Curie is off by 32.) A quick scan of the regionals finds in some cases no deviations whatsoever and <2 pts maximum in others. I suggest going back and recomputing the OPRs.

[efoote868]

A possible explanation is that Ed took into account surrogate matches.

[Ed Law]

That is exactly the problem. OPR is calculated using all matches including surrogate matches. I would still want to calculate OPR this way. More data point is better even if the match does not count for that team.

Unfortunately team standing from FIRST website only adds up the total of the auto, teleop and climb points of non surrogate matches. This means when I solve A x = b, the matrix A contains the surrogate match while vector b does not contain surrogate match.

My proposal is to scale the value of b for the teams that have the surrogate matches before solving A x = b. Does anybody have any other suggestion?

[Ether]

Originally Posted by Ed Law

My proposal is to scale the value of b for the teams that have the surrogate matches before solving A x = b. Does anybody have any other suggestion?

As a short-term solution that sounds like a reasonable approach to try to make the best out of the data that is available.

Going forward, perhaps someone who has Frank's ear and is interested in statistics could make an appeal to him to resolve the Twitter data issues. At the very least, store the data locally (at the event) and don't delete it until it has been archived at FIRST. Then make the data available to the community.

[Ed Law]

Originally Posted by Ether

As a short-term solution that sounds like a reasonable approach to try to make the best out of the data that is available.

Going forward, perhaps someone who has Frank's ear and is interested in statistics could make an appeal to him to resolve the Twitter data issues. At the very least, store the data locally (at the event) and don't delete it until it has been archived at FIRST. Then make the data available to the community.

Ether, I changed my mind about scaling numbers that has surrogate match in the vector b before solving Ax=b. I now propose to scale x(auto), x(tele) and x(climb) for each team proportionally so they will add up to the overall OPR.

We can test it afterwards and calculate the b and see how close it is to the missing subscore of the surrogate match.

[Citrus Dad]

Originally Posted by efoote868

A possible explanation is that Ed took into account surrogate matches.

I believe the method relies on the official score database, not on match by match reported scores. The surrogates don't show up there. He would have to use 2 different data sets to get different answers.

[Ether]

Originally Posted by Citrus Dad

I believe the method relies on the official score database, not on match by match reported scores. The surrogates don't show up there. He would have to use 2 different data sets to get different answers.

There are two different score datasets at USFIRST: "Match Results" and "Team Standings".

"Match Results" is necessary to construct the alliances matrix and obtain the total match score. It contains the surrogate matches.

"Team Standings" is necessary to obtain the Auto, TeleOp, and Climb alliance scoring. Problem is, the totals shown there do not include the scores for surrogate teams in matches where said teams played as surrogates.

Ed's proposed work-around to scale the "Team Standings" totals for teams which played as surrogates seems like a reasonable one. Do you have a different suggestion?

[MikeE]

Originally Posted by Ether

There are two different score datasets at USFIRST: "Match Results" and "Team Standings".

"Match Results" is necessary to construct the alliances matrix and obtain the total match score. It contains the surrogate matches.

"Team Standings" is necessary to obtain the Auto, TeleOp, and Climb alliance scoring. Problem is, the totals shown there do not include the scores for surrogate teams in matches where said teams played as surrogates.

Ed's proposed work-around to scale the "Team Standings" totals for teams which played as surrogates seems like a reasonable one. Do you have a different suggestion?

My preferred solution is for FIRST to move to an all district model with 12 matches per event and therefore no more surrogates

Until then...

If we have complete Twitter data for an event then we get the component scores for every match so we don't have an issue.

But to solve the surrogate problem we just need the component scores from the specific surrogate matches. There are at most 3 of these in any competition and typically just 1 or 2 consecutive matches in round 3.
Since there is a single surrogate team in an alliance we just need to add the Twitter component scores to their "Team Standing" score to get the corrected total scores for that surrogate team.

[Ed Law]

Originally Posted by Citrus Dad

I suggest going back and recomputing the OPRs.

Thank you for pointing out the issue with sum of individual categoty OPR do not add up to total OPR. I don't know exactly what you mean. You made it sound like I do the calculations by hand. I can ask the computer to run it 100 times and I can guarantee you that I will get the same answer every time.

[Citrus Dad]

Originally Posted by Ed Law

Thank you for pointing out the issue with sum of individual categoty OPR do not add up to total OPR. I don't know exactly what you mean. You made it sound like I do the calculations by hand. I can ask the computer to run it 100 times and I can guarantee you that I will get the same answer every time.

I see your explanation about inclusion of the surrogate match scores. I think one check would be to see if the deviations of the total OPR vs the sum of the individual components is larger with the inclusion of more surrogate matches. You may be able to derive a correction factor based on the number of surrogates. (I estimated the average foul scores with a correction factor against our scouting data.)

[IKE]

Originally Posted by Citrus Dad

I agree that the OPR is better than nothing--it still had a 0.91 correlation with our offensive scouting data. However, it tends to miss the outlier teams that may be hard to pick out otherwise. Our OPR was 27 points less than our actual offensive average--more than 50% off. But if your team has sufficient resources to calculate the OPR on the fly then you probably have enough to do full scouting. On the other hand, you might be relying on the OPR calculated with one of the apps tracking the competition, in which case then that's the best you have.

.....

Outliers happen. They tend to be worse if teams have a lot of variation, and the fewer the samples, the worse it is...

Being more than 50% off is rarer, but not unheard of. I once saw an OPR of 15 assigned to a team that had only competed in 2/8 matches. And they didn't score 60 points in those two matches...

Another killer of OPR is if a team that reliably does very well has a bad match with your team. For instance, in Archimedes, 469 ended up with an OPR over 80 points. they had a match where their shooter had an issue right from the start and I believe they only scored climb points. Unfortunately to their partners, the OPR calculations will likely penalize those other teams..

For these reasons, and many others, it is very important to scout.

OPR does accurrately though show that some teams are worth less than they score on average. Yep, you heard me right, there are many teams that are worth less than their average score. This is especially true of slow that used the middle shooting position to start their climb. While they would frequently get their 30 points, they would often cost the alliance many missed shots from cyclers that were 75%+ at taht position but 50%- from outside shooting positions. Yes, the climber did score 30 points, but the other two partners that usually put up 30 disc points, and only got 20 results in a -20 total points from them. Some of this if it occurs on a regular basis will get attributed to the climber team. This will also explain an imbalance if you summed the auton, disc points, and climbing OPR.

[Basel A]

Originally Posted by IKE

OPR does accurrately though show that some teams are worth less than they score on average. Yep, you heard me right, there are many teams that are worth less than their average score. This is especially true of slow that used the middle shooting position to start their climb. While they would frequently get their 30 points, they would often cost the alliance many missed shots from cyclers that were 75%+ at taht position but 50%- from outside shooting positions. Yes, the climber did score 30 points, but the other two partners that usually put up 30 disc points, and only got 20 results in a -20 total points from them. Some of this if it occurs on a regular basis will get attributed to the climber team. This will also explain an imbalance if you summed the auton, disc points, and climbing OPR.

This also explains the potential for negative OPR in a category. Such a team might have a negative teleop OPR.

[IKE]

Originally Posted by Basel A

This also explains the potential for negative OPR in a category. Such a team might have a negative teleop OPR.

Correct. Negative OPRs are much more rare now that penalties get added to the other teams score, but they do occur (sometimes for just reasons, sometimes not).

The phenomenon I talked about above is similar to what can occur with the +/- system in basketball. Sometimes a superstar doesn't score a lot of points due to getting double-teamed but his open teammates then score a bunch of points. If you only look at stats, it doesn't tell the whole story.

FRC 33 uses OPR to figure out schedule strength and to double check some of our Stats data. Ultimately I trust the stats more than I do OPR, but especially this year, I found a handful of errors in our scouting team data.

Like Citrus Dad, I generally found OPR to be within 15% of a teams average contribution. However there would be several teams with large deltas. Often this was due to a team not working for a while and then hitting a whole bunch of points. FCS teams would also create havoc in OPR. They would have a 80 point match, and then a 20 point match. Then and 80, then a 20... OPR math depends on a team being reasonably consistent. This behaviour will either dramatically over-predict or under predict...

[MikeE]

Originally Posted by Citrus Dad

I haven't seen the SEs posted with the OPR parameters, but I can tell you that the SEs are likely to be VERY large for so few observations--only 8 per team at the Champs, at most 12 in any of the regionals.

Small point: The Pinetree regional in Maine had 13 matches per team in qualifications; one of the reasons it was the Best Regional* of the 2013 season.

Bigger point: I've been playing around with maximum likelihood estimate models as an alternative (really an extension) to OPR, and these do provide both a mean and variance of team contribution. It's not quite ready to write up as a white paper but it's giving some interesting early results from Monte Carlo event simulations.

One more point: I'm a fan of the binary matrix approach to solving the regression described by Ryan since it's easy to add in additional match-by-match features such as average (or per team) score gradient during an event.

* from my very small sample of 4 events