View Full Version : OPR after Week Five Events
The OPR/CCWM numbers up to Week 5 events have been posted, please see
http://www.chiefdelphi.com/media/papers/2174
All events up to Week 5 are now included. I fixed a small bug for each event page in column T for "Record". The formula was wrong for rows beyond the first 37 teams.
If you find any error or have any questions, please let me know.
Weeks 1 thru 5 OPR & CCWM correlation to actual match results.
CCWM Column Headings & meaning:
E event
M Match
r1 r2 r3 b1 b2 b3 red & blue alliance teams
rs red score
bs blue score
crs sum of red alliance teams' CCWMs
cbs sum of blu alliance teams' CCWMs
rw rs>bs?
crw crs>cbs?
ccp rw==crw? (CCWM correct prediction of match outcome?)
OPR Column Headings & meaning:
E event
M Match
r1 r2 r3 b1 b2 b3 red & blue alliance teams
rs red score
bs blue score
ors OPR "expected" red alliance score
obs OPR "expected" blue alliance score
drs rs-ors
dbs bs-obs
rgo rs>ors?
bgo bs>obs?
rw rs>bs?
orw ors>obs? (red win based on OPR scores?)
ocp rw==orw? (OPR correct prediction of match outcome?)
efoote868
31-03-2013, 15:13
Once again, thank you Ed and Ether for posting this.
Summary of global OPR and CCWM match win/loss predictions: 81.72% and 82.47% respectively.
Does anyone have an idea of the number of teams yet to compete in week 6?
Once again, thank you Ed and Ether for posting this.
Summary of global OPR and CCWM match win/loss predictions: 81.72% and 82.47% respectively.
Does anyone have an idea of the number of teams yet to compete in week 6?
278 teams in Ed's database haven't competed yet.
278 teams in Ed's database haven't competed yet.
That would be a rough estimate. Some teams may have dropped out and not compete in any events.
Weeks 1 thru 5 OPR & CCWM correlation to actual match results
Oh, and here's a link to updated Twitter data for weeks 1 thru 5:
http://www.chiefdelphi.com/forums/showpost.php?p=1254753&postcount=1
As a second-year non-engineering mentor, I was wondering all off-season what my role on the team would be now the the team was able to mostly sustain itself. I discovered OPR and related statistics this year and they have kept me busy with scouting every week.
Thank you for all the work you do correlating this data each week! It has made FRC so much more interesting for me this year! I love watching the webcasts of high-OPR teams like 1114, and it doesn't hurt that OPR has made my own team look pretty darn good this year, (4th and 2nd in OPR, respectively, in their two district events, even without a banner to show for it).
So from a statistics geek from PA... thanks again!
It's bittersweet that we have the 5th highest OPR in the world and didn't qualify for Championship...
Thanks for this. It was a pleasure to meet and win with you last Saturday.
we have the 5th highest OPR in the world
What data are you looking at ?
scottandme
01-04-2013, 11:30
What data are you looking at ?
The "Max OPR" column in the OPR Results sheet. Returns the highest OPR earned by that team at any event. Can be a better number to use when teams make significant improvements to their robots midseason (adding 7 disk auton, getting 30 point hang working, tuning code, etc).
The "Max OPR" column in the OPR Results sheet. Returns the highest OPR earned by that team at any event. Can be a better number to use when teams make significant improvements to their robots midseason (adding 7 disk auton, getting 30 point hang working, tuning code, etc).
Yes, he was using the Max OPR and they have the 5th highest OPR out of all the teams at different regionals. That is a fact. However some people argue about some regionals being stronger than others. I don't know how strong Alamo is and I do not want people to get into a debate. 624 is a great team. Their world OPR ranking is 21.
The reason Ether was questioning is because when we compare teams across different regionals/district, we use World OPR and takes all interactions of teams into account. However it ranks team higher if they have been consistently good at different events than teams that did relatively not as well in their early events and improve a lot in later events. Both ways are valid in considering how good a team is.
My suggestion for 624 is to push for a district model in Texas. In the district model, you don't need to win an event to qualify. You just need to be consistently good. In fact it is possible for the 2nd round pick winning alliance team to not make it to State Championship or World Championship in the district model if they were just lucky in one event and do poorly in the other.
Here's a slightly different view of the data, FWIW.
I computed World OPR rankings based on Week5 data only, Week4 data only, Week3 data only, Weeks3&4 data only, Weeks3,4,&5 data only, and Weeks4&5 data only.
Results:
Weeks 624 Rank
5 only 3
4 only N/A
3 only 3
4&5 only 4
3,4,&5 only 6
3&4 only 9
So yeah, Team 624 is doing quite well.
DELurker
01-04-2013, 17:01
I guess the thing to note in the 624 discussion is that the OPR is a number which reflects past performance and does not predict future results. Every match is open to any alliance winning, whether through multiple miscellaneous technical fouls, superior play, random Murphyisms, or strategic miscellany. It sounds like 624, though playing strongly, has fallen victim to random chance.
OPR is a number which reflects past performance and does not predict future results.
Does not predict 100%. But its correlation with match win/loss outcomes is unarguably statistically significant (witness earlier posts in this thread), and it is one among many useful scouting tools.
Thanks Ed & Ether for this great resource.
However I do need to quibble about how OPR/CCWM is being discussed, as exemplified by several earlier posts. Picking one of these:
Summary of global OPR and CCWM match win/loss predictions: 81.72% and 82.47% respectively.
Those numbers are misleading since OPR/CCWM are calculated from the same data which are being used to test their predictive power, i.e. the training & test sets are the same.
It's analogous to (although not as extreme as) stating that final qualification ranking is a good predictor of the performance in earlier qualifying matches. Whereas obviously qualification ranking is a consequence of performance in earlier matches.
Good practice would use disjoint training and testing sets. I'm sure this analysis has been performed in previous seasons but I didn't see it from a brief search of CD.
Interestingly the simple baseline heuristic of "alliance with lower team numbers" has 59.1% predictive power for qualification matches this season. I'm assume that OPR and CCWM are better than that, but not as good as the ~82% claimed above.
Those numbers are misleading since OPR/CCWM are calculated from the same data which are being used to test their predictive power, i.e. the training & test sets are the same.
That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes. That would take a little bit more work, since Week5 may have teams which are competing for the first time, so they would have no OPR values. Those matches would have to be omitted from the analysis.
efoote868
01-04-2013, 18:35
Good practice would use disjoint training and testing sets. I'm sure this analysis has been performed in previous seasons but I didn't see it from a brief search of CD.
Interestingly the simple baseline heuristic of "alliance with lower team numbers" has 59.1% predictive power for qualification matches this season. I'm assume that OPR and CCWM are better than that, but not as good as the ~82% claimed above.
At Boilermaker Regional I calculated the OPR of teams using their Friday match results (about 8 or 9 matches), and tracked the qualification results on Saturday. OPR predictions were 20 for 24.
I'm guessing the easiest place to track predictive power would be the Championship event.
That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes. That would take a little bit more work, since Week5 may have teams which are competing for the first time, so they would have no OPR values. Those matches would have to be omitted from the analysis.
I'd be interested in substituting unknown values with world averages. I'll probably do this during Crossroads until each team has played 5 matches.
George1902
01-04-2013, 18:55
OPR and CCWM are calculated using only qualification match results, right?
So, one could test their predictive value using the elimination results of that event.
That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes.
OK, here's qual matches of Week5 events being predicted by OPR World rank based on weeks 1 thru 4 data only. 72% correct
This is a pessimistic estimate, since data gathered during the Week5 events for already-played qual matches could be used to supplement the weeks 1 thru 4 data to predict future qual matches in the Week5 events.
Based on the earlier post by efoote868, future qual matches at any given event are best predicted by OPR of already-played matches at that event, once a sufficient number of matches has been played.
Forgot to mention: columns V thru AA list the teams for each match for which no week 1 thru 4 OPR data is available.
faust1706
02-04-2013, 02:15
It's bittersweet that we have the 5th highest OPR in the world and didn't qualify for Championship...
we're in the same boat. 4th.
DELurker
02-04-2013, 08:56
Does not predict 100%. But its correlation with match win/loss outcomes is unarguably statistically significant (witness earlier posts in this thread), and it is one among many useful scouting tools.
True. Assuming that nothing changes from previous form, the OPR and CCWM can be very good indicators of probable results. However, there is an implicit error function on the numbers (otherwise, they would be 100% in the absence of meaningful changes). In addition, any team that knows they are on the short end of the statistics and does not try for a change in their favor is doing themselves a disservice.
I suppose that what I am trying to lead to is that the OPR and CCWM numbers are very useful tools (we use them a lot), but they only indicate based on the assumption that nothing will change from previous matches.
So I was just toying with this spreadsheet out of interest.
I added a field to the OPR results page for each team's state/province, and filtered it based on that field, because I wanted to support my intuitive feeling that Ontario's second tier is much stronger because of the influence of 1114 and 2056.
Calculating average OPR's for different regions gives me the following results:
Avg Max OPR FIRST-Wide: 16.3
Avg Max OPR in Ontario: 23.3
Avg Max OPR in Michigan: 22.8
Avg Max OPR in Ontario (not including 1114 and 2056): 21.1
Avg Max OPR in New Jersey: 18.8
Avg Max OPR in Ohio: 18.6
Avg Max OPR in Pennsylvania: 18.5
Avg Max OPR in Indiana: 17.4
Avg Max OPR in New York: 16.8
Avg Max OPR in Quebec: 16.7
Avg Max OPR in Massachusetts: 16.1
Avg Max OPR in New Hampshire: 14.0
Avg Max OPR in California: 14.0
Avg Max OPR in Texas: 13.8
Avg Max OPR in Minnesota: 13.0
Thanks for posting this information - I have fun with it. Also... congratulations on your team's win over the weekend.
DELurker
02-04-2013, 15:17
So I was just toying with this spreadsheet out of interest.
...
Calculating average OPR's for different regions gives me the following results:
Avg Max OPR FIRST-Wide: 16.3
Avg Max OPR in Ontario: 23.3
Avg Max OPR in Michigan: 22.8
Avg Max OPR in Ontario (not including 1114 and 2056): 21.1
Avg Max OPR in New Jersey: 18.8
...
(Edited for brevity)
What about Delaware? 27.8 Avg Max OPR. Of course, it's not as though we have enough teams to form a full alliance, much less a full match, but still...
(Edited for brevity)
What about Delaware? 27.8 Avg Max OPR. Of course, it's not as though we have enough teams to form a full alliance, much less a full match, but still...
Two data points does not a trend make :P
DELurker
02-04-2013, 15:55
Two data points does not a trend make :P
True enough... but it's good enough for a little bit of bragging... at least until Iowa, Brazil (RS), and Nevada get in on the act.
Of the states/provinces/areas that can field their own matches, Mississippi, Hawaii, and Ontario are 1-2-3.
*sigh* One day, maybe we'll grow up to be a bigger state...
DELurker
02-04-2013, 15:56
Two data points does not a trend make :P
Actually, isn't the definition of a trend the point at which you can draw a line through two points? Not only is it a trend, but it has 100% correlation. :P
Trend implies some sort of change over time, so really none of them are trends. They're all just statistical distributions.
Of the states/provinces/areas that can field their own matches, Mississippi, Hawaii, and Ontario are 1-2-3.
Field their own matches with teams that have played so far in 2013. HI has more than 6 teams, just only 6 of them play OUTSIDE of HI, since the Hawaii Regional hasn't happened yet.
And of those 3, only Ontario has had enough to hold a regional.
JohnSchneider
02-04-2013, 16:21
Texas and Minnesota both have a HUGE number of teams - most of which are very young. This could provide an explanation for the low scores.
The other interesting thing, is how low CA is ranked by this metric. Below the FIRST-wide average, and yet CA is seen as one of the stronger regions.
I think this is an artifact of the sheer number of CA teams. Yes, they have 254, 330, 973, and more, but there are just so many of the weaker teams in CA that it skews their average.
gabrielau23
02-04-2013, 16:26
Sorry, I'm new to this page and thought it very interesting. I'm not sure how to open the reports? *wince
JohnSchneider
02-04-2013, 16:28
The other interesting thing, is how low CA is ranked by this metric. Below the FIRST-wide average, and yet CA is seen as one of the stronger regions.
I think this is an artifact of the sheer number of CA teams. Yes, they have 254, 330, 973, and more, but there are just so many of the weaker teams in CA that it skews their average.
I think this pairs with the Texas/Minnesota thing. I'm interested to see if these regions will see an increase in strength with the move to districts (all. 3 regions are candidates) and the lengthened competition season and multiple regionals.
I think this pairs with the Texas/Minnesota thing. I'm interested to see if these regions will see an increase in strength with the move to districts (all. 3 regions are candidates) and the lengthened competition season and multiple regionals.
I hope we see more district models happening. I want to see a district model in Canada. An awfully large (and growing!) portion of Canadian teams have been forking out huge cash to compete in 2 or 3 regionals at $4k/pop.
Sorry, I'm new to this page and thought it very interesting. I'm not sure how to open the reports? *wince
Which reports are you referring to? The XLS attachments in this thread, or Ed Law's XLSM scouting database whitepaper?
gabrielau23
02-04-2013, 17:44
Well, considering that I don't even know the difference right now...both? XD
Well, considering that I don't even know the difference right now...both? XD
Do you have Microsoft Excel installed?
If so, what version is it?
If not, what OS are you using?
gabrielau23
02-04-2013, 18:03
Oh, I know how to download a file and use Excel. I just don't know which one to download out of the many, many files on his link.
Oh, I know how to download a file and use Excel. I just don't know which one to download out of the many, many files on his link.
You can use the filenames and dates to find the version you want...the latest at the moment, including week 5 results, is Team 2834 2013_Scouting_Database v5.0. The others are simply from past weeks and past years.
gabrielau23
02-04-2013, 18:14
OHHHHHHHHH. Wow. Now I feel a little stupid. wow. wow. Anyway, thanks Ether and Carr!
Curious me is wondering how to actually predict the outcomes of a match using OPR? I have a bit of AP Stat background, but not much in matrix math. Is there a simple, easy method (Ie totaling OPR for the Red and Blue alliance and the higher is the winner?) Or, if I've managed to miss the thread in my filtering that describes this, could someone direct me to it?
Curious me is wondering how to actually predict the outcomes of a match using OPR? I have a bit of AP Stat background, but not much in matrix math. Is there a simple, easy method (Ie totaling OPR for the Red and Blue alliance and the higher is the winner?) Or, if I've managed to miss the thread in my filtering that describes this, could someone direct me to it?Considering that OPR is supposed to just represent the average number of points one robot will score, I assume you can just total them.
That's what I figured. I just wasn't sure if it was more advanced than that.
Considering that OPR is supposed to just represent the average number of points one robot will score, I assume you can just total them.
Yeah, pretty much. I haven't seen any other approaches to estimating an alliances score. It isn't perfect, especially in scoring-limited games. 2011 is a great example. 3 teams that always scored 30 minibot points would only score 50 points, but OPR would have predicted 90. Similar case for combinations of teams that each score a high logo of tubes (they aren't going to get points for 3 high logos) or for combinations of teams that each use a lot of game pieces (this year, 3 teams that each typically score 40 discs probably aren't going to score 120 discs).
(this year, 3 teams that each typically score 40 discs probably aren't going to score 120 discs).
However, given the sheer number of discs available, the ability of teams to both human-load and floor-load, and the different climbing mechanisms, this year's estimations could be much more accurate than the last few. (Especially in relation to minibots, ubertubes, and coopertition points.)
...Which is why OPR sucked as a metric of team performance in 2011.
OPR in 2013: pretty good until you get to really high scores where multiple teams on an alliance could drain the alliance station of discs on their own.
Still waiting for the rest of the Qual Match data from Bridgewater, but in the meantime here's an interesting look at the OPR and CCWM based on Week6 events.
efoote868
06-04-2013, 23:18
Crossroads OPR from Friday matches were 16 for 25 in predicting outcomes, which is much worse than Boilermaker's 20 for 24.
Crossroads OPR from Friday matches were 16 for 25 in predicting outcomes, which is much worse than Boilermaker's 20 for 24.
Can you elaborate how you do the prediction?
efoote868
07-04-2013, 02:45
Can you elaborate how you do the prediction?
I used OPRNet's predictions after the Friday matches, and then kept track today.
Crossroads OPR from Friday matches were 16 for 25 in predicting outcomes, which is much worse than Boilermaker's 20 for 24.
I count only 8 matches not correctly predicted:
64
65
70
72
73
76
78
84
... what's the 9th one?
While we're waiting for Ed to update his superb scouting spreadsheet...
OPR & CCWM World Rankings based on Weeks 1 thru 6 Qual Match data
efoote868
07-04-2013, 12:32
I count only 8 matches not correctly predicted:
64
65
70
72
73
76
78
84
... what's the 9th one?
71 was a tie.
71 was a tie.
OK, let's call it 16 out of 24 then :-)
efoote868
07-04-2013, 12:43
OK, let's call it 16 out of 24 then :-)
When I did counts using OPR data and match outcome "predictions", I always counted ties as wrong.
Unless there's a confidence interval, I'm not exactly sure how to treat a tie statistically. And labeling a match "too close to call" isn't any fun. :p
JohnSchneider
07-04-2013, 12:47
When I did counts using OPR data and match outcome "predictions", I always counted ties as wrong.
Unless there's a confidence interval, I'm not exactly sure how to treat a tie statistically. And labeling a match "too close to call" isn't any fun. :p
Ties are outliers in binomial situations. Because you cant have 3 options for two choices. Which is why ether just excused it and lowered the sample size.
efoote868
07-04-2013, 12:59
Ties are outliers in binomial situations. Because you cant have 3 options for two choices. Which is why ether just excused it and lowered the sample size.
Right, but hypothetically if OPR predictions said the match would be 100-50, and there was a tie 50-50, the OPR prediction is wrong and shouldn't be excused as a tie.
Right, but hypothetically if OPR predictions said the match would be 100-50, and there was a tie 50-50, the OPR prediction is wrong and shouldn't be excused as a tie.
hypothetically if OPR prediction said the match would be 50.001-49.999, and there was a tie 50-50, should the OPR prediction be considered wrong and not excused as a tie? :-)
Maybe we should start publishing the residual vector (or the covariance matrix?) along with the OPR :-)
hypothetically if OPR prediction said the match would be 50.001-49.999, and there was a tie 50-50, should the OPR prediction be considered wrong and not excused as a tie? :-)
Maybe we should start publishing the residual vector (or the covariance matrix?) along with the OPR :-)
One metric I've used to avoid this problem is the mean and standard deviation of the distribution of alliance score residuals. I also considered using winning margin residuals, but decided against.
efoote868
07-04-2013, 14:02
I'm sure that integer rounding can be excused. :D
FWIW, I calculated OPR using all qual data for weeks 1 thru 5 PLUS week6 Friday, and used that to predict Saturday Qual matches at Crossroads.
It got Matches 65 and 84 right, but got Match 80 wrong.
64
70
72
73
76
78
80
efoote868
07-04-2013, 14:48
I'm wondering if OPR predictions at the Championship event will be similar; Crossroads had a fairly deep field with an average OPR of about 27.6.
Max Event OPR achieved by each of the 2,490 teams
OPR & CCWM from Weeks 1-6 Qual data:
_2490 teams
_6091 matches
12182 alliances
14.69 average OPR
12.20 median OPR
-2.11 average CCWM
-4.11 median CCWM
Histograms:
Wait a minute, Ether. How can average CCWM not be zero? For each event, the sum of CCWM should be zero. In most cases, they are. But there are some events that has a small positive or negative number. I think it must be due to round off error. Is that correct? Then how can the average CCWM be -2.11?
Joe Ross
07-04-2013, 22:50
What is the match with an OPR residual of 280+?
Wait a minute, Ether. How can average CCWM not be zero?
Take the average of cells Cells R4 thru R2493 on sheet "World Rank" in your spreadsheet.
What is the match with an OPR residual of 280+?
Back to the drawing board. Thanks for keeping me honest.
I found an equals sign where there should have been a minus sign in one of my AWK scripts.
Here's a corrected version.
Wait a minute, Ether. How can average CCWM not be zero? For each event, the sum of CCWM should be zero. In most cases, they are. But there are some events that has a small positive or negative number. I think it must be due to round off error. Is that correct? Then how can the average CCWM be -2.11?
It's not due to rounding error. It's due to the match schedule.
If you look carefully, you'll see that the average CCWM is zero only for those events where the schedule had each team playing the same number of times.
When you do World CCWM, this effect is exaggerated.
The CCWM residuals, however, will always sum to zero, regardless of schedule.
Programmer3941
09-04-2013, 23:40
Although this data sheet is now slightly old news...what is this black magic that puts my team's OPR above 341, 365, 233, and 379? :ahh: Last time I checked, 3941 was ranked 262...quite frankly, I am astonished by this considering our current regional ranking. What factors go into OPR that could produce such a result?
What factors go into OPR that could produce such a result?
The computation of OPR is explained here:
http://www.chiefdelphi.com/forums/showpost.php?p=1119150&postcount=36
and here:
http://www.chiefdelphi.com/forums/showpost.php?p=1253180&postcount=11
and here:
http://www.chiefdelphi.com/forums/showpost.php?p=1252686&postcount=9
and here:
http://www.chiefdelphi.com/media/papers/download/3321
Programmer3941
10-04-2013, 00:40
Aha. Makes better sense now. Thanks Ether.
What factors go into OPR that could produce such a result?
Luck.
The randomness of scheduling can boost OPR if you are partnered with relatively low scoring teams in a high scoring match.
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.