OPR after Week Five Events
 

The OPR/CCWM numbers up to Week 5 events have been posted, please see

http://www.chiefdelphi.com/media/papers/2174

All events up to Week 5 are now included. I fixed a small bug for each event page in column T for "Record". The formula was wrong for rows beyond the first 37 teams.

If you find any error or have any questions, please let me know.

Re: OPR after Week Five Events
 

Weeks 1 thru 5 OPR & CCWM correlation to actual match results.


CCWM Column Headings & meaning:

OPR Column Headings & meaning:

Re: OPR after Week Five Events
 

Once again, thank you Ed and Ether for posting this.

Summary of global OPR and CCWM match win/loss predictions: 81.72% and 82.47% respectively.

Does anyone have an idea of the number of teams yet to compete in week 6?

Re: OPR after Week Five Events
 

278 teams in Ed's database haven't competed yet.

Re: OPR after Week Five Events
 

That would be a rough estimate. Some teams may have dropped out and not compete in any events.

Re: OPR after Week Five Events
 

Oh, and here's a link to updated Twitter data for weeks 1 thru 5:

http://www.chiefdelphi.com/forums/sh...53&postcount=1

Re: OPR after Week Five Events
 

As a second-year non-engineering mentor, I was wondering all off-season what my role on the team would be now the the team was able to mostly sustain itself. I discovered OPR and related statistics this year and they have kept me busy with scouting every week.

Thank you for all the work you do correlating this data each week! It has made FRC so much more interesting for me this year! I love watching the webcasts of high-OPR teams like 1114, and it doesn't hurt that OPR has made my own team look pretty darn good this year, (4th and 2nd in OPR, respectively, in their two district events, even without a banner to show for it).

So from a statistics geek from PA... thanks again!

Re: OPR after Week Five Events
 

It's bittersweet that we have the 5th highest OPR in the world and didn't qualify for Championship...

Re: OPR after Week Five Events
 

Thanks for this. It was a pleasure to meet and win with you last Saturday.

Re: OPR after Week Five Events
 

What data are you looking at ?

Re: OPR after Week Five Events
 

The "Max OPR" column in the OPR Results sheet. Returns the highest OPR earned by that team at any event. Can be a better number to use when teams make significant improvements to their robots midseason (adding 7 disk auton, getting 30 point hang working, tuning code, etc).

Re: OPR after Week Five Events
 

Yes, he was using the Max OPR and they have the 5th highest OPR out of all the teams at different regionals. That is a fact. However some people argue about some regionals being stronger than others. I don't know how strong Alamo is and I do not want people to get into a debate. 624 is a great team. Their world OPR ranking is 21.

The reason Ether was questioning is because when we compare teams across different regionals/district, we use World OPR and takes all interactions of teams into account. However it ranks team higher if they have been consistently good at different events than teams that did relatively not as well in their early events and improve a lot in later events. Both ways are valid in considering how good a team is.

My suggestion for 624 is to push for a district model in Texas. In the district model, you don't need to win an event to qualify. You just need to be consistently good. In fact it is possible for the 2nd round pick winning alliance team to not make it to State Championship or World Championship in the district model if they were just lucky in one event and do poorly in the other.

Re: OPR after Week Five Events
 

Here's a slightly different view of the data, FWIW.

I computed World OPR rankings based on Week5 data only, Week4 data only, Week3 data only, Weeks3&4 data only, Weeks3,4,&5 data only, and Weeks4&5 data only.

Results:

So yeah, Team 624 is doing quite well.

Re: OPR after Week Five Events
 

I guess the thing to note in the 624 discussion is that the OPR is a number which reflects past performance and does not predict future results. Every match is open to any alliance winning, whether through multiple miscellaneous technical fouls, superior play, random Murphyisms, or strategic miscellany. It sounds like 624, though playing strongly, has fallen victim to random chance.

Re: OPR after Week Five Events
 

Does not predict 100%. But its correlation with match win/loss outcomes is unarguably statistically significant (witness earlier posts in this thread), and it is one among many useful scouting tools.

Re: OPR after Week Five Events
 

Thanks Ed & Ether for this great resource.

However I do need to quibble about how OPR/CCWM is being discussed, as exemplified by several earlier posts. Picking one of these:

Those numbers are misleading since OPR/CCWM are calculated from the same data which are being used to test their predictive power, i.e. the training & test sets are the same.

It's analogous to (although not as extreme as) stating that final qualification ranking is a good predictor of the performance in earlier qualifying matches. Whereas obviously qualification ranking is a consequence of performance in earlier matches.

Good practice would use disjoint training and testing sets. I'm sure this analysis has been performed in previous seasons but I didn't see it from a brief search of CD.

Interestingly the simple baseline heuristic of "alliance with lower team numbers" has 59.1% predictive power for qualification matches this season. I'm assume that OPR and CCWM are better than that, but not as good as the ~82% claimed above.

Re: OPR after Week Five Events
 

That's a valid criticism, which could be addressed by using Weeks 1 thru 4 OPR numbers to predict Week5 outcomes. That would take a little bit more work, since Week5 may have teams which are competing for the first time, so they would have no OPR values. Those matches would have to be omitted from the analysis.

Re: OPR after Week Five Events
 

At Boilermaker Regional I calculated the OPR of teams using their Friday match results (about 8 or 9 matches), and tracked the qualification results on Saturday. OPR predictions were 20 for 24.

I'm guessing the easiest place to track predictive power would be the Championship event.

I'd be interested in substituting unknown values with world averages. I'll probably do this during Crossroads until each team has played 5 matches.

Re: OPR after Week Five Events
 

OPR and CCWM are calculated using only qualification match results, right?

So, one could test their predictive value using the elimination results of that event.

Re: OPR after Week Five Events
 

OK, here's qual matches of Week5 events being predicted by OPR World rank based on weeks 1 thru 4 data only. 72% correct

This is a pessimistic estimate, since data gathered during the Week5 events for already-played qual matches could be used to supplement the weeks 1 thru 4 data to predict future qual matches in the Week5 events.

Based on the earlier post by efoote868, future qual matches at any given event are best predicted by OPR of already-played matches at that event, once a sufficient number of matches has been played.

Forgot to mention: columns V thru AA list the teams for each match for which no week 1 thru 4 OPR data is available.

Re: OPR after Week Five Events
 

we're in the same boat. 4th.

Re: OPR after Week Five Events
 

True. Assuming that nothing changes from previous form, the OPR and CCWM can be very good indicators of probable results. However, there is an implicit error function on the numbers (otherwise, they would be 100% in the absence of meaningful changes). In addition, any team that knows they are on the short end of the statistics and does not try for a change in their favor is doing themselves a disservice.

I suppose that what I am trying to lead to is that the OPR and CCWM numbers are very useful tools (we use them a lot), but they only indicate based on the assumption that nothing will change from previous matches.

Re: OPR after Week Five Events
 

So I was just toying with this spreadsheet out of interest.

I added a field to the OPR results page for each team's state/province, and filtered it based on that field, because I wanted to support my intuitive feeling that Ontario's second tier is much stronger because of the influence of 1114 and 2056.

Calculating average OPR's for different regions gives me the following results:

Avg Max OPR FIRST-Wide: 16.3

Avg Max OPR in Ontario: 23.3
Avg Max OPR in Michigan: 22.8
Avg Max OPR in Ontario (not including 1114 and 2056): 21.1
Avg Max OPR in New Jersey: 18.8
Avg Max OPR in Ohio: 18.6
Avg Max OPR in Pennsylvania: 18.5
Avg Max OPR in Indiana: 17.4
Avg Max OPR in New York: 16.8
Avg Max OPR in Quebec: 16.7
Avg Max OPR in Massachusetts: 16.1
Avg Max OPR in New Hampshire: 14.0
Avg Max OPR in California: 14.0
Avg Max OPR in Texas: 13.8
Avg Max OPR in Minnesota: 13.0

Re: OPR after Week Five Events
 

Thanks for posting this information - I have fun with it. Also... congratulations on your team's win over the weekend.

Re: OPR after Week Five Events
 

(Edited for brevity)

What about Delaware? 27.8 Avg Max OPR. Of course, it's not as though we have enough teams to form a full alliance, much less a full match, but still...

Re: OPR after Week Five Events
 

Two data points does not a trend make :P

Re: OPR after Week Five Events
 

True enough... but it's good enough for a little bit of bragging... at least until Iowa, Brazil (RS), and Nevada get in on the act.

Of the states/provinces/areas that can field their own matches, Mississippi, Hawaii, and Ontario are 1-2-3.

*sigh* One day, maybe we'll grow up to be a bigger state...

Re: OPR after Week Five Events
 

Actually, isn't the definition of a trend the point at which you can draw a line through two points? Not only is it a trend, but it has 100% correlation. :P

Re: OPR after Week Five Events
 

Trend implies some sort of change over time, so really none of them are trends. They're all just statistical distributions.

Re: OPR after Week Five Events
 

Field their own matches with teams that have played so far in 2013. HI has more than 6 teams, just only 6 of them play OUTSIDE of HI, since the Hawaii Regional hasn't happened yet.

And of those 3, only Ontario has had enough to hold a regional.

Re: OPR after Week Five Events
 

Texas and Minnesota both have a HUGE number of teams - most of which are very young. This could provide an explanation for the low scores.

Re: OPR after Week Five Events
 

The other interesting thing, is how low CA is ranked by this metric. Below the FIRST-wide average, and yet CA is seen as one of the stronger regions.

I think this is an artifact of the sheer number of CA teams. Yes, they have 254, 330, 973, and more, but there are just so many of the weaker teams in CA that it skews their average.

Re: OPR after Week Five Events
 

Sorry, I'm new to this page and thought it very interesting. I'm not sure how to open the reports? *wince

Re: OPR after Week Five Events
 

I think this pairs with the Texas/Minnesota thing. I'm interested to see if these regions will see an increase in strength with the move to districts (all. 3 regions are candidates) and the lengthened competition season and multiple regionals.

Re: OPR after Week Five Events
 

I hope we see more district models happening. I want to see a district model in Canada. An awfully large (and growing!) portion of Canadian teams have been forking out huge cash to compete in 2 or 3 regionals at $4k/pop.

Re: OPR after Week Five Events
 

Which reports are you referring to? The XLS attachments in this thread, or Ed Law's XLSM scouting database whitepaper?

Re: OPR after Week Five Events
 

Well, considering that I don't even know the difference right now...both? XD

Re: OPR after Week Five Events
 

Do you have Microsoft Excel installed?

If so, what version is it?

If not, what OS are you using?

Re: OPR after Week Five Events
 

Oh, I know how to download a file and use Excel. I just don't know which one to download out of the many, many files on his link.

Re: OPR after Week Five Events
 

You can use the filenames and dates to find the version you want...the latest at the moment, including week 5 results, is Team 2834 2013_Scouting_Database v5.0. The others are simply from past weeks and past years.

Re: OPR after Week Five Events
 

OHHHHHHHHH. Wow. Now I feel a little stupid. wow. wow. Anyway, thanks Ether and Carr!

Re: OPR after Week Five Events
 

Curious me is wondering how to actually predict the outcomes of a match using OPR? I have a bit of AP Stat background, but not much in matrix math. Is there a simple, easy method (Ie totaling OPR for the Red and Blue alliance and the higher is the winner?) Or, if I've managed to miss the thread in my filtering that describes this, could someone direct me to it?

Re: OPR after Week Five Events
 

Considering that OPR is supposed to just represent the average number of points one robot will score, I assume you can just total them.

Re: OPR after Week Five Events
 

That's what I figured. I just wasn't sure if it was more advanced than that.

Re: OPR after Week Five Events
 

Yeah, pretty much. I haven't seen any other approaches to estimating an alliances score. It isn't perfect, especially in scoring-limited games. 2011 is a great example. 3 teams that always scored 30 minibot points would only score 50 points, but OPR would have predicted 90. Similar case for combinations of teams that each score a high logo of tubes (they aren't going to get points for 3 high logos) or for combinations of teams that each use a lot of game pieces (this year, 3 teams that each typically score 40 discs probably aren't going to score 120 discs).

Re: OPR after Week Five Events
 

However, given the sheer number of discs available, the ability of teams to both human-load and floor-load, and the different climbing mechanisms, this year's estimations could be much more accurate than the last few. (Especially in relation to minibots, ubertubes, and coopertition points.)

Re: OPR after Week Five Events
 

...Which is why OPR sucked as a metric of team performance in 2011.

OPR in 2013: pretty good until you get to really high scores where multiple teams on an alliance could drain the alliance station of discs on their own.

Re: OPR after Week Five Events
 

Still waiting for the rest of the Qual Match data from Bridgewater, but in the meantime here's an interesting look at the OPR and CCWM based on Week6 events.

Re: OPR after Week Five Events
 

Crossroads OPR from Friday matches were 16 for 25 in predicting outcomes, which is much worse than Boilermaker's 20 for 24.

Re: OPR after Week Five Events
 

Can you elaborate how you do the prediction?

Re: OPR after Week Five Events
 

I used OPRNet's predictions after the Friday matches, and then kept track today.

Re: OPR after Week Five Events
 

I count only 8 matches not correctly predicted:

64
65
70
72
73
76
78
84

... what's the 9^th one?

Re: OPR after Week Five Events
 

While we're waiting for Ed to update his superb scouting spreadsheet...

OPR & CCWM World Rankings based on Weeks 1 thru 6 Qual Match data

Re: OPR after Week Five Events
 

71 was a tie.

Re: OPR after Week Five Events
 

OK, let's call it 16 out of 24 then :-)

Re: OPR after Week Five Events
 

When I did counts using OPR data and match outcome "predictions", I always counted ties as wrong.

Unless there's a confidence interval, I'm not exactly sure how to treat a tie statistically. And labeling a match "too close to call" isn't any fun. :p

Re: OPR after Week Five Events
 

Ties are outliers in binomial situations. Because you cant have 3 options for two choices. Which is why ether just excused it and lowered the sample size.

Re: OPR after Week Five Events
 

Right, but hypothetically if OPR predictions said the match would be 100-50, and there was a tie 50-50, the OPR prediction is wrong and shouldn't be excused as a tie.

Re: OPR after Week Five Events
 

hypothetically if OPR prediction said the match would be 50.001-49.999, and there was a tie 50-50, should the OPR prediction be considered wrong and not excused as a tie? :-)

Maybe we should start publishing the residual vector (or the covariance matrix?) along with the OPR :-)

Re: OPR after Week Five Events
 

One metric I've used to avoid this problem is the mean and standard deviation of the distribution of alliance score residuals. I also considered using winning margin residuals, but decided against.

Re: OPR after Week Five Events
 

I'm sure that integer rounding can be excused. :D

Re: OPR after Week Five Events
 

FWIW, I calculated OPR using all qual data for weeks 1 thru 5 PLUS week6 Friday, and used that to predict Saturday Qual matches at Crossroads.

It got Matches 65 and 84 right, but got Match 80 wrong.

64
70
72
73
76
78
80

Re: OPR after Week Five Events
 

I'm wondering if OPR predictions at the Championship event will be similar; Crossroads had a fairly deep field with an average OPR of about 27.6.

Re: OPR after Week Five Events
 

Re: OPR after Week Five Events
 

Wait a minute, Ether. How can average CCWM not be zero? For each event, the sum of CCWM should be zero. In most cases, they are. But there are some events that has a small positive or negative number. I think it must be due to round off error. Is that correct? Then how can the average CCWM be -2.11?

Re: OPR after Week Five Events
 

What is the match with an OPR residual of 280+?

Re: OPR after Week Five Events
 

Take the average of cells Cells R4 thru R2493 on sheet "World Rank" in your spreadsheet.


Back to the drawing board. Thanks for keeping me honest.

I found an equals sign where there should have been a minus sign in one of my AWK scripts.

Here's a corrected version.

Re: OPR after Week Five Events
 

It's not due to rounding error. It's due to the match schedule.

If you look carefully, you'll see that the average CCWM is zero only for those events where the schedule had each team playing the same number of times.

When you do World CCWM, this effect is exaggerated.

The CCWM residuals, however, will always sum to zero, regardless of schedule.

Re: OPR after Week Five Events
 

Although this data sheet is now slightly old news...what is this black magic that puts my team's OPR above 341, 365, 233, and 379? :ahh: Last time I checked, 3941 was ranked 262...quite frankly, I am astonished by this considering our current regional ranking. What factors go into OPR that could produce such a result?

Re: OPR after Week Five Events
 

The computation of OPR is explained here:

http://www.chiefdelphi.com/forums/sh...0&postcount=36

and here:

http://www.chiefdelphi.com/forums/sh...0&postcount=11

and here:

http://www.chiefdelphi.com/forums/sh...86&postcount=9

and here:

http://www.chiefdelphi.com/media/papers/download/3321

Re: OPR after Week Five Events
 

Aha. Makes better sense now. Thanks Ether.

Re: OPR after Week Five Events
 

Luck.

The randomness of scheduling can boost OPR if you are partnered with relatively low scoring teams in a high scoring match.