OPR after Week Four Events
 

The OPR/CCWM numbers up to Week 4 events have been posted, please see

http://www.chiefdelphi.com/media/papers/2174

All events up to Week 4 are now included. I also added the score of match 72 of New York and recalculated the OPR/CCWM.

If you find any error or have any questions, please let me know.

Re: OPR after Week Four Events
 

2577 was DQ'd in match 7, and 3929 was DQ'd in match 26 (Unfortunately neither the frcfms twitter nor the FIRST event pages log in which match DQ's happen, so this data is directly from the scorekeeper)

Re: OPR after Week Four Events
 

Sorry, while I was rushing to publish the data, I forgot to sort the World OPR ranking. The rank number in column P is correct. It is just that I usually sort all the teams by OPR rank before I publish. So ignore column A.

I am not going to re-publish the data. You can sort it to how you want it.

Re: OPR after Week Four Events
 

Ed,

Do you have a more up-to-date source for the photos? The link through the blue alliance page that is listed in the instructions does not appear to be functional for this year.

Re: OPR after Week Four Events
 

Ed,

There appears to be a minor discrepancy between the OPR Results and the CCWM Results page regarding the # of Events column. It looks like the CCWM page shows the number of events completed while the OPR page shows the number the team registered for. Is this an artifact of the calculations?

Re: OPR after Week Four Events
 

Toronto West has no teams listed. Perhaps that is why 2056 and 1114 only show 2 events, but are scheduled for a 3rd.

987 has 2 events for OPR, but only 1 for WM. They dropped out of Utah, but are still scheduled to attend Las Vegas later.

Re: OPR after Week Four Events
 

Yes I found an error in the formula for counting number of events. The one on OPR page is correct. The one on CCWM page only counts up to Column CC (PineTree) which is why events from Las Vegas and after do not show up. Last year there were less events that only went up to Column CC. I don't know why I would update the formula on one page and forgot to update it on another. It will be fixed next week.

Also I accidentally ran the macro on Toronto West because the color of the tab (green) for Week 5 is too similar to the color (blue) for Week 4. That is why the list of teams is gone. I manually copied the list of teams from last week's version onto the page CCWM Results and OPR Results but forgot to copy the # of Events column also. Hence all Toronto West teams will show one less in the # of Events column. It does not affect anything else. This will be corrected next week after the event is over. If you cannot wait, you can copy that column from Week 3 (Version 3.0) and paste into Week 4 (Version 4.0). Sorry for the confusion.

Re: OPR after Week Four Events
 

I find it interesting that a good auto (18 pts) and a 50 point climb and dump every match would get a team into the top 10 highest OPRs.

Re: OPR after Week Four Events
 

Sorry, I forgot to include an example for you to reference. By my count, there are 274 teams with a difference between the two pages, so it's more than just the Toronto West teams. I've attached a PDF of the list. Looking back at the week 3 data, the discrepancy is present there, as well.

Of course, the flip side is that the discrepancy affects nothing that I can see...

Re: OPR after Week Four Events
 

Hi Ed,

Forgive me if this has already been asked.

For an upcoming event the spreadsheet is of course mostly empty. Is there a way we could find the current OPR rankings for the teams coming to that event? I'd like to cut and paste that ranking order as a list into something I am working on.

Re: OPR after Week Four Events
 

FWIW:

I was playing around with OPR and put together this spreadsheet.

For each of the 3,833 Qual matches played so far, it shows the actual match score and the "expected" score based on the OPR of the teams in each alliance.

Re: OPR after Week Four Events
 

Is this done with the OPR at the time of the match. Because on more than one occasion our expected score is actually lower than our OPR alone?

What was the process used to do this?

Re: OPR after Week Four Events
 

How soon could this be generated after the qualifying matches are posted?

Re: OPR after Week Four Events
 

Can't find an easy way to say it, but:
Predicted OPR match results match actual match results 82% of the time (3142 out of 3833).

Re: OPR after Week Four Events
 

Is it more likely to be over the expected result or under the expected result.

Re: OPR after Week Four Events
 

My question is badly stated. How soon can projected results be generated once the qualifying schedule is published. How easy is this to do? I can do it myself if possible.

Re: OPR after Week Four Events
 

Averages out to zero, but that's to be expected.

Re: OPR after Week Four Events
 

I'd say that the OPR match results are more often than not higher than the actual scores.

Re: OPR after Week Four Events
 

Interesting observation, and I got the same answer you did.

In 82% of the matches, the outcome of the match (win or lose) was the same according to the actual score and the OPR "expected" score.

Re: OPR after Week Four Events
 

Id love to see some confidence intervals on this. With a .95 CI it might be neat to try and apply it to matches this weekend before they happen and see to what accuracy an event could be predicted.

Re: OPR after Week Four Events
 

3,713 (48.4%) of the actual alliance scores were greater than the OPR "expected" alliance score. 3,953 (51.6%) were less.

But the sum of all the actual alliance scores is equal to the sum of all the OPR "expected" alliance scores.

Re: OPR after Week Four Events
 

Here's a spreadsheet in a format which may be somewhat easier to use for various statistical analysis endeavors.

Column Headings & meaning:

Re: OPR after Week Four Events
 

Here's a spreadsheet using CCWM instead of OPR.

Column Headings & meaning:

Re: OPR after Week Four Events
 

Correctly predicting 83.3% of the outcomes. (I'm pretty sure.)

Re: OPR after Week Four Events
 

Yes, I got the same number.

Re: OPR after Week Four Events
 

Please read the instructions tab. The answer is there. The information you are looking for is available in one of the tabs. Filters are set up for you so you can select only the teams that is going to a particular regional/district.

Re: OPR after Week Four Events
 

Starting from row 154 are the qualification matches. Column L and M is the prediction of each match. However it cannot predict match results until there are data. It needs a few matches before it will even be able to solve the equations. To get accurate and meaningful numbers, I think you need at least 4 to 5 matches.

The over 80% accuracy is after all the qualifying matches are available. A least square fit is used which is why the error is minimized. The number is even more impressive if you consider the close matches that can swing either way. During the qualifying matches, even after 4 to 5 matches, the prediction is not as accurate. But it is the best method there is and that is what most people use. I have not done any studies but the accuracy should be over 66% this year and the later matches will get more and more accurate.

Ether, this is another interesting challenge for you.

However there is another way to do this. It is by using historical OPR or projected OPR. But it only works with all 6 teams in that match has played in another event.

Re: OPR after Week Four Events
 

What I would be interested in is a comparison between OPR for an event, and actual average points scored for each robot. This, of course, means that a very accurate log of all robots for all matches is needed. Anyone have that data? (Before we get flamed for poor scouting, we don't keep track of ALL robots in ALL matches. The majority, yes - but not all.)

We make videos of all of our matches so we can go back and look for improvements in the robot and the driving. Using the videos, I've kept track of all of the points our robot scored during qualifications at both of our districts so far. OPR has been within 10% of our actual average at both events.

Re: OPR after Week Four Events
 

We are starting to do that analysis for Grand Blanc. I have done it on a piece-meal basis for many other teams and the +/-10 is a pretty good rule of thumb this year, but there have been a handful of outliers (as to be expected). At Waterford, on firday night scouting, a team had an OPR between 15 and 20 pts (after 8 matches), but they were not even on the field for about 6 of those matches.

And of course our "actuals" are only as accurrate as the scouts. In general, they do a really good job, but I do frequently find errors (one scout recorded 33 putting up 8 discs in auton:yikes: )

Re: OPR after Week Four Events
 

I know some people have already seen this, but this forum has responded exactly to what my app does. Discussion on the topic can be found here:
NEW: OPR FIRST 2013 Android App
OPR and CCMW applications

Either way, I use the OPR's to calculate a predicted score in real time. Like stated earlier, it is very accurate in terms of who wins/loses. As of scoring, the percentage varies. It would be cool to use past OPRs to predict the results. My app uses data specific to that regional and only that regional. So the longer the regional goes on, the more accurate the data is.

In addition, it has information about the average alliance score and OPR right next to each other to compare. I know, its not the same as a robots average points, but it would be hard to get that information for every team in a regional that hasnt been played yet because no one can keep track of individual scouting data for each team at each regional. Our OPR was 42.7 ish (I dont remember the actual number) but according to our scouting data, our average scoring points were around 43-44 points.

Re: OPR after Week Four Events
 

Hi stingray27,

I saw your thread last night and have already installed and used it. Brilliant!

You say you use OPR and not CCMW. Being new to this, what is CCMW and is it not preferable to OPR? Just asking, because I am fine with the OPR results.

Re: OPR after Week Four Events
 

CCWM is Calculated Contribution to the Winning Margin. For a given team, instead of using purely the team's score (OPR), CCWM uses the team's alliance score minus the opponent's score.

Re: OPR after Week Four Events
 

Hi Ed,

Thanks. I am getting the hang of it now. After filtering it gave me exactly the list that I wanted.

Now that I have done the filtering, I know its the Excel way to check all the boxes. I decided to make the list for our entire region of >100 teams. I would have to do this each week, but I am thinking I could have written a macro to do it. I think it may be done even if I have to generate the criteria statement, but if I remember right the criteria can refer to a range of cells but it has been a while. I am using Excel 2007 and so I don't think I can refer to a range anywhere but on the OPR results worksheet itself.

Re: OPR after Week Four Events
 

It's CCWM. "Calculated Contribution to Winning Margin"

It's just like OPR, except instead of using alliance score as input to the computation, it uses the difference between the two alliance scores for each match.

See Ed Law's paper (linked in post#1 in this thread). There's a discussion about it there.

Re: OPR after Week Four Events
 

Thanks, Ether. Some other interesting phenomena:
- For matches predicted incorrectly, the median actual margin was 10 and the median OPR margin was 9. This is compared to the overall median margin of 27 and OPR margin of 23. (21% and 22% of all margins are in these upper bounds, respectively.) Pretty impressive!

- The overall median score error (note everything here is in absolute values) is 26%.
- The winning score error is 23%, while the losing is 32%. (For matches predicted correctly, the former is 21% while the latter is the same.)
- OPR underpredicts the winning score 60% of the time, and overpredicts the losing score 63% of the time.

I wonder if there's some way to identify good defense based on the over and under-prediction trends.

Re: OPR after Week Four Events
 

So, I did a quick scrape through our data from Grand Blanc. I found a handful off questionable values, but most of them were realtively low impact to averages.

The highest delta between OPR and what our scouts provided was 13 points. This particualr instance was 13 points favorable to that particular team. This team also had a few of the values I question as it appears that they did not score outside of auton in their final 2 matches (which I find hard to believe, but will verify later). This was one of the top scoring teams at the event.

The highest "unfavorable" OPR reading was 8.8 off from the scouts average. This particualr team also had some questionable data for one of their matches. Adjusting the values for that match to what I beleive were more accurrate (second scouting source), this delta went down to 5, and a different team became the most disadvantaged at 7.0. The team with this delta was a lower scoring team that OPR seems to be especially harsh on when comparing their 9.5 average to their 2.3 OPR.

To get average error, I took the absolute value of the error and found the average to be 3.5 pts, and the median error to be 2.9 pts.

Average OPR for the event was 24.8 and median OPR was 18.8. Thus the average error and median error for this event seems to be coming in at 15-16%.

Re: OPR after Week Four Events
 

I found an easy way to do this that just takes a minute. An index look up can be used. This is what, because of a limitation in Excel 2007, that this has to be done on the same worksheet and cannot be done in another workbook or even on another worksheet within the same workbook.

For an example index statement

=INDEX($E$1:$F$2542,MATCH(A2545,$E$1:$E$2542,0),2)

where A2545 is the cell containing a team number.

Re: OPR after Week Four Events
 

I really like this year's way of ranking teams. It is very straight forward and it does not depend on whether your opponent wants to let you have more ranking points. Every team tries to score as much as they can without holding back or worse trying to score for the other alliance. It is getting closer and closer to real sports which makes it exciting to watch. I hope the GDC recognize that and continues in this direction. It also makes the OPR more accurate which is what I like to see and better chance that the right teams are ranked at the top.

Re: OPR after Week Four Events
 

I think you're right...but at large events where teams may only play 8 matches, it seems there are still scenarios where the best teams are not necessarily at the top. Not really the fault of the game or even the ranking method, though.

Re: OPR after Week Four Events
 

Here's a sneak peek at Week5 OPR World Rankings, while we're waiting for Ed to post the update to his amazing spreadsheet.

Re: OPR after Week Four Events
 

Thanks Ed for helping Code Red & The Mavericks to Buckeye Regional Championship!

Been a big fan of your work for a long time!