TLDR
I looked at OPR and CCWM from qualification matches, and then added these numbers for each elims alliance. I then looked at who “should” have won each individual game in playoffs (based on the greater sum of each team’s stats), and who actually got the higher score. Below are the “correct guessing” rates for OPR and CCWM from 2002 - 2016:
YEAR OPR(%) CCWM(%) NUMBER OF PLAYOFF GAMES
2002 57.836 57.090 268
2003 62.162 60.811 370
2004 62.694 62.953 386
2005 67.435 65.994 347
2006 65.709 64.368 522
2007 53.835 55.572 691
2008 71.902 67.607 815
2009 66.667 62.583 906
2010 72.398 70.362 884
2011 74.153 71.926 1033
2012 68.746 65.272 1267
2013 71.534 68.708 1486
2014 69.188 65.102 1811
2015 70.672 69.073 2189
2016 70.007 66.013 2704
Thoughts
- I think this “OPR Prediction Percentage” is a relatively good metric to tell how “good” OPR is for a given year.
- CCWM is almost always worse than OPR, which I found interesting.
- I thought 2015 would have a much better OPR prediction score than 2014 did, but it turns out they were within a percent of each other.
Method
I used a Python script, pulling data from TBA to get OPRs and playoff results. For each event in a year, I got the OPRs/CCWMs for each team (which TBA gets only from quals matches, I believe). I then went through each playoff game (not the Bo3), and compared the two alliances scores to determine the winner (so not exactly how the tournament worked in 2015, but pretty much every other year). If the summed OPRs/CCWMs predicted the same winner as who actually won that game, it was a correct prediction. I decided to simply ignore ties and bad data from TBA.
# I beleive Python 2.7.9 or greater is required for this script
import requests
headers={'X-TBA-App-Id' : 'frc4917:customOPRCalculator:1'}
def get_alliance_stat(stats, alliance):
total_stat = 0;
for team in alliance'teams']:
total_stat += stats[team[3:]]
return total_stat
def prediction_percentage(stats, matches):
total = 0
correct = 0
for match in matches:
redAlliance = match'alliances']'red']
blueAlliance = match'alliances']'blue']
if (redAlliance'score'] == blueAlliance'score']): continue
try:
redAllianceStat = get_alliance_stat(stats, redAlliance);
blueAllianceStat = get_alliance_stat(stats, blueAlliance);
except KeyError:
continue
if ((redAllianceStat > blueAllianceStat) == (redAlliance'score'] > blueAlliance'score'])):
correct += 1
total += 1
return correct, total
def do_event(event_code, totals, playoffs_only=True):
url = 'https://www.thebluealliance.com/api/v2/event/' + event_code + '/stats'
r = requests.get(url, headers=headers)
stats_contents = r.json()
if not ('oprs' in stats_contents and stats_contents'oprs'] and stats_contents'ccwms']): return
url = 'https://www.thebluealliance.com/api/v2/event/' + event_code + '/matches'
r = requests.get(url, headers=headers)
matches_contents = r.json()
if playoffs_only:
matches_contents = [x for x in matches_contents if x['comp_level'] != 'qm']
correct, total = prediction_percentage(stats_contents'oprs'], matches_contents)
totals'opr_correct'] += correct
totals'num_games'] += total
correct, total = prediction_percentage(stats_contents'ccwms'], matches_contents)
totals'ccwm_correct'] += correct
for year in range(2002, 2017):
url = 'https://www.thebluealliance.com/api/v2/events/' + str(year)
r = requests.get(url, headers=headers)
events_contents = r.json()
totals = {'num_games': 0, 'opr_correct': 0, 'ccwm_correct': 0}
for event in events_contents:
do_event(event'key'], totals)
print(year)
print(totals)
print('OPR ' + str(totals'opr_correct'] / float(totals'num_games'])))
print('CCWM ' + str(totals'ccwm_correct'] / float(totals'num_games'])))
I looked around a bit, and hadn’t seen this type of evaluation of OPR on a year by year basis - sorry if I missed an existing one!
What do you think 2017’s “OPR Prediction Percentage” will be?