Overview and Analysis of FIRST Stats
 

Another thread for stat nerds. Again, if you don't know or don't care about OPR and CCWM and how they're calculated, this thread will probably not interest you.
----------------------------------------------------------------

Based on the recent thread on stats, I did way too much study and simulation of what the different stats do in different situations. I've attached a very long paper with all of the findings.

Many thanks to Ether who helped a lot with behind-the-scenes comments and data generation. I think he's off working on some related ideas that I suspect we'll all hear about soon.

The Overview and Conclusions of the paper are included below.

-----------------------------------------------------------------

Overview:

This paper presents and analyzes a wide range of statistical techniques that can be applied to FIRST Robotics Competition (FRC) and FIRST Tech Challenge (FTC) tournaments to rate the performance of teams and robots competing in the tournament.

The well-known Offensive Power Rating (OPR), Combined Contribution to Winning Margin (CCWM), and Defensive Power Rating (DPR) measures are discussed and analyzed.

New measures which incorporate knowledge of the opposing alliance members are discussed and analyzed. These include the Winning Margin Power Rating (WMPR), the Combined Power Rating (CPR), and the mixture-based Ether Power Rating (EPR).

New methods are introduced to simultaneously estimate separate offensive and defensive contributions of teams. These methods lead to new, related simultaneous metrics called sOPR, sDPR, sWMPR, and sCPR.

New MMSE estimation techniques are introduced. MMSE techniques reduce overfitting problems that occur when Least Squares (LS) parameter estimation techniques are used to estimate parameters on a relatively small data set. The performance of LS and MMSE techniques is compared over a range of scenarios.

All of the techniques are analyzed over a wide range of simulated and actual FRC tournament data, using results from the 2013, 2014, and 2015 FRC seasons.

-----------------------------------------------------------------

Conclusions

New improved techniques for incorporating defense into FRC and FTC tournament statistics have been introduced.

New MMSE techniques for estimating model parameters have been introduced.

Most FRC tournaments do suffer from a small data size, causing Least Squares estimates to be overfit to the noisy tournament data which degrades their performance in predicting match outcomes not in the Training set.

MMSE techniques appear to provide limited but significant and consistent improvements in match score and winning margin prediction compared to similar Least Squares techniques.

While incorporating defense into the statistics using MMSE estimation techniques does not result in any decrease in the statistical prediction performance, the advantages in doing so are usually quite small and may make it not worth the effort to do so unless a given FRC season is expected to have substantial defensive components. Occasionally incorporating defense can result in around an 8-12% further reduction in winning margin prediction error (e.g., 2014 casb, 2015 incmp, 2015 micmp tournaments), but this is rare.

MMSE based estimation of the sOPR, sDPR, and sCPR parameters results in the smallest squared prediction error for match scores and match winning margins across all of the studied parameters. MMSE based estimation of OPR parameters often produces results that are quite close.

Least Squares estimates of OPR, CCWM, and DPR using FRC tournament data probably overestimate the relative differences in ability of the teams. MMSE estimates probably underestimate the relative differences.

The small amount of data created in FRC tournaments results in noisy estimates of statistics. Testing set match outcomes from 2013-2015 often had very significant random components to them that could not be predicted by the best linear prediction methods, most likely due to purely random issues that occur in FRC matches.

Re: Overview and Analysis of FIRST Stats
 

Thanks William. That's a super useful paper. My takeaway is that while stats that incorporate defence might help a little, good old (LS based) OPR seems to be just about as good in most cases.

I have another area of study you could look into. I like to use OPR during tournaments for two purposes:

1) to derive a strength of schedule (basically the predicted winning margin). This is useful for deciding where best to spend limited resources on strategy planning, and alliance-mate repairs.

2) as a first order sort for pick list order

Both of these use cases suffer from even more drastic lack of data points. For the SOS, I use previous tournament data, and for the pick list, we've usually only completed 70% of the matches. For these use cases, it would be valuable to know:

for 1) Which predictor does best when the training set and the testing set are from different tournaments.

For 2) which predictor does best with 70% or less of the matches in the training set. (Maybe this is where MMSE solutions will shine)

Lastly, I'm intrigued by the possibility of incorporating scouting data (such as individual team point counts) into the MMSE calculation as a way to verify scouting records, but i have to think about that more.

If you can respond to any of these questions, I'd be obliged, otherwise thanks again for sharing your work!

Re: Overview and Analysis of FIRST Stats
 

This is a really well written paper, thanks for putting it together!

I have some questions about how to choose VarD/VarO and VarN/VarO since I'm unfamiliar with MMSE estimation. How would you go about choosing these values during/before an event?

Does this method lead to the same overfitting that using the training data as the testing data did with the LS estimators? Choosing the apriori variances after the fact to get the best results seems wrong, or is the effect actually too small in reality to be a factor? It seems like each set of training data also needs to find what variances work best, and then apply them to the testing data, instead of "searching" for the best values and applying them after the fact.


From this I'd expect that the values for VarD/VarO to be largely dependent on the game, yet the data shows that the "best" values depend very little on the game. For example, in the 2014 Newton Division the best values for VarD/VarO for sCPR was 0.10, but for 2014 Galileo it was 0.00! The complete other side of the search range! How can two divisions in the same year have such different values?

Re: Overview and Analysis of FIRST Stats
 

Yes, these are fair points. They're only picking 1 or 2 values for an entire tournament, but yes, they are searching. I might ideally suggest that the values be searched for an entire season (where the game is the same) to find the 1 or 2 values that are best for the entire season's worth of tournaments. It might also show how things are different from year to year.

They do vary from tournament tournament because again there just isn't enough data in a tournament to settle on a "true" underlying value.

Re: Overview and Analysis of FIRST Stats
 

This whole thing came about because I wrote an App for FTC tournaments where I wanted live estimates of stats and predicted match outcomes. The app predicts future match results based on stats so teams can know which matches are likely to be hard vs. easy. It also does this for sub-parts of the game (e.g., 2 matches from now, our opponents are really good at autonomous but not at the end game, so maybe we should play autonomous defense [in FTC], etc.).

OPR doesn't even exist when the # of matches played is less than the number of teams/2, and then suddenly it exists but is noisy, and then it progressively gets less noisy as more matches are played. So I was looking for a way to show stats, and it seemed like the stats should slowly incorporate information as matches are played.

The app currently predicts match scores and winning margin, but I'd also like to incorporate a "probability of victory" measure to show what kind of confidence exists.

The MMSE approach allows for estimated stats regardless of how many matches are played. I'll try to run some sims with 0-100%of matches played to see how well things work over time.

It also occurred to me to try to predict the match outcomes for the simulated tournaments where the underlying stats are completely known just to see the limits of how well match prediction could be if perfect knowledge of the underlying parameters existed.

Another thing that I could do would be to simulate how picking alliances based on the estimated stats would do vs. picking them based on the underlying stats. For example, if the top 3 teams are picked based on the various estimates (LS OPR, MMSE OPR, MMSE sCPR, etc.) and they are compared with the top 3 teams in the simulated tournaments where the underlying actual data is known (the actual underlying O and D), how many fewer points will the alliance end up scoring on average? This might be the real question that folks want to know... Gotta run now: more later.

Re: Overview and Analysis of FIRST Stats
 

One more quick thought on this, with more to come later:

Both 2014 galileo and newton sCPR searches picked VarN/VarO=3.

If VarD/VarO = 0, then the total match variance would be 3*VarO + VarN = 6*VarO.

If on the other hand VarD/VarO = 0.1, then the total match variance would be 3*VarO + 3*VarD + VarN = 6.3*VarO.

So while this looks like a really different result, we're only talking about a change in about 5% of the overall variance in a match that could be predicted with VarD/VarO being 0.0 or 0.1. Instead, it might be helpful to increase the step size in the VarN/VarO search, which is currently 1 (!), so each step in that search could cause a much greater change in the match variance.

Re: Overview and Analysis of FIRST Stats
 

The attached png file shows some interesting data.

This is for the 2014 casa tournament with simulated data using the model from the paper with Var(O)=100 (or stdDev(O)=10), Var(D)=0, and Var(N)=3*Var(O). The tournament had 54 teams, so each team got to play 1 time every 9 total matches. The tournament had 108 total matches, or 12 matches played by every team.

Each of the first 4 plots shows the estimated OPRs vs. the number of matches played per team (so X=1 means 9 total matches, X=2 means 18 total matches, etc.). The data points from 1-12 on the X axis correspond to 1 match per team, ... up to 12 matches per team (the whole tournament). The 13th point on the X axis is the actual underlying O values.

Plot 1 corresponds to the traditional Least Squares (LS) OPRs, which is also the MMSE solution where Var(N) is estimated to be equal to 0. Note that there are no OPR values until each team has played 4 matches, as that's the number of matches needed to make the matrix invertible.

Plot 2 corresponds to the MMSE OPR estimates where Var(N) is estimated to be equal to 1* Var(O). As the actual Var(N)=3*Var(O), this is underestimating the noise in each match.

Plot 3 corresponds to the MMSE OPR estimates where Var(N) is estimated to be equal to 3* Var(O) (the "correct" value).

Plot 4 corresponds to the MMSE OPR estimates where Var(N) is estimated to be equal to 10* Var(O), greater than the actual noise.

Plot 5 shows the percentage error each curve has in estimating the actual underlying O values.

Comments:

The LS OPR values start out crazy and then settle down a bit. Looking at the step from X=12 (the final OPRs) to X=13 (the "real" O values), you can see that the final OPRs have more variance than the real O values. This means that the final OPRs are still overestimating the variance of the abilities of the teams.

Look at the X=1 points for Plots 2-4. The MMSE estimates start conservatively with the OPRs bunched around the mean and then progressively expand out. Plot 4 shows the noise overestimated (the most conservative estimate), so the OPRs start out very tightly bunched and stay that way. Plot 2 starts out wider, and Plot 3 starts out in the middle.

Interestingly, you can see that each X=1 point for the MMSE plots have 3 teams with the same estimate. This makes sense, as after having played 1 match, the 3 teams on each alliance are indistinguishable from each other and it requires more than 1 match played by each team to start separating them.

Look at the X=12 (the final estimates) vs X=13 (the "real" O values) points for Plots 2-4. Plot 2 looks like it's still over estimating the variance, Plot 3 has it about right, and Plot 4 has underestimated the true variance even at the end of the tournament (you see the Plot 4 OPRs expand out from X=12 to X=13). [Edit: checking the numbers for the run shown, the variances of the OPRs computed by LS, MMSE 1, MMSE 3, and MMSE 10 were respectively 164, 138, 102, and 47, confirming the above comment. The MMSE 3 solution using the "right" Var(N) estimate is quite close to the true underlying variance of 100. Over multiple runs, the MMSE 3 solution is slightly biased under 100 on average, showing that more matches are needed for it to converge to the "right" variance. All of the techniques do eventually converge to the right solution and variance if the tournament is simulated to be much greater than 108 matches.]

In Plot 5, the performances of the different techniques get close to each other as the tournament nears completion. They should all converge as the number of matches grows large as the LS and MMSE solutions will eventually converge to each other. But they are off by quite a bit early on. Even though the MMSE 1 solution with Var(N) underestimated at 1*Var(O) is underestimating the Var(N), it still gives pretty good results.

Re: Overview and Analysis of FIRST Stats
 

Super cool. It definitely looks like MMSE OPR gives better early predictions, at least in that data set. Now I need to write an app to give us live calculations during an event...

Did you get a chance to look at using training data from previous events? For instance, does MMSE OPR values from the last regional accurately predict a teams performance at CMP? What should we give for the MMSE parameter estimates in this case?

Re: Overview and Analysis of FIRST Stats
 

This is a great question but will take me a lot more time to do. I'd have to figure out which teams from CMP played in which regionals, then get all of the parameters computed from all of the regionals, and then do the study. I'll put this on my todo list but this would take a while even if I made it my top priority.

Re: Overview and Analysis of FIRST Stats
 

Thanks Will, the fact that even choosing a bad value for Var(N) gives decent results alleviates a lot of my concerns about searching for Var(N) and Var(D) after the fact.

It’s also impressive at how much better MMSE techniques are for when an event is underway and not a lot of matches have been played. This is helpful for predicting the outcome of the second day of matches (and thus seeing who is likely to be a captain).
Is this behavior typical for all stats or just OPR? Could you run a similar plot for sCPR? (since that stat seems to do slightly better than OPR).


Additionally, how would you implement the techniques described in the “Advanced MMSE Estimation” section? What would you change in the pseudocode to, for instance, change a team’s apriori Oi?

Re: Overview and Analysis of FIRST Stats
 

Plots are attached for 2 runs: one with the true Var(D)=0.1 & Var(N)=3, and one for Var(D)=0.0 & Var(N)=3. (these are relative to Var(O)).

The top row is with the estimated Var(D)=0 and the estimated Var(N)=0, 1, 3, and 10 as before. So the top left corner is regular OPRs and the top row is MMSE OPRs.

The middle row is the same but with Var(D) estimated at 0.1 and the bottom row is the same with with V(D) estimated at 0.2.

Note that with Var(D)>0 and Var(N)=0, the results are always the same, the "vanilla" LS sCPR. That's shown in the left middle. Even worse than the OPR, it doesn't really start having values until the rank of the matrix is 2*#teams-1 which is at the 7th match per team. It is VERY overfit which is why it starts noisy and stays noisy.

The bottom left shows the plots of the percentage of the combined O+D (or O in the case when D=0) left after prediction. It's saturated to be no worse than 100%, though the LS OPR and sCPR are worse than 100% when the number of matches is small, meaning that the prediction error has more variance than the original set of parameters (!). The black curve is the LS OPR and the red curve is the LS sCPR which is so overfit that it's worse than nothing until the very last match is played.

The regular MMSE equation is like the equation shown in Appendix B for EPR, except that 1/sig2C I there is the inverse of the correlation matrix of the expected parameters. If the expected means change, you don't add Oave at the end but a vector of your expected means. If the expected variances change for the noise or the parameters, then you change the covariance matrices.

Basically, there's a general equation for arbitrary expected mean vectors and covariance matrices of both the parameters and the noise, so you can run the estimation algorithm given any set of expected mean vectors and covariance matrices.

Re: Overview and Analysis of FIRST Stats
 

Following up on this, the attached image is snipped from the Wikipedia Page on MMSE Estimation about half way down.

In this equation, x is the parameter you're trying to estimate (like O), z is the noise (like N), xhat is your estimated parameters, xbar is the expected mean, Cx is the covariance matrix of x, and Cz is the covariance matrix of the noise.

For example, in my MMSE equation for the OPRs, xbar is just Oave (but you could have it be a vector with team specific expectations). A* xbar is just the average match outcome which I have as 3*Oave (but again, if you expect xbar to be team specific then that would cause non-constant match mean scores). Cz is just sig2n * I and Cx is just sig2o * I. I plugged these in and simplified the equations. But if things are more complicated (like with EPR), then you just plug in whatever complicated assumptions you have and go from there.

It would be neat if we could study the best predictor of a team's OPR at championships from the OPR they had in their last regional before championships using data from previous years. We'd probably come up with a mean and variance of this best predictor, and then we could plug these in and have some expectations for what championships would look like even before the first match was played. Then as matches are played, the values update to include the new information using the MMSE equation with changing A and Mo.

Re: Overview and Analysis of FIRST Stats
 

The dust seems to have settled in this thread so I thought I'd toss this out for discussion.

Re: Overview and Analysis of FIRST Stats
 

Neat idea!

I think this is essentially minimizing another "mixture of errors" like the original EPR proposal. In this case, you're minimizing the sum of the standard error measure that leads to your normal parameters plus the squared error between the parameters and your a priori expectations of them, with a weighting factor added in to adjust the importance of the original measure and your new second measure. If you form your error measure this way, then take the derivative of the error squared, set the result equal to zero, and solve for the parameters, you end up with the equation that you're solving.

That's another way to incorporate a priori expectations of the means, like the MMSE estimates do. And if you keep the weights constant, it would progressively go from a priori info at the start of a tournament to incorporating mainly match info later in a tournament, again like the MMSE estimates do. About the only thing that would be hard to do using this method would be incorporating a priori variance estimates, but then I suspect that this "feature" of the MMSE estimates has limited practical utility anyway.

Did you do any testing of this method to see how it does and how well it compares?

If you were so inclined, I'd love to see a database with the following data for Championships for the past 3 years:

. Ar, Ab, Mr, Mb for the CMP divisions (which we already have because you already generated them: thanks again!)

. OPR, OPRm1, and OPRm3 for all of the regional tournaments that all of the CMP teams played in that year, and perhaps the mean and variance of each of these statistics at the respective tournaments. I'm calling OPRm1 and OPRm3 the MMSE estimates with Var(N)/Var(O) estimated at 1 and 3 respectively.

Ideally, there might be csv files with a row for each team in each CMP division (in the same order as the columns of the corresponding CMP Ar and Ab matrices) and columns with a 0 if the team didn't play in that week or a number if the team did play (e.g., OPR, or mean OPR for that tournament that week, etc). So, for example, for 2014 archi, there might be the following files, each containing 100 rows (one per team) by 7 columns (one per week):

ARCHI_OPR
ARCHI_OPRm1
ARCHI_OPRm3
ARCHI_OPR_mean
ARCHI_OPRm1_mean
ARCHI_OPRm3_mean
ARCHI_OPR_var
ARCHI_OPRm1_var
ARCHI_OPRm3_var


Any chance you'd be up for generating this data?


Given this data, we could test out different methods for predicting the match outcomes of the CMP divisions. I suggest that it would be interesting to try to predict

A. The results of CMP matches completely based off the results from previous tournaments.

B. The results of the last 3/4 of the CMP matches based off the results from previous tournaments AND the first 1/4 of the CMP matches.

C. The results of the last 1/2 of the CMP matches based off the results from previous tournaments AND the first 1/2 of the CMP matches.

Re: Overview and Analysis of FIRST Stats
 

I think this might just be mmse the more I think about it. Plug Oave in for your a priori means and stdevN /stdevO in for your weights and solve and I think you just get the mmse equations. I'm out now and typing on my phone but will look into it more later.

Re: Overview and Analysis of FIRST Stats
 

hi all,

i'm not nearly on your level of mathematics, as i'm only about to graduate high school. but for my final project of my AP statistics class, i did a study on FRC playoff performances by alliances ranked 1-8.

there are a lot of problems with my study and i'm not afraid to admit it in terms of hypothesis testing and whatnot, so i'll just provide some statistics that i found interesting. if anyone wants the python script i used to download the data i can provide it.

The #1 seed alliances won 71% of regionals, districts, and district championships this year (so it does not include world champ data nor exhibition data).

The #1 seed alliances failed to advance past quarterfinals in just 4 of 109 events.

For the second place alliance, the #1 and #2 seeds made up 54% of those. This does not account for which alliance finished first.

The #8 alliances did not win a single event. Also, the #8 alliances made it to finals only 4 times.

The #7 alliance won a single event (Finger Lakes / NYRO). The #7 alliances finished second place only twice.

The #3, #4, #5, and #6 seeds were victorious in 5, 2, 4, and 5 events overall, respectively.

Thus, the #1 and #2 alliances contributed to 84.4% of event victories.

The mean points per game for all 8 alliances (total number of points worldwide divided by total number of games worldwide) were:

145, 108, 90, 80, 75, 75, 70, 60. (these are rough numbers as i'm reading them off a png image and i'm too lazy to download the excel file again, heres some more stats on points per game overall: https://i.imgur.com/dgIYLrQ.png )

some interesting stuff i guess. I'd be interested to do more with OPR/CCWM/CPR/EPR and whatnot once I get more experience with maths. might be a little bit though :p

Re: Overview and Analysis of FIRST Stats
 

Yep, I just ran through the equations and Ether's doc is just MMSE estimation where the expected means are given and where the weights are the ratios of the standard deviation of the match noise over the standard deviation of the variance of your estimate of the mean. So Ether's document is just another way of describing MMSE with a priori info!

Re: Overview and Analysis of FIRST Stats
 

I did some tests on how apriori information can improve the estimation. I wanted to see how these stats would do in predicting the outcomes for the last day of an event (to project the standings). The following data is from the four 2014 Championship Divisions. For each division 150 qual matches were played on the first two days, leaving 17 on the last one. Since the event was basically complete, LS OPR did pretty well. I suspect that for an event with more matches on the last day (like a District Championship) it would perform worse.

I used the LS “World OPR” as the prior for one of the tests. For another I averaged Oavg and World OPR (essentially regressing them to the mean). All of the MMSE calculations were done with VarN/VarO at 2.5

LS OPR (no prior): 69.12%
MMSE OPR (Oavg prior): 69.12%
MMSE OPR (World OPR prior): 72.06%
MMSE OPR (regressed World OPR prior): 72.06%

Adding prior information improved the predictions by a couple of matches, I’ll try this again later for a District Championship and see how it turns out.

Re: Overview and Analysis of FIRST Stats
 

Thanks, AGPapa! I wonder if you could do the following:

1. Could you report the average squared error in the scores and/or winning margin, in addition to the probability of correct match outcome?

2. Your runs were using 150 values as the "training set" and 17 as the "testing set". Could you run them with other splits, like using 50 or 100 matches as training and the remaining part as testing? It would be interesting to see how the different techniques perform as the tournament progresses.

Cheers.

Re: Overview and Analysis of FIRST Stats
 

Here's the average squared error in the match scores.

LS OPR (no prior): 3555.75
MMSE OPR (Oavg prior) : 3088.70
MMSE OPR (World OPR prior): 2973.91
MMSE OPR (regressed World OPR prior): 3000.04


I'll work on the 50 and 100 match splits and put them up later.

EDIT: I think I should elaborate a bit more on how I'm doing the priors. I've been adding a constant to everyone's World OPR so that the average of the OPRs is equal to Oavg. Essentially I'm saying that every robot improved by a constant. (These constants are very small, only Archimedes was greater than 1.5 points)

Re: Overview and Analysis of FIRST Stats
 

Check out the attached image. It has a lot of data in it. Here's the explanation:

I simulated a tournament using the 2014 casa structure with the underlying O components having mean 25 and stdev 10, as in most of my sims. I set the D component to 0 (no defense) and the match noise component N to 3 (so random variability roughly equals the variability due to the 3 offensive teams).

The top left plot shows the LS OPRs after each team has played 1-12 matches. X=13 corresponds to the actual, underlying O values.

Across the top row is the standard MMSE OPRs with Var(N)/Var(O) estimated at 1, 3, and 5. You can see that the OPRs start progressively tighter at X=1 and then branch out, with 3 and 5 branching out more slowly.

So the top row is basically the same stuff from the paper, just MMSE OPR estimation. It's assuming an apriori OPR distribution with Oave mean and varying standard deviation for all of the OPRs.

Now, the plots on the 2nd and 3rd rows are MMSE estimation but where we have additional guesses about what the specific OPRs should be before the tournament starts (!!). For example, this could happen if we try to estimate the OPRs before the tournament from previous tournament results.

The 2nd row assumes that we have an additional estimate for each team's OPR with the same standard deviation of all of the OPRs. So for example, the underlying Os are chosen with mean 25 and stdev 10. Row 2 assumes that we have additional guesses of the real underlying O values, but the guesses have random noise added to them with a stdev of 10 also. So they're noisy guesses, similar to what we might have if we predicted the championship OPRs from the teams' OPRs in previous tournaments.

The 3rd row is the same, but the extra guess has a standard deviation of only 0.3 of the stdev of the actual underlying O. In this example, we have another guess of the real O values with standard deviation of only 3 before the tournament starts.

The left center plot shows how well the techniques do at estimating the true, underlying O values over time. You can see the clumping on the left based on how much knowledge was known ahead of time.

The left bottom plot shows how well the techniques do at estimating the match scores. Note that with Var(N)/Var(O)=3, the best we should be able to do is 50% so the fact that we're just a bit under 50% is an artifact of this being only 1 simulation run. Again, you can see the clustering on the left of the plot based on how much apriori info was known.

For the most part, the results are pretty similar at the end of the tournament, but you can clearly see the advantage of the apriori inforomation at the start of the tournament.

If all you ever use OPRs for is for alliance selection at the end of the tournament, then there's not much advantage to going with MMSE over LS. But if you would use live stat information to plan out your strategy in upcoming matches during a tournament, then MMSE could provide you with benefits.

So, the question is: how good of an estimate can we get for the OPRs before a tournament starts based on previous tournament data?

Re: Overview and Analysis of FIRST Stats
 

By using priors, you are entering in Bayesian statistics. This is very interesting analysis, and is the actual way that we do statistical analysis. It's been quite a while since I studied the Bayesian technique, but here's a primer.

Re: Overview and Analysis of FIRST Stats
 

Yes. In fact the MMSE techniques described in the paper are also called "Bayesian Linear Regression" by some. Compare the equations for mu_n in that link with the MMSE equations in the paper and you'll see that they're basically the same.

Re: Overview and Analysis of FIRST Stats
 

I've calculated the errors in match score predictions for every 10th split from 50 to 150 matches.

The X axis in the attached chart is the number of matches in each division used as training set. The Y axis is the square root of the averaged squared error per match in the testing set (the remaining matches).

You can see that adding apriori estimations makes the predictions better across the board.


Interestingly, the MMSE (Oavg prior) estimation starts out pretty good at 50 matches (a little under a third of the way through). I wonder why Will's simulated results are different? You can see that the gradual improvements still exist, just as in the simulation (except for the last datapoint. I think that on the last day teams played differently than before).

Re: Overview and Analysis of FIRST Stats
 

Thanks AGPapa! As always, I have a few questions:

Can you elaborate on exactly what you mean by Oavg prior and World OPR prior? Does Oavg prior mean that you're setting them all to the same average a priori guess? Does World OPR prior mean that you're setting the a priori guesses to their final OPRs (as if you knew ahead of time what the OPRs actually were)?

I might expect that using the actual Worlds OPRs as priors would be uniformly good, as you're telling the algorithm ahead of time what the final guess would be (if I'm understanding you correctly, which I might not be). But I'm surprised that just setting the priors to be the overall average gives such uniform results.

Since your sim is probably easy to modify now (?), could you start it all off at 10 matches instead of 50?

Also, it doesn't surprise me that things get a little noisy at the very end because at that point you're only averaging the few matches left. They could just happen to be easy to predict or hard to predict and now there's only a few of them, so it seems entirely reasonable that they might move a bit from the trendline.

Re: Overview and Analysis of FIRST Stats
 

Ah, sorry for the confusion! "World OPR" refers to the OPR prior to the Championship event. It combines all regionals into one big "A" and "b" matrices and solves it. I just took that number from Ed Law's spreadsheet. It doesn't take any data from matches that haven't been played yet.

Oavg is just setting the priors to the average score of matches in the training set divided by 3.

I've attached the chart going back to match 10.

Re: Overview and Analysis of FIRST Stats
 

Thanks! That clears things up a bit. And these are all still with Var(N)/Var(O)=2.5 like before? Can you post a link to Ed Law's spreadsheet? (Sorry, I'm really an FTC guy coming in late to the CD FRC forums.)

Compare your chart with the bottom left chart of this (from a few posts ago, and from a simulated 2014 casa tournament). The blue lines at the bottom of that chart are if the OPRs are known to within a standard deviation of 0.3 times the overall standard deviation of all of the OPRs, and the red/pink/yellow lines are if the OPRs are known to within 1.0 times the standard deviation of all of the OPRs. Your bottom chart looks like it could be somewhere between those two. (Also note that my chart is percent of the variance of the error in the match result prediction, whereas yours is the absolute (not percent), standard deviation (not variance) of the error in the match result prediction, so they're just a bit different that way. As the stdev is the square root of the variance, I would expect that the stdev plot to be flatter than the variance plot, as it seems to be.)

Could you compute the standard deviation of the error in your "OPR prediction" (i.e., compute the OPR from worlds minus the OPR from the previous regional tournaments, and take the standard deviation of the result)? And then compare that to the standard deviation of all of the OPRs from all of the teams at Worlds (i.e., just compute all of the OPRs from Worlds for all of the teams and compute the standard deviation of those numbers). It would be interesting to know the ratio of those two numbers and how it compares to the plots in my simulated chart where the ratios are 1 and 0.3 respectively.

And I guess while I'm asking: what was the standard deviation of the match scores themselves?

Re: Overview and Analysis of FIRST Stats
 

Yes, still with VarN/VarO = 2.5. This was chosen fairly arbitrarily, better numbers probably exist.

Here's Ed Law's database. He has the LS OPRs for every regional/district since 2009. He's also has the "World OPRs" for each team in each year in the Worldrank tab.

Thanks, that makes a lot of sense.

I'm having some difficulty understanding what you're asking for here, but here's what I think you're looking for.

std dev of Previous OPR (LS) - Champs OPR (LS): 18.9530
std dev of Champs OPR (LS): 25.1509
std dev of match scores: 57.6501


I want to point out that the previous OPR is not an unbiased estimator of the Champs OPR. Champ OPRs are higher by 2.6 on average. (In my MMSE calculations I added a constant to all of the priors to try and combat this).


EDIT: I think we can use this to find a good value for VarN/VarO.

Var(Match Score) = Var(O) + Var(O) + Var(O) + Var(N)
Assuming that Var(Champs OPR) = Var(O) then we can solve the above equation and get that Var(N) = 1425.8, so Var(N)/Var(O) is 2.25. Now this is only useful after the fact, but it confirms that choosing VarN/VarO to be 2.5 wasn't that far off.

Re: Overview and Analysis of FIRST Stats
 

Yes, that all makes sense and looks good. The ratio of the stdev of the opr guess/ the stdev of the oprs themselves was 18.95/25.15 = 0.75 which is between the values of 1.0 and 0.3 that I used in my sims, so it seems consistent that your plot shape is sort of between those two.

I also renormalize the oprs to agree with the new average, as you described.

I found that the variance of the ls oprs was often a bit above the variance of the true underlying o values because of the slight overfitting, so if anything you might be slightly underestimating the variance of n. 2.5 - 3.0 is probably a good guess.

Thanks for all of the great data!

Re: Overview and Analysis of FIRST Stats
 

We're digging into your stats paper. It's quite useful for thinking about this issue.

However you wrote on p. 9:

Defense - Average of Opponent’s Match Scores
One simple way to measure a team’s defensive power is to simply average their opponent’s match scores.

That's not correct--it measures the average offensive output of the opponents in those matches. This is the error also of the assertion that OPR-CCWM=DRP as a defensive metric. The DPR is actually the OPR of the opposing teams in those matches played. A measure of defense will measure how much that average output differs from the average offense of those opponents in other matches.

(I'll have other thoughts as I go through this paper.)

Re: Overview and Analysis of FIRST Stats
 

I felt like the intention was to take the differences from the opposing team's average scores? That would make sense and is fairly obvious. Otherwise, I'm fairly lost there too!

Re: Overview and Analysis of FIRST Stats
 

Well, essentially I agree with you for real tournaments. Both the average of the opponent's scores and DPR will only be a meaningful measure of defensive performance if averaged over MANY matches, but quite frankly typical FRC and FTC tournaments don't have enough matches to make that average meaningful.

If you look at the simulated tournaments at the end of the paper with 10x the number of matches as a normal tournament, you'll see that these simple stats will in fact converge to the actual defensive contribution (but again, with 10x+ the normal number of matches). This is also noted in the "Limiting Behavior" discussion on page 11.

For example, the Dave measure for team "n" is the average of an opponent's scores, or (starting with the equations on page 6)

Dave_n = 1/# * Sum_{i,j,k,l,m} (Oi + Oj + Ok - Dl - Dm - Dn + Noise)

where # is the number of matches played by team "n" and where the summation is over team "n"'s opponents i, j, and k in each match and with team "n"'s alliance partners l and m.

Since the Dn term is the same in each term, you can pull this out and get

Dave_n = -Dn + 1/# * Sum_{i,j,k,l,m} (Oi + Oj + Ok - Dl - Dm + Noise}

If # is small and the O values are much bigger than the D values, then Dave_n will still be dominated by the O values. But if # is large, this becomes

Dave_n -> -Dn + 3*average(O) - 2*average(D)

In the paper, I normalize average(D) to be zero, so this becomes

Dave_n -> -Dn + 3*average(O) (the first equation on page 11 of the paper)

DPR has similar limiting behavior.

Both of them are measures of how well your opponents do in matches, which is some combination of the offensive strength of your opponents and your defensive contribution. If you average over a LOT of matches, then the strength of your opponents will eventually average out and all that you'll be left with is your defensive contribution. But in typical tournaments with typical numbers of matches, this will primarily be more of a measure of strength of opposition with perhaps a very small component being your team's defensive ability (except for perhaps the most severe outlier like the team that only plays strong defense every single match).

The Simultaneous OPR+DPR calculations described in the paper on pages 15-16 (and on page 19 if using MMSE instead of LS) do a better job of estimating defense as they take the opponent's offensive ability into account when computing defense, as you suggest. That's why, for example on the simulated tournament on page 27, Da (like average of opponent's scores) and DPRb (like DPR) do so much worse at predicting the defensive contributions compared with sDPR.

But I guess my only quibble with your post would be to say that opponent's average and DPR do very slightly measure defensive ability, though I agree that in practice for normal FRC and FTC tournaments this measurement is much more of a measurement of strength of your opposition's offensive contributions rather than strength of your defensive contribution.



BTW, I recently released a new version of my app for FTC which computes MMSE-based CPR and OPR live for FTC tournaments.

For exactly the reasons you mention, I refer to the difference between these as "Dif" on the stats screen rather than "Defense" as I had in earlier versions of the app. And in the app on the screen that explains the stats, I write "Dif measures a combination of the strength of the opposing alliances a team has faced and the strength of that team's defense. A positive Dif is good for a team, and signifies that the team's opponents have scored fewer points than average by that amount." (Note that in the app, Dif is normalized to be zero-mean across all teams, and with a positive number indicating that a team's opponents score fewer points than average by that number.)

In practice, I use the Dif stat more to see who was lucky by playing against weaker alliances than as a measure of defense.

Re: Overview and Analysis of FIRST Stats
 

Yes, that's the added step that's needed. As mentioned below, look at the equations on p. 15 and p. 19 to see how to do this.

I also like the MMSE adjustment. However, one important point. This is a Bayesian estimation method. The errors terms around initial "guess" is normally distributed (or log normally since scores are constrained at zero), and presumably so are the regression results errors. But adding together 2 normal distributions does not result in a normal distribution. It's been 20 years since I looked at this issue, but there is software out there that correctly computes the resulting distribution.

Re: Overview and Analysis of FIRST Stats
 

However, what you say about the DPR is also true of the OPR: it is a function of both the offensive output of the alliance AND the defensive ability of the opposing alliances. As you point out in your (impressive) paper, these scores are mostly impacted by the offense (except in Stack Attack :ahh: ) with a small impact by defense on both sides. It's the deviation from the expected opposing OPR created the presence of a particular team that measures the defensive impact.

Re: Overview and Analysis of FIRST Stats
 

??? A random variable produced by summing two independent normally distributed random variables is also normally distributed. See, for example, this.

Re: Overview and Analysis of FIRST Stats
 

Is it because they're not independent? (Unless my understanding of random variables is way off) Otherwise I feel like two independent random variables (normal) in summation still leaves you with a new normal random variable. In terms of software, I know that numpy does a ton of those corrections, but not sure about this specific one.

Re: Overview and Analysis of FIRST Stats
 

On a tangentially related note: I came across "Regularized Adjusted Plus/Minus" stats for NBA basketball players today. They are essentially exactly the stats we're using in FIRST! See for example, this link.

In our terminology, they view each basketball possession as an "alliance" of 5 players versus a defensive "alliance" of the 5 players on the other team. They count each possession as a "match" and then compute the stats just like we do. Every time a new player is subbed in, the alliance changes. "Raw plus/minus" is just like the averages discussed in the paper. "Adjusted plus/minus" is like sOPR and sDPR (compare the first equation in the link above with the equations on page 6 of the paper). "Regularlized adjusted plus/minus" is just like the MMSE version.

This link shows ORPM, DRPM and RPM for NBA players which are essentially just like sOPR, sDPR, and sCPR.