Overview and Analysis of FIRST Stats

[araniaraniratul]

Originally Posted by Citrus Dad

We're digging into your stats paper. It's quite useful for thinking about this issue.

However you wrote on p. 9:

Defense - Average of Opponent’s Match Scores
One simple way to measure a team’s defensive power is to simply average their opponent’s match scores.

That's not correct--it measures the average offensive output of the opponents in those matches. This is the error also of the assertion that OPR-CCWM=DRP as a defensive metric. The DPR is actually the OPR of the opposing teams in those matches played. A measure of defense will measure how much that average output differs from the average offense of those opponents in other matches.

(I'll have other thoughts as I go through this paper.)

I felt like the intention was to take the differences from the opposing team's average scores? That would make sense and is fairly obvious. Otherwise, I'm fairly lost there too!

[wgardner]

Originally Posted by Citrus Dad

Defense - Average of Opponent’s Match Scores
One simple way to measure a team’s defensive power is to simply average their opponent’s match scores.

That's not correct--it measures the average offensive output of the opponents in those matches. This is the error also of the assertion that OPR-CCWM=DRP as a defensive metric. The DPR is actually the OPR of the opposing teams in those matches played. A measure of defense will measure how much that average output differs from the average offense of those opponents in other matches.

Well, essentially I agree with you for real tournaments. Both the average of the opponent's scores and DPR will only be a meaningful measure of defensive performance if averaged over MANY matches, but quite frankly typical FRC and FTC tournaments don't have enough matches to make that average meaningful.

If you look at the simulated tournaments at the end of the paper with 10x the number of matches as a normal tournament, you'll see that these simple stats will in fact converge to the actual defensive contribution (but again, with 10x+ the normal number of matches). This is also noted in the "Limiting Behavior" discussion on page 11.

For example, the Dave measure for team "n" is the average of an opponent's scores, or (starting with the equations on page 6)

Dave_n = 1/# * Sum_{i,j,k,l,m} (Oi + Oj + Ok - Dl - Dm - Dn + Noise)

where # is the number of matches played by team "n" and where the summation is over team "n"'s opponents i, j, and k in each match and with team "n"'s alliance partners l and m.

Since the Dn term is the same in each term, you can pull this out and get

Dave_n = -Dn + 1/# * Sum_{i,j,k,l,m} (Oi + Oj + Ok - Dl - Dm + Noise}

If # is small and the O values are much bigger than the D values, then Dave_n will still be dominated by the O values. But if # is large, this becomes

Dave_n -> -Dn + 3*average(O) - 2*average(D)

In the paper, I normalize average(D) to be zero, so this becomes

Dave_n -> -Dn + 3*average(O) (the first equation on page 11 of the paper)

DPR has similar limiting behavior.

Both of them are measures of how well your opponents do in matches, which is some combination of the offensive strength of your opponents and your defensive contribution. If you average over a LOT of matches, then the strength of your opponents will eventually average out and all that you'll be left with is your defensive contribution. But in typical tournaments with typical numbers of matches, this will primarily be more of a measure of strength of opposition with perhaps a very small component being your team's defensive ability (except for perhaps the most severe outlier like the team that only plays strong defense every single match).

The Simultaneous OPR+DPR calculations described in the paper on pages 15-16 (and on page 19 if using MMSE instead of LS) do a better job of estimating defense as they take the opponent's offensive ability into account when computing defense, as you suggest. That's why, for example on the simulated tournament on page 27, Da (like average of opponent's scores) and DPRb (like DPR) do so much worse at predicting the defensive contributions compared with sDPR.

But I guess my only quibble with your post would be to say that opponent's average and DPR do very slightly measure defensive ability, though I agree that in practice for normal FRC and FTC tournaments this measurement is much more of a measurement of strength of your opposition's offensive contributions rather than strength of your defensive contribution.



BTW, I recently released a new version of my app for FTC which computes MMSE-based CPR and OPR live for FTC tournaments.

For exactly the reasons you mention, I refer to the difference between these as "Dif" on the stats screen rather than "Defense" as I had in earlier versions of the app. And in the app on the screen that explains the stats, I write "Dif measures a combination of the strength of the opposing alliances a team has faced and the strength of that team's defense. A positive Dif is good for a team, and signifies that the team's opponents have scored fewer points than average by that amount." (Note that in the app, Dif is normalized to be zero-mean across all teams, and with a positive number indicating that a team's opponents score fewer points than average by that number.)

In practice, I use the Dif stat more to see who was lucky by playing against weaker alliances than as a measure of defense.

[Citrus Dad]

Originally Posted by araniaraniratul

I felt like the intention was to take the differences from the opposing team's average scores? That would make sense and is fairly obvious. Otherwise, I'm fairly lost there too!

Yes, that's the added step that's needed. As mentioned below, look at the equations on p. 15 and p. 19 to see how to do this.

I also like the MMSE adjustment. However, one important point. This is a Bayesian estimation method. The errors terms around initial "guess" is normally distributed (or log normally since scores are constrained at zero), and presumably so are the regression results errors. But adding together 2 normal distributions does not result in a normal distribution. It's been 20 years since I looked at this issue, but there is software out there that correctly computes the resulting distribution.

[Citrus Dad]

Originally Posted by wgardner

But I guess my only quibble with your post would be to say that opponent's average and DPR do very slightly measure defensive ability, though I agree that in practice for normal FRC and FTC tournaments this measurement is much more of a measurement of strength of your opposition's offensive contributions rather than strength of your defensive contribution.

However, what you say about the DPR is also true of the OPR: it is a function of both the offensive output of the alliance AND the defensive ability of the opposing alliances. As you point out in your (impressive) paper, these scores are mostly impacted by the offense (except in Stack Attack ) with a small impact by defense on both sides. It's the deviation from the expected opposing OPR created the presence of a particular team that measures the defensive impact.

[wgardner]

Originally Posted by Citrus Dad

But adding together 2 normal distributions does not result in a normal distribution. It's been 20 years since I looked at this issue, but there is software out there that correctly computes the resulting distribution.

??? A random variable produced by summing two independent normally distributed random variables is also normally distributed. See, for example, this.

[araniaraniratul]

Originally Posted by Citrus Dad

Yes, that's the added step that's needed. As mentioned below, look at the equations on p. 15 and p. 19 to see how to do this.

I also like the MMSE adjustment. However, one important point. This is a Bayesian estimation method. The errors terms around initial "guess" is normally distributed (or log normally since scores are constrained at zero), and presumably so are the regression results errors. But adding together 2 normal distributions does not result in a normal distribution. It's been 20 years since I looked at this issue, but there is software out there that correctly computes the resulting distribution.

Is it because they're not independent? (Unless my understanding of random variables is way off) Otherwise I feel like two independent random variables (normal) in summation still leaves you with a new normal random variable. In terms of software, I know that numpy does a ton of those corrections, but not sure about this specific one.

[wgardner]

On a tangentially related note: I came across "Regularized Adjusted Plus/Minus" stats for NBA basketball players today. They are essentially exactly the stats we're using in FIRST! See for example, this link.

In our terminology, they view each basketball possession as an "alliance" of 5 players versus a defensive "alliance" of the 5 players on the other team. They count each possession as a "match" and then compute the stats just like we do. Every time a new player is subbed in, the alliance changes. "Raw plus/minus" is just like the averages discussed in the paper. "Adjusted plus/minus" is like sOPR and sDPR (compare the first equation in the link above with the equations on page 6 of the paper). "Regularlized adjusted plus/minus" is just like the MMSE version.

This link shows ORPM, DRPM and RPM for NBA players which are essentially just like sOPR, sDPR, and sCPR.