paper: How Accurate is OPR?

[Basel A]

Originally Posted by Joe Ross

How did you validate the accuracy of your scouting data? Why did you choose to use percent error, rather then absolute error?

A few more things that would be interesting to look at:

Do the results change if you only look at the top 24 or 30 teams at an event (the teams you would be considering when forming a pick list)?
Is there an OPR at which it becomes more accurate? Looking at chart 13, OPRs above 15 seem much better then those below 15 (obviously game dependent).
Can you quantify the percent chance that Team A is better then Team B, given a specific OPR difference (IE, Team A has an OPR that is 1 higher then Team B, and has a 55% chance of being better then Team B, but Team C has an OPR that is 10 higher then Team B, and has a 90% chance of being better then Team B.
Does the percent error histogram still look normal if you discard the outliers and put more bins between -100% and +200%?

These are all good things to think about. Particularly, you may want to reconsider your use of percent error as opposed to absolute error. I'm estimating here, but it looks like absolute error would've been pretty consistent regardless of true scoring. Take a look at the correlation of those; I'd bet very little of the variation in absolute error is explained by variation in true scoring average.

If there were such a correlation, you would have noticed it in the residual plot for your OPR-True Average linear regression. I didn't notice a residual plot in your pdf; they are essential for determining if your model is a good fit for the data.