TL;DR: I observe a thing I don’t understand. I think about it. I come up with possible explanations. I find one that seems particularly plausible (at least to me). If that theory is correct, then I can make a prediction about the future, which I do. Soon (tonight) I will test this prediction. I will publish results as I get them. More science to come…
Most of you probably haven’t been following this thread (warning it’s a long one) Comparison of DIV MAX OPRs.
In that thread, there is a lot of good analysis of Newton and the CMP Divisions. One is by Yours Truly. In that posting, I have the following chart:
http://i.imgur.com/fVX1Z1v.jpg
If you look at the chart, there are a few things that catch my eye. First it looks a lot like a typical Gaussian Distribution (I did some checking, it isn’t quite Gaussian but it is pretty close).
Second, it has a weird bump in the top and a few wiggles here and there. These don’t trouble me at all. There are lot of oddball steps in the making of divisions. It doesn’t surprise me that some of those get reflected in some unexplained wiggles. Plus also, I only did 2,500 simulated Championships. Those wiggles might smooth out if I did 2,500,000 (which I am not going to do) or I might end up with slightly different wiggles if I reran the simulation. But frankly, I don’t think it matters much and I didn’t lose any sleep of that unexplained bump.
Third, the average of the distribution is not centered around zero. THAT is weird in a way that catches my attention. Think about it. There is no reason that the way we divide up the teams should be allowed to generate divisions that, ON AVERAGE, are BETTER than the CMP Teams as a whole (we are not systematically dropping low performing teams for example).
This one I literally DID lose sleep over – not because I think it is important so much as I think it is interesting in ways the bump is not. Part of the reason I found it interesting is that even though the effect is small (+0.16 point shift using Method #1, and 0.36 point shift using Method #2) it is the kind of thing that COULD point to something being wrong with my simulation that could be hiding more fundamental problems. And that might make an insignificant shift be very important indeed. It is often the case that these types of things help me find errors in my modeling or sometimes lead to new insights into the problem that I would have have paid attention to if I were only concerned about the magnitudes.
So… …I lay in bed last night thinking about this unexplained shift. Morning found no solution but an inkling of an idea came to mind as I cut up my apples for the day (if 1 apple keeps the doctor away, what does 4 do?). I thought about the Trapezoid Rule and Integration. It is well understood that for some types of functions (specifically functions with positive second derivatives in the range of interest) the Trapezoid Rule over estimates the area under the curve. Here is a graph that helps show why (image source):
The trapezoids have a little area they count that is not quite under the curve in these cases and that causes a systematic over estimate of the area under the curve.
I was doing integration. Was that my problem? I thought and thought and decided that it wasn’t the source of this bias. BUT… Then it came to me.
The Percentile Function that I was so blithely using in my Excel simulation, what about that? Yes. That is it!!! (at least I think so).
My method has me calculate the Nth Percentile Max OPR for the 75 team Division as well as the Nth Percentile Max OP for the 600 team Championships. I generate these values in tables and then do the subtraction. It will not surprise you that my table has 100%, 99%, 98%, … , 50% in the left hand column.
Excel is a great (and terrible) program and so it is perfectly happy spitting out numbers to the right of these whole number Percents under column headings like Div0, Div1, … , Div7, AllCMP.
Think about what Excel is doing. In the case of the AllCMP life is easy for Excel. I ask it to give me the 97th Percentile team and it says to itself, 600 teams in the range AllCMP, 97th Percentile? No prob: that is just the 18th highest Max OPR (1-0.97)*600 = 18.
In the case of the 75 Team Division, the 97th Percentile Max OPR is not on the list because (1-0.97)*75=2.25. Excel has to ESTIMATE that by taking the 2nd best Max OPR (which the 97.333rd Percentile) and the 3rd best Max OPR (which is the 96th Percentile). How does it do the estimate?
From Excel Help:
If k is not a multiple of 1/(n - 1), PERCENTILE.INC interpolates to determine the value at the k-th percentile.
So. I think I have found the culprit. Based on the graph below, I think you will agree that the shape of the Max OPR function has a bias to be concave up in the region of interest.
http://i.imgur.com/QGMmpz2.jpg
I think for the same reason that the Trapezoidal Rule over estimates the area under there curve, the Excel’s PERCENTILE.INC will over estimate Max OPR when it has to interpolate. Also, Trapezoidal Rule case, the curve is the greater the curvature of the function you are integrating, the greater the error. And in my case, when I included the right most portion of the curve (Method #2) which tends to have the Poofs, Robonauts, etc. and therefore have more curvature, my error went up. Doesn’t prove anything but it IS suggestive.
That is my theory. Now for my prediction that, if true, will support my theory.
If I change the values in the left column of my table to
- 100.00%= 75/75
- 98.67% = 74/75
- 97.33% = 73/75
- 96.00% = 72/75
- 94.67% = 71/75
- …
Then, Excel will not have to use its interpolation feature so the shift should go away.
Let’s see how this turns out.
More science to come.
Dr. Joe J.