Quote:
Originally Posted by Eugene Fang
- Isn't best linear combination kind of cheating? It seems like you're picking weights after knowing the outcome.
|
Essentially, yes, it is cheating. The linear combination wasn't a particularly important part of this effort, I kind of tacked it on at the end. My interest in the linear combination is twofold:
I wanted to know how much "better" the models could get by adding two or more models together. The result was that it has reasonable, but not extremely high value, so it is worth more investigation.
I wanted to know how "different" each model's predictions were to each other. If two models both have reasonable predictive power, and their correct predictions are uncorrelated, taking a combination of the two will provide much better predictions than either could provide individually. It turned out that the good model's all made pretty similar predictions.
But yes, it is cheating in that I knew the results as I was finding the best linear combination. In comparison, almost everything else I did was tuned using the period 2012-2014, so the 2016 predictions are actually true predictions for the other models.
Quote:
|
- Do you have any high-level takeaways for what you think will contribute to a better predictive model moving forward? (Assuming the 2017 game isn't super weird)
|
I will be looking more at making a better model over the next 2 weeks, and I will gladly share my insights there. One big takeaway that I didn't put in here was how "predictive" the calculated contribution to score model was when it knew team's contributions from the end of that event. That model had a Brier score of .1232, but it was clearly cheating because it knew the results of matches when it was making predictions. However, that value is important because I think it is a good approximation of an upper (lower? Brier scores are weird) bound on the predictive power of any predictive model. Alternatively, this value could be used to approximate the inherent uncertainty in all FRC matches.
My biggest takeaway though was best summarized by Richard Wallace above.