paper: Comparison of Statistical Prediction Models

[Richard Wallace]

Nice work! Please share results, when you can, on your effort to improve predictions using a weighted combination of OPR and ELO.

For now, we can all continue to curse OPR as the worst way to rank teams, except for all the other ways.

[Eugene Fang]

I have a couple of questions after looking at it more deeply.

- Isn't best linear combination kind of cheating? It seems like you're picking weights after knowing the outcome.
- Do you have any high-level takeaways for what you think will contribute to a better predictive model moving forward? (Assuming the 2017 game isn't super weird)

Cool stuff overall, thanks for sharing!

[Caleb Sykes]

Originally Posted by Eugene Fang

- Isn't best linear combination kind of cheating? It seems like you're picking weights after knowing the outcome.

Essentially, yes, it is cheating. The linear combination wasn't a particularly important part of this effort, I kind of tacked it on at the end. My interest in the linear combination is twofold:
I wanted to know how much "better" the models could get by adding two or more models together. The result was that it has reasonable, but not extremely high value, so it is worth more investigation.
I wanted to know how "different" each model's predictions were to each other. If two models both have reasonable predictive power, and their correct predictions are uncorrelated, taking a combination of the two will provide much better predictions than either could provide individually. It turned out that the good model's all made pretty similar predictions.

But yes, it is cheating in that I knew the results as I was finding the best linear combination. In comparison, almost everything else I did was tuned using the period 2012-2014, so the 2016 predictions are actually true predictions for the other models.

- Do you have any high-level takeaways for what you think will contribute to a better predictive model moving forward? (Assuming the 2017 game isn't super weird)

I will be looking more at making a better model over the next 2 weeks, and I will gladly share my insights there. One big takeaway that I didn't put in here was how "predictive" the calculated contribution to score model was when it knew team's contributions from the end of that event. That model had a Brier score of .1232, but it was clearly cheating because it knew the results of matches when it was making predictions. However, that value is important because I think it is a good approximation of an upper (lower? Brier scores are weird) bound on the predictive power of any predictive model. Alternatively, this value could be used to approximate the inherent uncertainty in all FRC matches.

My biggest takeaway though was best summarized by Richard Wallace above.

[Ether]

Originally Posted by Caleb Sykes

I will be looking more at making a better model over the next 2 weeks

If you haven't read the following thread in a while, you might want to revisit it and see if any of wgardner's suggestions might be worth comparing to your other methods:

https://www.chiefdelphi.com/forums/s....php?p=1483963

Also, take a look at Eugene's very interesting work here:

https://github.com/the-blue-alliance...ment-210564302

[Caleb Sykes]

Originally Posted by Ether

If you haven't read the following thread in a while, you might want to revisit it and see if any of wgardner's suggestions might be worth comparing to your other methods:

https://www.chiefdelphi.com/forums/s....php?p=1483963

I had totally forgotten about this thread, I'll look more into this.

Also, take a look at Eugene's very interesting work here:

https://github.com/the-blue-alliance...ment-210564302

Yep, I used this method extensively for each of the "calculated contribution" models.