paper: FRC Elo 2008-2016

[Caleb Sykes]

Thread created automatically to discuss a document in CD-Media.

FRC Elo 2008-2016 by Caleb Sykes

[Caleb Sykes]

Sorry, I found a bug about 15 seconds after posting, I am uploading a revised version now.

[Caleb Sykes]

This workbook describes the Elo ratings of every team in FRC since 2008. Every match since 2008 is used, and the model predictions and results can be found in the year sheets. Team Elo ratings at the end of each season can be found in the "End of Season Elos" sheet. Average team Elo ratings for each season can be found in the "Average Elos" sheet. Detailed information about each team can be found in the "Team Lookup" sheet. To use the "team lookup" sheet, simply enter a team number into cell B2 and press the "Update" button.

My biggest takeaway from this whole endeavor was how incredibly dominant 1114 was during the period 2008-2011. After their first event in 2008, this model predicts them to win every single remaining match in 2008, every match in 2009, every match in 2010, and every match at Pittsburgh in 2011. Their end of season Elo in 2008 was 200 points higher than the next highest rated team.

I will be following up soon with a comparison of predictive models.

[artK]

Very nice tool you have here. 1114 is a fun team to watch the ELO for, which got quite high in 2010 until they threw the match.

Is it possible to adjust some of the parameters? Because the end of season reversion to the mean seams to small. 538's models for basketball and football have at least a 25% reversion to mean. 25% seems like a lower bound since each team loses a class of seniors and build a new robot each season, and the latter should really drives this model.

As an example of this, the highest ELO from 2016 was a 254 qual match at their first event, which is really a carryover from their 2015 season. But this might be inevitable in some cases, like 538 notes in their NBA model that teams with superstars like Bulls and Cavs maintained high ELOs for a while after Jordan and LeBron left.

[Caleb Sykes]

Originally Posted by artK

Very nice tool you have here. 1114 is a fun team to watch the ELO for, which got quite high in 2010 until they threw the match.

Is it possible to adjust some of the parameters? Because the end of season reversion to the mean seams to small. 538's models for basketball and football have at least a 25% reversion to mean. 25% seems like a lower bound since each team loses a class of seniors and build a new robot each season, and the latter should really drives this model.

As an example of this, the highest ELO from 2016 was a 254 qual match at their first event, which is really a carryover from their 2015 season. But this might be inevitable in some cases, like 538 notes in their NBA model that teams with superstars like Bulls and Cavs maintained high ELOs for a while after Jordan and LeBron left.

I chose the parameters I did based on what was the most predictive. I would have expected the mean reversion to be stronger, but 20% seemed to work the best. Here are the Brier scores for 2012-2014 for various mean reversion parameters.

100%	0.213986028
90%	0.210139727
80%	0.206626172
70%	0.203459364
60%	0.200667838
50%	0.198303146
40%	0.196450541
30%	0.19524134
20%	0.194865801
10%	0.195581409
0%	0.197702345

[artK]

Originally Posted by Caleb Sykes

I chose the parameters I did based on what was the most predictive. I would have expected the mean reversion to be stronger, but 20% seemed to work the best. Here are the Brier scores for 2012-2014 for various mean reversion parameters.

What about the Brier score for longer windows? Between 2012 and 2014, the mean only reverts twice, and the relative error (|a-b|/|b|) between the Brier scores for different parameters is less than .1 in even the most extreme cases. With more reversions, that parameter should effect the accuracy more extremely, giving a better parameter estimate.

[Caleb Sykes]

Originally Posted by artK

What about the Brier score for longer windows? Between 2012 and 2014, the mean only reverts twice, and the relative error (|a-b|/|b|) between the Brier scores for different parameters is less than .1 in even the most extreme cases. With more reversions, that parameter should effect the accuracy more extremely, giving a better parameter estimate.

2008-2016 quals and playoffs:

100%	0.201437771
90%	0.198323411
80%	0.195629645
70%	0.193352899
60%	0.191478127
50%	0.189986095
40%	0.188863874
30%	0.188124089
20%	0.187848762
10%	0.188296258
0%	0.190149139

[Caleb Sykes]

I ran the model for 2008-2016, but only took the Brier score for 2016.

100%	0.203023179
90%	0.199892169
80%	0.197203494
70%	0.19494991
60%	0.193105915
50%	0.191632998
40%	0.190483555
30%	0.189609862
20%	0.189008209
10%	0.188894837
0%	0.190274481

Interestingly, these results imply that 10% mean reversion would have been ever so slightly better than 20% for the 2015-2016 offseason. I don't want to draw any larger conclusins though since 2015 was a weird year.

[MARS_James]

This is honestly one of the coolest documents to look at, especially to look at teams elo when they have either gained or lost key mentors and seeing how it had a short or long term impact in comparison to the field.

[remulasce]

Hey, I appreciate you spending the time to put this together.

Question: Is there any way to use the team lookup, without having to purchase Excel? Google Docs obviously won't run the program. Neither will LibreOffice. The free Windows Modern (nee' Metro) app won't run the function, and I don't have access to Dreamspark any more.

Obviously, Excel is a powerful tool which is standard in many environments. But you're really limiting who can actually use your work if we need to pay MS a $150 entry fee to do so.

[Mark McLeod]

OpenOffice works with it.
Sorry, it was a later version of Excel.

[Caleb Sykes]

Okay, I spent a bunch of time looking at the mean-reversion parameter and the results are extremely interesting. First, I tried running every 2-year period individually and found the best mean reversion just for that period. Here were the results:

2008-2009	35%
2009-2010	40%
2010-2011	40%
2011-2012	30%
2012-2013	30%
2013-2014	35%
2014-2015	35%
2015-2016	35%

The mean reversion was pretty high and relatively constant for all years.

Next, I found the best mean reversion for 2009 given 2008. Then I found the best mean reversion for 2010 given 2008 and 2009, and so on. In this way, each year would have a distinct mean reversion that builds off of the previous mean reversions. Here were the results:

2008-2009	35%
2009-2010	35%
2010-2011	30%
2011-2012	20%
2012-2013	20%
2013-2014	25%
2014-2015	30%
2015-2016	25%

These values start high, as in the previous case, but they seem to drop after a while as the model learns more about the teams.

Finally, I compared how predictive the previous model was in comparison to my original 20% for all years, the results are attached.

Interestingly, adjusting the mean reversion every year actually fares worse overall than just using 20% every year, even if you throw out 2015 and 2016 because 2015 was an outlier year in many respects. I think the reason for this is because team performance 2 years in the future can still be reasonably well predicted by a current season's performance. The constantly updating model seems to put the mean reversion parameter too high to fully account for this 2 year explanatory effect.

[Caleb Sykes]

Originally Posted by Caleb Sykes

The results are attached.

Well, they would be if I could figure out how to attach them. I don't seem to have permission to add attachments, even though this is my thread.

[EricH]

Originally Posted by Caleb Sykes

Well, they would be if I could figure out how to attach them. I don't seem to have permission to add attachments, even though this is my thread.

Put 'em in the whitepaper's slot--you can attach multiple documents to one whitepaper. The Extra Discussion forum doesn't allow attachments (or deletions, generally speaking) due to some logic that I don't remember but makes sense from when I was informed about it.

[SoftwareBug2.0]

Originally Posted by remulasce

Hey, I appreciate you spending the time to put this together.

Question: Is there any way to use the team lookup, without having to purchase Excel? Google Docs obviously won't run the program. Neither will LibreOffice. The free Windows Modern (nee' Metro) app won't run the function, and I don't have access to Dreamspark any more.

Obviously, Excel is a powerful tool which is standard in many environments. But you're really limiting who can actually use your work if we need to pay MS a $150 entry fee to do so.

I don't know if this will make you feel any better, but I opened it with Excel and it didn't work and brought up a VB debugging window.