I have just published an update to my Elo model titled “FRC_Elo_2005-2017.” This model incorporates a couple of major changes to the Elo model I used during the 2017 season, and this will be the model I use for my 2018 predictions. The key changes I have made are outlined below, along with the reasoning for the changes. I still have a couple of ideas for things I would have liked to try to add, but I’ll have to get to them next year, because I want to start attempting to improve calculated contributions. My priorities for this model are, in rough order from most important to least important:
Making as few assumptions as possible, I want this model to work for future games, which means I don’t want to assume things that may not be true for those games
Maximizing predictive power for qual matches for the period 2012-2014
Making the ratings easy to understand
Making the ratings from different seasons directly comparable to each other
Here are the key changes I made to the model in comparison to the 2017 version:
- I have shifted all team Elo ratings up by approximately 70 points relative to my 2017 Elo model. This change in isolation does not actually change any predictions, since only Elo differences are used in predictions, not their absolute values. This does however, both make the ratings easier to interpret as well as make the ratings more comparable to other Elo rating systems. With this change, the long-run average Elo will now be approximately 1500 points, although the average Elo in any season may vary between about 1490 to about 1510.
While looking at other Elo rating systems, I noticed that nearly all of them either set 1500 as the long-run average rating (as in 538’s ratings), or set all new entries into the system with a rating of 1500 (as in FIBS ratings). My previous Elo model did neither of these, as the previous rookie starting rating was 1350, and the 2017 average Elo was ~1440. The other advantage to this change is that there is now a (relatively) solid reference point for how good a team is, >1500 = above average, and <1500 = below average. Additionally, I have found that a rating of 1600 is the rating of a 90th (±3%) percentile team in almost every season. One additional reference point for these ratings is that the average Elo rating of rookie teams is 1450.
I have extended my Elo ratings back in time to 2005. When I originally built my Elo model, I did not include any years prior to 2008 since some 2007 events contained no match data. After reviewing the years 2005-2007, I have decided to include them in my ratings, although I advise caution when looking at Elo ratings from this period. There are some unfortunate teams that competed in this period but do not have any rating because none of their matches were recorded. I decided to include these years because, even though the data are incomplete, including 2007 does improve 2008 predictions, and likewise for 2006 and 2005. In the future, it is possible that I may extend the ratings back as far as 2002, but that would require some larger changes to my model since these years had 2v2 matches and not 3v3 matches.
I set all team’s starting Elos during the first year of the model (2005) to 1500 instead of the normal starting Elo of 1450. This change stabilizes the average Elo rating for future seasons around 1500. This stabilization actually improves predictive power for all future seasons, although this improvement is not drastic for years other than 2006 and 2007. This is likely because this change makes rookie ratings consistently 50 points below the average, instead of varying depending on the year. This change also allows for more direct comparison of Elos between seasons, although Elos above 1600 will need to be interpreted differently depending on the season, since each season has a different distribution of high-level Elos.
In my previous model, I would find each team’s start of season Elo by taking their previous season’s Elo and reverting it 20% toward the mean. I have found though that taking each team’s start of season Elo to be a weighted average of their previous two season’s end of season Elo ratings provides a much better predictor of their next season’s performance. Specifically, in this model, a team’s start of season Elo rating is found by taking 70% of their previous season’s rating plus 30% of their end of season rating from two seasons ago, and then reverting this weighted average by 20% toward the pseudo-mean (1550). This change provides substantial predictive power improvement, particularly at the start of the season.
Due to all of the aforementioned changes, some of my model’s other parameters have new optimal values. I have already mentioned that I am now starting rookies with a rating of 1450 instead of 1350 previously, and that the new pseudo-mean will be 1550 instead of 1500. These two changes were mostly due to the shifting up of ratings (change 1), although the other changes may have made a small effect. In addition to these, my new model has lower values for both kQuals and kPlayoffs. kQuals moved from 15 under the previous model to 12 under the current model. This means that team ratings will change 20% more slowly in response to qual match performances. kPlayoffs moved from 5 under the previous model to 3 under the current model. This means that team ratings will change 40% more slowly in response to playoff match performances. I expect that these changes are largely due to the improved start of season Elo ratings. If the start of season Elo ratings are better, it means that the model doesn’t need to respond as quickly to performance at the start of events.
In addition to the above 5 changes to the Elo model itself, I have also made a few aesthetic changes to the workbook.
- I have added sheets for the years 2005-2007
- Just for fun, I added in the ability to directly compare teams in the “Team Lookup” sheet. To do this, enter a team number into cell B2. To view only a single team, leave cell B2 blank. The second team will show up on the graph with “+” markers.
- Since the graph can get difficult to see with two teams over 13 years, I have also added a range setting in the “Team Lookup” sheet. Just enter your desired year range into cells B3 and B4 and click “Update Range” after importing your desired teams. I had to work out a few bugs for this macro, and I might not have hit them all, so if you notice anything weird let me know.
- I have updated the “Instructions and FAQ” sheet to cover much of the information above.