2016 Pre-Champs ELO Ratings

[Citrus Dad]

How do you account for the lack of scoring for breach and capture during quals? That's critical to comparisons for elims.

[logank013]

Originally Posted by Citrus Dad

How do you account for the lack of scoring for breach and capture during quals? That's critical to comparisons for elims.

Are you talking about the missed 25 and 20 point bonuses in the eliminations by not including the elims? Thanks.

[Chris is me]

Originally Posted by Citrus Dad

How do you account for the lack of scoring for breach and capture during quals? That's critical to comparisons for elims.

As stated in the OP, elims results are not accounted for in the model.

[wjordan]

Originally Posted by Citrus Dad

How do you account for the lack of scoring for breach and capture during quals? That's critical to comparisons for elims.

It doesn't account for elimination matches, but I've thought about adding the point bonuses for quals matches. I need to write some sort of prediction accuracy code to see if it's any more predictive.

[Richard Wallace]

One source of error in this system arises because non-district teams play fewer qualifying matches than district teams. More capable teams, such as 16, 254, 330, 971, 2481, etc. would need a few more matches for their Elo ratings to converge from the initial seed (1500) toward a figure that better represents their performance.

[wjordan]

Originally Posted by Richard Wallace

One source of error in this system arises because non-district teams play fewer qualifying matches than district teams. More capable teams, such as 16, 254, 330, 971, 2481, etc. would need a few more matches for their Elo ratings to converge from the initial seed (1500) toward a figure that better represents their performance.

This is a major flaw to the Elo system (at least when applied to FRC), and it's (as well as general normalizing between events) been something I've been trying to figure out for awhile. I was thinking about doing a straightforward multiplier to amount of Elo added for each match depending on the type of the event (i.e. District Qualifiers would add half as much Elo per match as a regional / DCMP would), but that would still massively favor teams that go to 3 regionals or 3+ district events. I also thought about normalizing the amount of Elo added per match by how many matches the team has played previously, but that would then disproportionately favor the teams that go to a single 8-play regional.

[Richard Wallace]

Originally Posted by wjordan

This is a major flaw to the Elo system, and it's (as well as general normalizing between events) been something I've been trying to figure out for awhile. I was thinking about doing a straightforward multiplier to amount of Elo added for each match depending on the type of the event (i.e. District Qualifiers would add half as much Elo per match as a regional / DCMP would), but that would still massively favor teams that go to 3 regionals or 3+ district events. I also thought about normalizing the amount of Elo added per match by how many matches the team has played previously, but that would then disproportionately favor the teams that go to a single 8-play regional.

What about running the Elo twice, using first pass results to seed the second pass?

[Carl C]

I have been developing my own Elo rating system for FRC over the past few years. The way it works differs from the one in this post enough that I thought it might by interesting to compare.

The data in my ratings are based on the history of each team dating back to 2002. At the end of each season, ratings are truncated 80% closer to the starting rating. Since 0 is the starting rating, a team with a score of 100 at the end of one season will begin the next season at 80. As this system uses matches from different games, it does not use win margins. A "K factor" of 32 is used in each match except in playoff matches, where the K factor is 16 (I too found that playoff matches seemed less predictive of future matches than qualification matches).

At the 2014 FIRST Championship, this system had a 0.190 Brier score, so it at least performs better than flipping a coin!

Anyways, enough methodology. Here are the current rankings:

Rank | Team   |  Rating
   1. frc254            448
   2. frc1519           421
   3. frc225            388
   4. frc1678           381
   5. frc195            373
   6. frc2481           368
   7. frc118            364
   8. frc987            345
   9. frc2056           344
  10. frc359            340
  11. frc1023           336
  12. frc1241           333
  13. frc1114           333
  14. frc525            329
  15. frc148            327
  16. frc1986           325
  17. frc330            323
  18. frc67             321
  19. frc133            312
  20. frc2767           310
  21. frc33             306
  22. frc3310           300
  23. frc16             299
  24. frc1806           299
  25. frc4564           295
  26. frc1501           292
  27. frc125            290
  28. frc3238           284
  29. frc5172           278
  30. frc971            278
  31. frc368            276
  32. frc494            275
  33. frc5254           270
  34. frc2122           269
  35. frc2771           268
  36. frc3130           267
  37. frc1619           266
  38. frc1024           266
  39. frc4334           264
  40. frc4469           263
  41. frc3683           263
  42. frc2974           262
  43. frc179            259
  44. frc3314           259
  45. frc27             259
  46. frc3230           256
  47. frc1540           254
  48. frc1318           254
  49. frc1983           254
  50. frc3990           253
  51. frc3339           253
  52. frc126            249
  53. frc2451           247
  54. frc107            245
  55. frc1730           245
  56. frc3604           244
  57. frc2067           243
  58. frc3824           243
  59. frc2468           243
  60. frc233            242
  61. frc2590           242
  62. frc4103           239
  63. frc180            238
  64. frc1918           237
  65. frc973            236
  66. frc1418           236
  67. frc4488           236
  68. frc25             236
  69. frc3688           235
  70. frc5050           235
  71. frc70             234
  72. frc2883           232
  73. frc1261           232
  74. frc2848           232
  75. frc341            229
  76. frc1296           228
  77. frc1746           227
  78. frc177            227
  79. frc868            226
  80. frc4039           226
  81. frc3937           226
  82. frc744            225
  83. frc1717           222
  84. frc2614           222
  85. frc4188           221
  86. frc85             220
  87. frc2137           220
  88. frc3309           219
  89. frc217            219
  90. frc1065           219
  91. frc1425           218
  92. frc4967           218
  93. frc836            218
  94. frc1126           217
  95. frc1836           216
  96. frc2471           216
  97. frc3255           215
  98. frc2338           215
  99. frc231            214
 100. frc4003           214

I will try to post the source code and some of the research I used to develop this system at some point after finals.

EDIT: Here are the full rankings if anyone is interested.

[wjordan]

Someone pointed out an error in the script I posted, which made the ratings slightly incorrect. I fixed the error and updated the rankings in the OP.

As for predictive power, this set of rankings had a Brier score of 0.155.

[Richard Wallace]

Originally Posted by Carl C

...ratings are based on the history of each team dating back to 2002 ...

83. frc1717 222

...

I can see at least one reason that "past performance does not guarantee future results", as the investment prospectus always says.