OCCRA
Go to Post I love you ChiefDelphi times! - Amanda Morrison [more]
Home
Go Back   Chief Delphi > ChiefDelphi.com Website > Extra Discussion
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #31   Spotlight this post!  
Unread 06-30-2017, 10:28 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Elo was very well calibrated in 2017, as it has been in previous years: http://imgur.com/a/cYBKS

That was all 2017 matches, quals and playoffs. Total Brier score = 0.2114.
Reply With Quote
  #32   Spotlight this post!  
Unread 08-01-2017, 02:28 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Updated to include FoC.

Basically the only substantial change is that 1676 went down 20 points and 862 went up 20 points.
Reply With Quote
  #33   Spotlight this post!  
Unread 11-19-2017, 07:56 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

I have just published an update to my Elo model titled "FRC_Elo_2005-2017." This model incorporates a couple of major changes to the Elo model I used during the 2017 season, and this will be the model I use for my 2018 predictions. The key changes I have made are outlined below, along with the reasoning for the changes. I still have a couple of ideas for things I would have liked to try to add, but I'll have to get to them next year, because I want to start attempting to improve calculated contributions. My priorities for this model are, in rough order from most important to least important:
Making as few assumptions as possible, I want this model to work for future games, which means I don't want to assume things that may not be true for those games
Maximizing predictive power for qual matches for the period 2012-2014
Making the ratings easy to understand
Making the ratings from different seasons directly comparable to each other

Here are the key changes I made to the model in comparison to the 2017 version:
1. I have shifted all team Elo ratings up by approximately 70 points relative to my 2017 Elo model. This change in isolation does not actually change any predictions, since only Elo differences are used in predictions, not their absolute values. This does however, both make the ratings easier to interpret as well as make the ratings more comparable to other Elo rating systems. With this change, the long-run average Elo will now be approximately 1500 points, although the average Elo in any season may vary between about 1490 to about 1510.

While looking at other Elo rating systems, I noticed that nearly all of them either set 1500 as the long-run average rating (as in 538's ratings), or set all new entries into the system with a rating of 1500 (as in FIBS ratings). My previous Elo model did neither of these, as the previous rookie starting rating was 1350, and the 2017 average Elo was ~1440. The other advantage to this change is that there is now a (relatively) solid reference point for how good a team is, >1500 = above average, and <1500 = below average. Additionally, I have found that a rating of 1600 is the rating of a 90th (+-3%) percentile team in almost every season. One additional reference point for these ratings is that the average Elo rating of rookie teams is 1450.

2. I have extended my Elo ratings back in time to 2005. When I originally built my Elo model, I did not include any years prior to 2008 since some 2007 events contained no match data. After reviewing the years 2005-2007, I have decided to include them in my ratings, although I advise caution when looking at Elo ratings from this period. There are some unfortunate teams that competed in this period but do not have any rating because none of their matches were recorded. I decided to include these years because, even though the data are incomplete, including 2007 does improve 2008 predictions, and likewise for 2006 and 2005. In the future, it is possible that I may extend the ratings back as far as 2002, but that would require some larger changes to my model since these years had 2v2 matches and not 3v3 matches.

3. I set all team's starting Elos during the first year of the model (2005) to 1500 instead of the normal starting Elo of 1450. This change stabilizes the average Elo rating for future seasons around 1500. This stabilization actually improves predictive power for all future seasons, although this improvement is not drastic for years other than 2006 and 2007. This is likely because this change makes rookie ratings consistently 50 points below the average, instead of varying depending on the year. This change also allows for more direct comparison of Elos between seasons, although Elos above 1600 will need to be interpreted differently depending on the season, since each season has a different distribution of high-level Elos.

4. In my previous model, I would find each team's start of season Elo by taking their previous season's Elo and reverting it 20% toward the mean. I have found though that taking each team's start of season Elo to be a weighted average of their previous two season's end of season Elo ratings provides a much better predictor of their next season's performance. Specifically, in this model, a team's start of season Elo rating is found by taking 70% of their previous season's rating plus 30% of their end of season rating from two seasons ago, and then reverting this weighted average by 20% toward the pseudo-mean (1550). This change provides substantial predictive power improvement, particularly at the start of the season.

5. Due to all of the aforementioned changes, some of my model's other parameters have new optimal values. I have already mentioned that I am now starting rookies with a rating of 1450 instead of 1350 previously, and that the new pseudo-mean will be 1550 instead of 1500. These two changes were mostly due to the shifting up of ratings (change 1), although the other changes may have made a small effect. In addition to these, my new model has lower values for both kQuals and kPlayoffs. kQuals moved from 15 under the previous model to 12 under the current model. This means that team ratings will change 20% more slowly in response to qual match performances. kPlayoffs moved from 5 under the previous model to 3 under the current model. This means that team ratings will change 40% more slowly in response to playoff match performances. I expect that these changes are largely due to the improved start of season Elo ratings. If the start of season Elo ratings are better, it means that the model doesn't need to respond as quickly to performance at the start of events.


In addition to the above 5 changes to the Elo model itself, I have also made a few aesthetic changes to the workbook.
1. I have added sheets for the years 2005-2007
2. Just for fun, I added in the ability to directly compare teams in the "Team Lookup" sheet. To do this, enter a team number into cell B2. To view only a single team, leave cell B2 blank. The second team will show up on the graph with "+" markers.
3. Since the graph can get difficult to see with two teams over 13 years, I have also added a range setting in the "Team Lookup" sheet. Just enter your desired year range into cells B3 and B4 and click "Update Range" after importing your desired teams. I had to work out a few bugs for this macro, and I might not have hit them all, so if you notice anything weird let me know.
4. I have updated the "Instructions and FAQ" sheet to cover much of the information above.
Reply With Quote
  #34   Spotlight this post!  
Unread 04-29-2018, 09:38 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

I have added an update which includes 2018 data. I also changed the "Team Lookup" sheet to no longer import matches which are missing score data. This should make the graphs nicer for teams like 2590. Thanks to AGPapa for identifying this problem.

I will very likely be making changes to this model before 2019, but here are the 2018 end of season ratings:
Code:
Team	Elo
254	2032
2056	2015
1678	1995
2046	1955
694	1952
1323	1943
118	1938
1114	1930
2910	1930
148	1929
2481	1928
2590	1924
195	1923
2767	1917
3309	1907
842	1895
3357	1893
4539	1892
125	1886
225	1884
3310	1883
5406	1879
133	1875
27	1874
217	1874
1241	1874
1574	1874
1538	1871
230	1867
1629	1864
1619	1862
4613	1859
1796	1858
624	1856
3707	1856
319	1856
2122	1855
1747	1854
3478	1852
3130	1848
5050	1846
2337	1844
2471	1842
4003	1839
176	1838
85	1838
3538	1837
340	1836
971	1836
2478	1834
5172	1833
1318	1832
141	1830
25	1830
16	1830
399	1829
1690	1826
1325	1821
180	1820
3005	1820
1519	1816
1706	1816
2791	1815
3452	1813
3476	1813
4910	1811
67	1810
291	1810
179	1808
5460	1807
5190	1806
3339	1806
2168	1805
4917	1805
4488	1804
2614	1803
1986	1800
1806	1800
1712	1799
3959	1799
494	1799
1918	1796
1640	1795
868	1795
4028	1794
2642	1792
3646	1792
4362	1791
1756	1790
2059	1788
5818	1788
3512	1787
330	1785
5987	1784
987	1783
177	1781
4237	1781
1533	1779
2403	1779
364	1779
359	1777
56	1775
1425	1771
234	1771
1648	1770
1259	1769
269	1768
1731	1765
2557	1764
910	1764
4618	1763
302	1763
610	1762
4476	1761
2834	1760
865	1760
5895	1759
5883	1759
1102	1759
3539	1759
3937	1758
1876	1756
1024	1755
3098	1755
1506	1754
33	1753
4513	1752
525	1752
233	1752
365	1752
6329	1751
5687	1749
5472	1749
368	1748
1741	1748
341	1747
5985	1747
4564	1747
3641	1745
973	1743
2522	1742
384	1742
5572	1742
5199	1742
1622	1741
1726	1741
1923	1741
2877	1740
2052	1740
95	1739
2630	1738
4522	1736
846	1735
6763	1735
3316	1735
1723	1734
4451	1734
3802	1734
2485	1734
6705	1733
2013	1733
3674	1731
3990	1729
832	1729
2451	1729
4678	1729
870	1728
3128	1728
4265	1727
3250	1727
175	1727
1023	1726
1736	1726
1305	1726
772	1725
4946	1724
4253	1723
744	1723
2130	1723
2659	1723
71	1722
294	1721
687	1721
2976	1721
503	1720
1577	1719
1025	1719
4004	1718
346	1718
957	1718
2502	1717
3015	1717
2992	1717
188	1717
1414	1716
78	1716
4911	1716
5803	1716
1836	1715
379	1715
488	1713
20	1713
1492	1712
604	1712
3546	1711
1339	1711
1684	1711
3616	1711
2410	1710
1391	1710
4855	1710
2771	1710
1983	1709
5012	1709
2996	1708
5817	1707
4635	1706
4188	1706
3255	1706
5505	1706
1197	1704
4146	1704
1156	1704
1100	1704
2175	1704
4944	1703
6090	1702
2706	1701
5254	1701
1730	1701
2655	1701
2054	1701
1262	1701
3683	1701
4272	1701
1218	1701
4776	1700
3663	1700
5046	1700
4405	1700
4655	1699
2826	1699
6579	1699
2383	1699
5584	1699
3656	1698
66	1698
3536	1698
6672	1697
4980	1697
862	1697
1011	1696
3750	1696
2708	1696
639	1695
88	1695
4587	1695
1559	1694
1065	1694
456	1693
4541	1692
4795	1691
876	1691
4063	1690
422	1690
103	1690
5802	1689
2194	1689
4290	1688
1421	1688
3824	1688
1322	1688
6077	1688
1599	1688
3542	1688
649	1687
4143	1687
1073	1687
1477	1687
2075	1687
5501	1686
70	1686
5152	1686
1772	1686
3197	1685
4388	1685
6800	1685
3314	1684
314	1684
303	1683
2556	1683
3620	1682
2974	1682
1746	1682
4533	1682
3328	1681
3547	1681
3534	1681
3276	1681
5567	1680
2200	1680
1744	1679
968	1679
4145	1679
7021	1678
6025	1678
3847	1678
5010	1677
5053	1677
888	1677
597	1677
930	1676
1360	1676
6886	1675
2990	1675
2202	1675
316	1675
5603	1674
6411	1673
237	1673
2848	1673
5026	1672
5986	1672
3940	1672
4550	1672
1189	1672
4334	1671
3929	1671
4213	1671
6201	1671
5404	1671
1296	1671
834	1670
6429	1669
2338	1669
1507	1669
6502	1669
5813	1669
4561	1669
115	1669
6418	1668
4087	1668
4908	1668
3489	1667
2783	1667
5663	1666
4239	1666
1572	1666
3835	1665
5530	1665
2137	1665
2474	1664
1987	1664
245	1663
6474	1662
333	1662
238	1662
4189	1662
5842	1662
342	1661
3374	1661
4959	1661
2930	1661
2682	1661
1729	1661
5036	1661
5554	1660
2102	1660
1902	1660
3695	1660
3223	1660
1018	1660
5314	1660
2987	1658
3230	1658
167	1657
386	1657
5889	1657
578	1657
2960	1657
708	1657
1718	1656
3481	1656
3853	1656
61	1655
2231	1655
6328	1654
3236	1654
2468	1654
343	1654
3461	1653
2658	1653
58	1652
4469	1652
5155	1652
1710	1652
3667	1652
2363	1651
126	1651
4361	1651
2811	1651
2144	1651
1792	1651
6334	1650
2486	1650
1625	1650
2521	1650
5431	1650
1319	1650
5675	1650
5013	1650
5436	1650
2199	1649
1816	1649
670	1649
2377	1649
2710	1649
4967	1648
5420	1648
59	1648
5576	1648
75	1648
6933	1647
3770	1647
2765	1647
418	1646
226	1646
1885	1646
358	1645
1255	1645
3164	1644
135	1644
4391	1644
3711	1644
836	1644
5544	1644
3737	1643
1418	1643
7179	1643
5285	1643
3218	1643
1058	1642
3171	1642
3256	1642
3140	1642
4818	1642
3623	1642
4976	1642
236	1641
1306	1641
383	1641
3244	1641
612	1641
120	1640
4525	1640
4027	1640
3997	1640
3179	1640
4392	1640
829	1639
6117	1639
4531	1639
2851	1639
5114	1638
4039	1638
4198	1638
4026	1638
2601	1638
3010	1638
4786	1638
5614	1637
360	1636
533	1636
447	1636
4276	1636
2152	1635
1071	1635
433	1634
5434	1634
4415	1633
63	1633
2959	1633
7039	1633
1523	1633
287	1633
348	1632
5234	1632
5517	1632
3931	1632
4499	1631
1540	1631
5596	1630
1817	1630
5453	1630
3260	1630
5528	1630
614	1630
4623	1629
4041	1629
623	1629
4639	1628
3242	1628
2537	1628
2359	1628
3767	1628
2702	1628
5338	1628
395	1628
6121	1628
5216	1628
5499	1627
2611	1627
3459	1627
527	1627
4085	1627
3492	1627
3419	1627
3792	1626
3100	1626
5509	1626
2823	1626
3604	1626
6104	1626
5099	1626
1178	1626
159	1625
2164	1625
469	1625
3313	1625
4607	1624
999	1624
2846	1624
4653	1624
6548	1624
4930	1623
8	1623
5407	1623
3196	1623
5546	1623
5945	1623
4557	1623
5846	1622
2977	1622
3970	1622
7225	1622
6406	1622
1251	1622
1778	1622
6193	1622
4216	1622
1317	1621
4472	1621
5823	1621
3360	1621
4586	1621
1596	1621
3986	1621
1592	1621
1683	1620
5193	1620
4020	1620
467	1620
6907	1620
5188	1620
1002	1620
1410	1619
3574	1619
93	1619
997	1619
6378	1619
1939	1619
4450	1618
2170	1618
4061	1618
1626	1618
323	1618
3535	1618
4183	1617
3668	1617
5654	1617
2283	1617
4984	1617
4965	1617
3042	1617
6647	1616
4325	1616
2040	1616
1126	1616
1721	1616
5674	1616
2145	1616
3555	1616
122	1616
4079	1615
4646	1615
222	1615
5427	1615
3184	1615
3603	1615
5437	1615
663	1614
3635	1614
1551	1614
156	1614
1511	1614
181	1614
2342	1614
3277	1614
3928	1614
21	1613
1699	1613
1501	1613
4230	1613
1595	1613
2181	1613
3026	1613
1675	1613
1108	1613
3572	1613
3883	1612
3161	1612
3410	1612
1982	1612
1708	1612
1768	1611
4468	1611
1310	1611
4096	1610
5414	1610
5990	1609
2928	1609
5763	1609
2509	1608
3647	1608
4983	1608
207	1608
4611	1608
991	1608
191	1608
6493	1607
1099	1607
4688	1607
4256	1607
2227	1607
2609	1607
2352	1606
4181	1606
4077	1606
2225	1606
201	1606
1658	1606
354	1605
3137	1605
703	1605
2016	1605
5712	1605
2090	1604
1	1604
4904	1604
1825	1603
558	1603
51	1603
1311	1603
5740	1603
6055	1602
2607	1602
6831	1602
4920	1602
3034	1602
1732	1602
747	1601
5587	1601
3239	1601
4909	1600
23	1600
5030	1600
2579	1600
2491	1600
6508	1600
1208	1600
3039	1599
3838	1599
3072	1599
228	1599
1711	1599
5804	1599
5976	1598
1807	1597
2883	1597
2526	1597
5102	1597
5735	1597
6415	1597
5550	1597
1745	1596
2512	1596
2445	1596
4159	1596
3472	1596
5618	1596
3284	1596
1657	1595
86	1595
2594	1595
4069	1595
3882	1594
6803	1594
6424	1594
2638	1594
2903	1593
308	1593
5914	1593
3844	1592
907	1592
4192	1592
353	1592
7048	1592
131	1592
4738	1591
2180	1591
3414	1591
847	1591
2386	1591
4779	1591
4176	1591
263	1591
4774	1591
1553	1590
1720	1590
948	1590
611	1590
5148	1589
6386	1589
5910	1589
2657	1589
223	1589
144	1589
2643	1589
3637	1588
6829	1588
3038	1588
3965	1587
2457	1587
4150	1587
1405	1586
3526	1586
1764	1586
3506	1586
173	1586
2539	1586
5524	1586
5424	1586
6978	1586
1676	1586
4638	1586
4400	1586
2832	1585
4905	1585
5150	1585
5907	1585
5462	1585
2333	1585
401	1584
5039	1584
5205	1584
123	1584
2584	1584
3044	1584
1257	1583
5496	1583
4103	1583
1369	1583
3863	1583
5843	1583
4	1583
5492	1583
5826	1582
3550	1582
4055	1582
1557	1582
4512	1582
5410	1582
580	1582
5291	1582
949	1582
2147	1582
5006	1582
111	1581
3293	1581
4970	1581
554	1581
3996	1581
4458	1581
6445	1581
3618	1581
4961	1581
5590	1581
3258	1580
3880	1580
1072	1580
5442	1580
1989	1579
102	1579
3581	1579
4819	1579
1014	1579
696	1579
4342	1579
4401	1578
3009	1578
1610	1578
5458	1578
2543	1578
6359	1578
5000	1578
6072	1578
5653	1577
3999	1577
108	1577
4521	1577
2106	1577
3602	1577
7160	1577
3932	1576
5801	1576
2508	1576
2252	1576
2635	1576
107	1576
6880	1576
5145	1575
5951	1575
457	1575
818	1575
6517	1575
3624	1575
6357	1574
2907	1574
1481	1574
6100	1574
79	1574
219	1574
4409	1574
4130	1574
6343	1574
470	1574
4584	1573
5086	1573
3354	1573
1671	1573
6615	1573
190	1572
4508	1572
2472	1572
6027	1572
1795	1572
2395	1572
2839	1572
1518	1572
7034	1572
2169	1572
4859	1571
498	1571
1038	1571
6618	1571
1243	1570
4174	1570
601	1570
5805	1570
7154	1570
279	1570
5449	1570
2869	1570
1250	1570
5847	1569
1155	1569
2733	1569
4376	1569
5278	1569
2501	1569
6177	1569
3633	1569
668	1569
4956	1568
4698	1568
894	1568
1287	1568
1937	1568
6112	1568
7200	1568
4355	1568
548	1567
6352	1567
5811	1567
4941	1567
3008	1567
5612	1567
1051	1567
3676	1567
1880	1567
3324	1567
2073	1566
2830	1566
2875	1566
3490	1566
2067	1566
801	1566
6200	1566
1701	1566
6390	1566
3379	1566
2264	1565
3075	1565
3480	1565
7127	1565
1860	1565
5222	1565
5950	1565
4048	1565
620	1565
5948	1565
203	1565
4384	1565
568	1565
3691	1565
5638	1565
4296	1565
3840	1564
2220	1564
5724	1564
1086	1564
4462	1564
5819	1564
6224	1564
6753	1564
3749	1563
871	1563
5535	1563
2081	1562
5480	1562
4131	1562
4241	1562
5242	1562
1089	1562
6394	1562
145	1561
4322	1561
4122	1561
3397	1561
5522	1561
4206	1561
3335	1561
6110	1561
369	1561
3793	1561
292	1561
3003	1561
4931	1561
4014	1560
6621	1560
6009	1560
4495	1560
812	1560
281	1560
5439	1560
4814	1560
2084	1559
1868	1559
1094	1559
7329	1559
192	1559
1245	1559
4775	1559
3593	1559
835	1559
4972	1558
5930	1558
1495	1558
2158	1558
4994	1558
4381	1558
1466	1558
3926	1558
839	1558
74	1558
5676	1558
5137	1558
5015	1558
3653	1558
5940	1558
4175	1558
6582	1558
5561	1558
5045	1557
571	1557
6510	1557
4903	1557
4565	1557
7178	1557
337	1557
1413	1557
5822	1557
3504	1557
967	1557
5429	1557
41	1557
702	1557
3274	1557
3082	1557
4537	1556
5523	1556
60	1556
2503	1556
6479	1556
4990	1556
1444	1556
4470	1556
5179	1556
4603	1556
4207	1556
5690	1555
441	1555
5317	1555
4656	1555
3755	1555
5507	1555
4486	1555
6377	1555
5052	1555
5450	1555
4242	1554
3473	1554
3753	1554
2586	1554
3418	1554
5464	1554
1522	1554
2648	1554
4050	1554
2370	1554
6022	1554
48	1554
5468	1553
6996	1553
4459	1553
3322	1553
2980	1553
5913	1553
3630	1552
2415	1552
3679	1552
1289	1552
3953	1551
1279	1551
2605	1551
2191	1551
272	1551
6869	1551
5608	1550
1662	1550
2465	1550
7250	1550
5426	1550
231	1550
5810	1550
1248	1550
3176	1550
3330	1550
6613	1550
6172	1549
7079	1549
5413	1549
3299	1549
4089	1549
3655	1549
4075	1549
6237	1548
2587	1548
229	1548
5635	1548
2534	1548
2836	1548
5997	1548
5124	1548
3998	1548
6131	1548
3692	1547
1468	1547
2085	1547
3386	1547
7140	1547
4293	1547
3495	1547
5204	1547
3826	1546
509	1546
1285	1546
6941	1546
1561	1546
5839	1546
4582	1546
1388	1546
5599	1546
2890	1546
3735	1546
7109	1546
2582	1546
5563	1546
6630	1546
5892	1546
4915	1545
2412	1545
2341	1545
4121	1545
293	1545
815	1545
6217	1545
4509	1545
3958	1544
2035	1544
4936	1544
6550	1544
4918	1544
5467	1544
2606	1544
6024	1544
6945	1544
3341	1544
5403	1543
6802	1543
6043	1543
1895	1543
2679	1543
2612	1543
3238	1543
5331	1543
3925	1543
2062	1542
4375	1542
3658	1542
3501	1542
4750	1542
3021	1542
5547	1542
1124	1542
3814	1542
3660	1541
157	1541
5901	1541
6988	1541
3556	1541
4201	1541
5702	1541
2637	1540
5623	1540
4737	1540
2518	1540
68	1540
2028	1540
5866	1540
3294	1540
5860	1539
3522	1539
1750	1539
599	1539
3688	1538
2405	1538
716	1538
461	1538
6344	1538
2989	1538
329	1538
5588	1537
5451	1537
250	1537
4991	1537
6434	1537
2449	1537
1280	1537
5903	1537
2549	1537
7187	1536
5511	1536
5577	1536
4669	1536
5299	1536
935	1536
3389	1536
4330	1536
2729	1536
2972	1536
4306	1536
6951	1536
3966	1536
6544	1536
3200	1536
6317	1536
2354	1536
5658	1535
7117	1535
3345	1535
1810	1535
3024	1535
6836	1535
2143	1535
6465	1535
1661	1535
114	1535
6353	1535
1225	1535
3777	1535
2212	1535
5224	1534
178	1534
858	1534
3355	1534
4485	1534
3219	1534
3275	1534
288	1534
1884	1534
3951	1534
4481	1533
4523	1533
7054	1533
5899	1533
4810	1533
2221	1533
5232	1533
5504	1533
4906	1532
7074	1532
6909	1532
3494	1532
4264	1532
6020	1531
271	1531
1266	1531
4395	1531
7162	1531
1777	1531
6852	1531
3570	1531
4769	1531
5109	1531
5096	1531
6052	1531
1261	1530
5952	1530
5508	1530
6500	1530
5112	1530
4010	1530
6919	1530
3991	1530
5696	1530
977	1530
7026	1530
4226	1530
492	1530
6421	1529
5182	1529
1763	1529
6417	1529
1228	1529
6153	1529
4125	1529
5422	1529
1188	1529
4939	1529
6573	1529
2797	1528
2172	1528
4592	1528
4680	1528
5181	1528
931	1528
1997	1528
501	1528
6908	1528
101	1528
3303	1528
213	1527
2183	1527
4035	1527
1246	1527
3381	1527
7051	1527
4758	1527
6323	1527
5982	1526
3760	1526
2335	1526
5553	1526
3875	1526
6823	1526
4778	1526
4343	1526
3247	1526
5502	1526
1164	1526
6722	1526
4947	1526
3405	1525
174	1525
2373	1525
3950	1525
378	1525
6483	1525
2204	1525
4619	1525
1153	1525
6300	1525
4952	1525
5533	1525
3145	1525
4356	1525
246	1525
6471	1524
6696	1524
6366	1524
138	1524
6591	1524
2827	1524
2583	1524
7213	1524
5980	1524
540	1524
4263	1523
2662	1523
6340	1523
514	1523
5632	1523
900	1523
1510	1523
4845	1523
5084	1523
6824	1523
6174	1523
1138	1523
2619	1523
6957	1522
2647	1522
4624	1522
6122	1522
6070	1522
1584	1522
3216	1522
1539	1522
5560	1522
5230	1522
932	1522
7231	1522
4546	1521
1660	1521
6085	1521
6225	1521
2340	1521
2973	1521
6843	1521
4935	1520
980	1520
852	1520
296	1520
6936	1520
4711	1520
5428	1520
2141	1520
2186	1520
11	1520
691	1520
2626	1520
3588	1520
1493	1520
5686	1520
5775	1519
5669	1519
6443	1519
3158	1519
1797	1519
4011	1519
1160	1519
2438	1519
6003	1519
7287	1519
4536	1519
3146	1518
587	1518
4246	1518
1569	1518
6717	1518
3488	1518
3487	1518
5171	1518
484	1518
3871	1518
4320	1518
247	1518
449	1518
2795	1517
6827	1517
4001	1517
2487	1517
7111	1517
5715	1517
2165	1517
4099	1517
4664	1517
6420	1517
2656	1517
3484	1517
4735	1517
4160	1516
5212	1516
6451	1516
3560	1516
2353	1516
4191	1516
172	1516
4135	1516
2053	1516
224	1516
5419	1516
6628	1516
4731	1516
1775	1516
94	1516
2135	1515
2852	1515
5209	1515
4209	1515
3598	1515
2560	1515
5059	1515
7110	1515
7169	1515
7239	1514
2375	1514
3267	1514
2789	1514
1091	1514
811	1514
4064	1514
486	1514
2022	1514
4060	1514
5461	1514
1967	1514
4073	1514
3018	1514
6155	1513
121	1513
5409	1513
1111	1513
6361	1513
5176	1513
4534	1513
4398	1513
2805	1513
2849	1513
6590	1513
3011	1513
2867	1513
547	1513
5856	1513
4632	1512
569	1512
662	1512
6484	1512
2425	1512
7157	1512
5276	1512
4728	1511
3511	1511
4999	1511
5827	1511
1389	1511
6758	1511
677	1511
6856	1511
3160	1511
5160	1511
1635	1511
3206	1511
6878	1511
5526	1510
1351	1510
7144	1510
5032	1510
116	1509
3337	1509
6490	1509
6813	1509
5104	1509
6336	1509
6026	1509
3117	1509
2876	1509
5024	1509
5809	1508
2421	1508
2559	1508
6442	1508
5736	1508
573	1508
5927	1508
2230	1508
3627	1508
3933	1508
3606	1508
7211	1507
28	1507
4091	1507
709	1507
6401	1507
4576	1507
5998	1507
6845	1507
6425	1507
3408	1507
5173	1507
2096	1507
753	1507
6458	1507
6335	1507
6364	1507
3597	1506
3289	1506
5238	1506
5339	1506
5781	1506
6081	1506
5417	1506
4622	1506
6056	1506
5031	1506
2551	1506
6065	1506
5229	1506
283	1506
5568	1506
5119	1506
6679	1505
2393	1505
5525	1505
2367	1505
6861	1505
5836	1505
6766	1505
2500	1505
2815	1505
7047	1505
2080	1505
6556	1505
4456	1505
648	1505
3007	1504
6113	1504
5261	1504
4594	1504
3142	1504
6084	1504
6067	1504
2148	1504
5716	1504
3482	1504
3974	1504
6538	1504
2704	1503
5924	1503
6875	1503
4782	1503
4627	1503
4585	1503
4019	1503
4267	1503
2499	1503
5327	1503
810	1503
5348	1503
4153	1503
1891	1503
5872	1503
3537	1503
5831	1503
5891	1503
857	1503
6358	1502
4255	1502
3455	1502
3187	1502
4377	1502
6731	1502
619	1502
6391	1502
3513	1502
2861	1502
7116	1502
3083	1501
1758	1501
3954	1501
1714	1501
6036	1501
6898	1501
5196	1501
2399	1501
2994	1501
2993	1501
2078	1501
3491	1501
2856	1501
1984	1501
5098	1500
2879	1500
6704	1500
3136	1500
5774	1500
4593	1500
6322	1500
3941	1500
1785	1500
6574	1500
3234	1499
4416	1499
6695	1499
5684	1499
6744	1499
2640	1499
6203	1499
3388	1499
2443	1499
6440	1499
5830	1499
5093	1499
3070	1499
2723	1499
6567	1499
5531	1499
4009	1499
2634	1499
6901	1499
3189	1499
686	1498
3936	1498
1080	1498
4514	1498
6033	1498
6422	1498
2914	1498
5542	1498
701	1498
5959	1498
2171	1498
5113	1498
4829	1497
3202	1497
5865	1497
5900	1497
4243	1497
2713	1497
5161	1497
7331	1497
4057	1497
1157	1497
3848	1497
334	1497
4012	1497
5864	1497
3577	1497
6348	1496
5806	1496
2190	1496
2079	1496
6086	1496
6949	1496
1403	1496
7120	1496
1209	1496
2495	1496
5975	1496
339	1496
4327	1495
5851	1495
7142	1495
4314	1495
5627	1495
4021	1495
100	1495
5506	1495
3661	1495
7043	1495
6527	1495
2344	1495
3229	1494
5659	1494
3734	1494
6530	1494
2374	1494
2881	1494
7278	1494
3654	1494
4482	1494
4454	1494
7013	1494
6929	1494
3061	1493
6341	1493
4277	1493
6050	1493
6844	1493
6175	1493
6702	1493
3719	1493
3058	1493
1334	1493
3132	1493
5730	1493
7322	1493
6314	1493
6098	1493
6068	1493
6047	1493
1847	1492
2129	1492
3023	1492
5090	1492
6694	1492
6392	1492
5347	1492
7262	1492
751	1492
3859	1492
6637	1492
4648	1492
4076	1492
4074	1492
244	1491
3596	1491
5217	1491
321	1491
6111	1491
5274	1491
5933	1491
3881	1491
3163	1491
5915	1491
1740	1490
6038	1490
4630	1490
5964	1490
1965	1490
6150	1490
6888	1490
5783	1490
5920	1490
6432	1490
6671	1490
7222	1490
5421	1490
4180	1490
3799	1490
3036	1490
4467	1489
4595	1489
4608	1489
4600	1489
4663	1489
7137	1489
4942	1489
6054	1489
6914	1489
5082	1489
5993	1488
3081	1488
5484	1488
6398	1488
640	1488
7028	1488
5688	1488
6842	1488
5078	1488
5126	1487
5183	1487
2813	1487
3166	1487
4403	1487
7004	1487
6164	1487
1647	1487
4353	1487
6808	1487
1212	1487
6503	1487
4749	1487
5935	1486
7226	1486
6032	1486
6160	1486
6537	1486
6771	1486
4998	1486
5555	1486
6354	1486
5857	1486
6351	1486
6657	1486
3205	1486
2345	1486
4013	1485
4378	1485
1912	1485
7123	1485
5115	1485
3930	1485
6087	1485
4601	1485
4982	1485
7190	1484
5457	1484
6990	1484
4951	1484
1781	1484
5624	1484
1972	1484
5816	1484
7069	1484
6063	1484
6230	1484
4717	1484
1517	1483
4643	1483
6934	1483
5837	1483
2498	1483
4120	1483
6202	1483
3138	1483
4950	1483
2437	1483
1735	1483
5960	1483
6191	1483
4054	1483
904	1483
6518	1483
2398	1483
2984	1482
6801	1482
5881	1482
1329	1482
6044	1482
5056	1482
2574	1482
6342	1482
6325	1482
5720	1481
5934	1481
6305	1481
7267	1481
988	1481
1600	1481
4123	1481
3245	1481
2473	1481
1458	1481
5038	1481
5970	1481
6851	1481
6404	1481
5332	1481
451	1481
2600	1480
6659	1480
97	1480
1515	1480
6746	1480
6321	1480
3609	1480
2489	1480
7214	1480
6402	1480
4799	1480
5518	1480
6709	1480
7153	1480
2493	1480
6106	1480
6514	1480
2882	1479
2872	1479
3182	1479
7053	1479
5072	1479
5510	1479
6168	1479
4473	1479
5641	1479
4119	1479
6441	1479
5516	1479
295	1479
5243	1479
5443	1479
1646	1479
6186	1479
3786	1479
5678	1478
6001	1478
3544	1478
1591	1478
1159	1478
4748	1478
972	1478
2411	1478
6970	1478
5494	1478
981	1478
1991	1478
7148	1478
6655	1478
4141	1477
2358	1477
5260	1477
6833	1477
6660	1477
4505	1477
5754	1477
4765	1477
4652	1477
7230	1477
2531	1477
3212	1477
5263	1477
6476	1477
7275	1476
4572	1476
3745	1476
3946	1476
2916	1476
1450	1476
1672	1476
4262	1476
4573	1476
1793	1476
4571	1476
5136	1475
5943	1475
6742	1475
2444	1475
2896	1475
6885	1475
5248	1475
1771	1475
5753	1475
1143	1475
1633	1475
6088	1475
6076	1475
5953	1475
2458	1475
2523	1475
6464	1474
3839	1474
5085	1474
3669	1474
3510	1474
6019	1474
3721	1474
6720	1474
5834	1474
1258	1474
6841	1474
4914	1474
7103	1474
3043	1474
4107	1473
4471	1473
3612	1473
1571	1473
3096	1473
1382	1473
6961	1473
3220	1473
5110	1473
1076	1473
2773	1473
6991	1473
5578	1473
7068	1473
4693	1473
4500	1473
3102	1472
4809	1472
2900	1472
6097	1472
6199	1472
7198	1472
5295	1472
589	1472
6132	1472
3533	1472
1590	1472
6867	1472
4215	1472
5787	1472
1378	1472
5470	1472
1247	1472
1583	1472
6899	1471
4152	1471
581	1471
1293	1471
5709	1471
7056	1471
7234	1471
6918	1471
6964	1471
6542	1471
4590	1471
1288	1471
2613	1471
6002	1471
7058	1471
945	1471
6083	1471
3759	1471
7112	1471
6188	1471
7206	1470
4112	1470
7217	1470
1757	1470
3451	1470
6585	1470
6194	1470
3573	1470
4049	1470
6337	1470
7166	1470
3266	1470
6403	1470
4610	1470
2978	1470
5534	1470
5973	1469
2240	1469
4704	1469
2577	1469
4740	1469
6712	1469
6883	1469
6302	1469
4692	1469
6706	1468
5440	1468
5988	1468
5855	1468
6718	1468
5401	1468
6423	1468
5134	1468
3004	1468
2051	1468
6318	1468
7080	1468
6489	1468
3173	1467
5159	1467
4746	1467
2207	1467
6397	1467
4104	1467
5992	1467
6566	1467
6400	1467
1818	1467
5968	1467
4173	1467
4992	1467
5928	1467
3175	1467
2895	1467
6993	1467
6874	1467
6708	1467
6485	1467
6226	1467
3975	1467
31	1466
6107	1466
1332	1466
6331	1466
7067	1466
6820	1466
6356	1466
698	1466
2439	1466
199	1466
3134	1466
3467	1466
3211	1466
5459	1466
4332	1465
6504	1465
4761	1465
5519	1465
4373	1465
1398	1465
4466	1465
6968	1465
6238	1465
5002	1465
6467	1465
4328	1465
2450	1465
4979	1465
4787	1465
3796	1465
5911	1465
4645	1465
6482	1464
2576	1464
5253	1464
1504	1464
3591	1464
7174	1464
3543	1464
2957	1464
6569	1464
6524	1464
7319	1464
6546	1464
313	1464
69	1464
6016	1464
4008	1463
3019	1463
4309	1463
3807	1463
4418	1463
7135	1463
5071	1463
4580	1463
2667	1463
5411	1463
6444	1463
5622	1463
4005	1463
3319	1462
6170	1462
5923	1462
5966	1462
4978	1462
4080	1462
117	1462
45	1462
5333	1462
7216	1462
5748	1462
6896	1462
2532	1462
6326	1462
6729	1462
5143	1462
6436	1462
6889	1461
6367	1461
4284	1461
2177	1461
6677	1461
4151	1461
4598	1461
6952	1461
6668	1461
5685	1461
4301	1461
6093	1461
6166	1461
34	1461
5613	1461
7097	1461
435	1461
4796	1460
6593	1460
6560	1460
6855	1460
5874	1460
1359	1460
6765	1460
2721	1460
7158	1460
3035	1460
7006	1460
5999	1460
4285	1460
7018	1460
4270	1459
6558	1459
5746	1459
1927	1459
6920	1459
6969	1459
5142	1459
4127	1459
7295	1459
5681	1459
4383	1459
2641	1459
3055	1459
7055	1459
4511	1459
5729	1459
6664	1459
6847	1459
5719	1459
5057	1459
2905	1459
4496	1459
6871	1459
5162	1459
2484	1459
4932	1459
7326	1459
4097	1458
5022	1458
6830	1458
6648	1458
4336	1458
2036	1458
5700	1458
7091	1458
5682	1458
6215	1458
5235	1458
7274	1458
6947	1458
6985	1457
3288	1457
4821	1457
2187	1457
4248	1457
2545	1457
2554	1457
706	1457
4461	1457
2854	1457
3619	1457
6171	1457
6204	1457
3464	1457
3952	1456
5849	1456
1254	1456
4763	1456
4729	1456
848	1456
5167	1456
5144	1456
4338	1456
7202	1456
6324	1456
4229	1456
4078	1456
7149	1456
3037	1456
3407	1456
1294	1456
3180	1455
1422	1455
6935	1455
6701	1455
6730	1455
4694	1455
6882	1455
7108	1455
1482	1455
3957	1455
2604	1455
7064	1455
7271	1455
5538	1455
6622	1455
5310	1455
6565	1455
2347	1455
5077	1455
6953	1455
996	1455
2935	1455
4053	1454
3139	1454
6133	1454
6223	1454
6013	1454
6239	1454
3566	1454
6928	1454
6120	1454
6900	1454
4578	1454
6570	1454
3398	1453
4501	1453
555	1453
806	1453
830	1453
4925	1453
1452	1453
6727	1453
6468	1453
4065	1453
5493	1453
5853	1453
5175	1453
6540	1453
1872	1453
5515	1453
7240	1452
5630	1452
7165	1452
3152	1452
6499	1452
5894	1452
3225	1452
5147	1452
6492	1451
6887	1451
955	1451
6922	1451
1432	1451
3559	1451
6619	1451
3329	1451
6962	1451
1787	1451
5919	1451
5863	1451
6989	1451
4506	1451
3017	1451
6515	1451
6060	1451
7317	1450
4741	1450
5293	1450
6507	1450
4212	1450
5637	1450
3147	1450
7059	1450
1370	1450
5728	1450
5268	1450
4114	1450
6315	1450
5665	1450
7195	1450
5418	1450
5773	1449
3568	1449
6870	1449
841	1449
5514	1449
4083	1449
2550	1449
7119	1449
1985	1449
4674	1449
2986	1449
7094	1449
3120	1449
6721	1449
6955	1449
3361	1449
5350	1448
4006	1448
6129	1448
7308	1448
3090	1448
3402	1448
4726	1448
4217	1448
5660	1448
5683	1448
6127	1448
5593	1448
7152	1448
3468	1448
5633	1448
4926	1448
5069	1448
5777	1448
6806	1448
5908	1448
4169	1447
6640	1447
2232	1447
2840	1447
1303	1447
3387	1447
5708	1447
4702	1447
7185	1447
5213	1447
7258	1447
2603	1447
2228	1447
6519	1447
7220	1447
4329	1446
3292	1446
6822	1446
442	1446
4240	1446
5994	1446
7060	1446
6979	1446
6644	1446
1123	1446
1308	1446
6184	1446
3265	1446
5595	1445
3193	1445
4919	1445
6697	1445
3878	1445
166	1445
4568	1445
2525	1445
5483	1445
7065	1445
7244	1445
7010	1445
5737	1445
5482	1445
6220	1444
3157	1444
4299	1444
4056	1444
7256	1444
6412	1444
3562	1444
5611	1444
1727	1444
3172	1444
2961	1444
4768	1444
5932	1444
2654	1444
5095	1444
4118	1444
7086	1444
1230	1443
6506	1443
3822	1443
4171	1443
6686	1443
193	1443
7066	1443
6146	1443
423	1443
5996	1443
6632	1443
6091	1443
7243	1443
3466	1442
7040	1442
7312	1442
7173	1442
2513	1442
6189	1442
6586	1442
496	1442
7297	1442
3571	1442
2927	1442
7138	1442
4397	1442
5041	1442
4295	1442
5282	1442
3240	1442
2544	1442
4828	1441
6144	1441
5465	1441
5123	1441
4579	1441
6520	1441
6543	1441
5503	1441
7201	1441
3751	1441
5657	1441
4412	1441
5477	1441
2424	1441
6350	1441
4723	1441
4616	1441
6435	1440
6700	1440
7070	1440
4232	1440
4205	1440
6974	1440
7019	1440
6649	1440
5697	1440
6926	1440
6682	1439
7235	1439
5558	1439
7121	1439
2480	1439
6071	1439
3939	1439
5835	1439
3680	1439
3955	1439
7318	1439
1585	1439
3740	1439
5722	1439
5067	1439
743	1439
5454	1439
2591	1438
7177	1438
4364	1438
4453	1438
6039	1438
4823	1438
6995	1438
1977	1438
6904	1438
3359	1438
7296	1438
6690	1438
7251	1437
4360	1437
5202	1437
4555	1437
4973	1437
5066	1437
7303	1437
6312	1437
2010	1437
4955	1437
4504	1437
5625	1437
6868	1437
6473	1437
3453	1437
6911	1437
7257	1437
6559	1437
5486	1437
5048	1437
4957	1437
6950	1436
3545	1436
4238	1436
5498	1436
7264	1436
3815	1436
6805	1436
6740	1436
155	1436
6457	1436
5539	1435
2064	1435
453	1435
6932	1435
2511	1435
7122	1435
7301	1435
6535	1435
6872	1435
6741	1435
4182	1435
6608	1435
1811	1435
1994	1435
1940	1435
3045	1434
6141	1434
256	1434
4707	1434
2423	1434
5929	1434
4269	1434
3390	1434
6597	1434
3268	1434
7229	1434
5636	1434
6976	1434
5661	1434
2585	1434
4575	1434
3204	1434
6004	1434
6956	1434
1566	1434
5634	1434
3366	1434
5648	1434
2853	1433
6035	1433
2964	1433
5941	1433
5800	1433
7293	1433
5129	1433
5655	1433
885	1433
4068	1433
7327	1433
695	1433
1075	1433
4933	1433
5025	1433
5974	1433
6041	1432
1277	1432
1751	1432
6913	1432
3297	1432
7245	1432
2530	1432
6998	1432
7031	1432
5962	1432
6525	1432
7023	1432
5016	1432
6219	1432
6656	1432
6992	1432
3062	1432
5312	1432
2461	1432
816	1431
7071	1431
4308	1431
2506	1431
4847	1431
7155	1431
1942	1431
7016	1431
6743	1431
5118	1431
6854	1431
6606	1431
6575	1431
3006	1431
7072	1430
2077	1430
5852	1430
4567	1430
6674	1430
3754	1430
4730	1430
7188	1430
4132	1430
7260	1430
877	1430
6526	1430
6078	1430
6382	1430
1374	1430
6616	1430
6433	1430
7316	1430
6381	1429
6750	1429
6723	1429
3947	1429
2404	1429
3693	1429
3648	1429
6488	1429
5060	1429
7255	1429
6666	1429
6749	1429
5250	1429
4464	1429
6103	1429
6167	1429
5589	1429
4193	1429
2259	1429
5887	1429
3067	1429
4260	1428
6130	1428
3454	1428
3843	1428
4225	1428
5256	1428
7212	1428
7248	1428
1168	1428
7022	1428
5265	1428
6023	1428
3373	1428
6413	1427
4964	1427
4835	1427
6850	1427
3457	1427
2906	1427
4683	1427
4186	1427
2027	1427
4292	1427
253	1427
6099	1427
5639	1427
1446	1427
7025	1426
2966	1426
6612	1426
714	1426
5957	1426
5178	1426
7038	1426
7041	1426
6536	1426
4247	1426
5701	1426
6809	1426
1474	1426
4043	1426
5003	1425
5219	1425
6670	1425
4519	1425
4034	1425
4815	1425
1759	1425
4187	1425
6684	1425
6651	1425
1137	1425
6190	1425
6865	1425
1799	1425
6755	1425
6419	1425
4665	1425
4311	1424
5947	1424
6157	1424
4516	1424
5721	1424
6416	1424
3103	1424
5752	1424
5937	1424
1665	1424
7315	1424
5168	1424
6915	1424
7193	1424
5876	1423
6683	1423
6927	1423
5983	1423
5695	1423
3470	1423
6646	1423
5047	1423
7035	1423
4854	1423
3065	1423
5019	1423
6407	1423
1737	1422
2239	1422
3651	1422
2825	1422
6643	1422
3610	1422
4913	1422
4528	1422
5478	1422
2912	1422
6309	1422
6963	1422
3976	1422
6916	1422
6866	1422
1290	1421
5141	1421
6014	1421
1908	1421
7272	1421
6393	1421
537	1421
2167	1421
4024	1421
7085	1421
6588	1421
5841	1421
4662	1421
3507	1420
6509	1420
5885	1420
7002	1420
2234	1420
6059	1420
6371	1420
6892	1420
5714	1420
1831	1420
4219	1420
3133	1419
6699	1419
6738	1419
2262	1419
5149	1419
3151	1419
6834	1419
5130	1419
5438	1419
3352	1419
4851	1418
5727	1418
4640	1418
6455	1418
1165	1418
6944	1418
6917	1418
3748	1418
5829	1418
6604	1418
4549	1418
7104	1418
6140	1418
3636	1418
6346	1417
6101	1417
5877	1417
5133	1417
6838	1417
4838	1417
1896	1417
746	1417
6306	1417
6986	1417
6662	1417
4767	1417
3122	1417
5725	1416
7030	1416
2538	1416
5963	1416
3257	1416
6000	1416
5541	1416
6075	1416
2001	1415
216	1415
6642	1415
7146	1415
3298	1415
1307	1415
2470	1415
5275	1415
5200	1415
3728	1415
766	1414
1915	1414
6832	1414
5776	1414
5344	1414
1769	1414
5089	1414
3984	1414
4780	1414
7046	1414
4161	1414
6825	1414
7036	1414
5241	1414
6369	1414
6320	1414
1512	1414
6921	1414
5621	1413
6645	1413
6487	1413
4348	1413
7299	1413
6360	1413
5626	1413
4637	1413
7265	1412
3851	1412
6005	1412
3615	1412
7289	1412
7189	1412
2261	1412
1642	1412
6858	1411
6804	1411
5549	1411
3243	1411
2725	1411
3729	1411
6408	1411
5203	1411
3300	1411
6958	1411
3575	1411
3091	1411
7017	1411
1573	1411
5471	1411
3785	1411
5058	1410
6818	1410
545	1410
7232	1410
5194	1410
4940	1410
7081	1410
6762	1410
4744	1409
7298	1409
6332	1409
4386	1409
6498	1409
1943	1409
7219	1409
3029	1409
5283	1409
1716	1409
6768	1409
3812	1409
5662	1409
6512	1409
3865	1409
564	1409
1546	1408
6119	1408
7107	1408
4111	1408
2998	1408
5103	1408
3498	1408
4156	1408
7084	1408
5747	1408
3684	1408
6481	1408
6807	1407
5918	1407
4988	1407
5583	1407
4547	1407
5227	1407
5125	1407
4757	1407
867	1407
5186	1407
4804	1407
4614	1407
2620	1407
884	1406
6079	1406
4958	1406
7237	1406
4924	1406
6877	1406
6010	1406
6116	1406
3704	1406
5699	1405
5854	1405
4715	1405
3131	1405
1803	1405
7254	1405
2068	1405
2942	1405
4015	1405
6817	1405
1555	1405
7088	1405
5921	1405
6161	1405
3988	1405
6094	1405
6379	1404
5326	1404
7192	1404
4166	1404
7101	1404
4604	1404
5340	1404
5906	1404
5412	1403
6138	1403
6859	1403
3650	1403
1027	1403
5785	1403
6128	1403
4762	1403
2707	1403
7224	1403
3013	1402
6624	1402
6860	1402
7218	1402
6846	1402
7115	1402
2915	1402
2496	1402
6459	1402
7221	1402
606	1402
7277	1402
6910	1402
6362	1401
2048	1401
781	1401
6754	1401
1719	1401
2709	1401
6178	1401
2515	1401
6819	1401
2929	1401
3723	1401
854	1401
6581	1401
7076	1401
468	1401
5132	1401
4817	1401
6897	1401
7001	1401
5074	1401
3465	1401
6547	1401
6693	1401
3201	1400
7175	1400
4281	1400
6149	1400
6840	1400
2095	1400
6156	1399
7269	1399
1649	1399
5005	1399
7223	1399
5570	1399
6931	1399
3944	1399
5672	1399
2819	1399
4042	1399
7093	1399
2039	1398
4235	1398
6564	1398
6073	1398
5063	1398
6494	1398
7124	1398
6685	1398
6626	1398
1944	1398
5512	1397
3499	1397
2408	1397
2945	1397
4337	1397
5491	1397
4987	1397
5247	1397
4773	1397
6410	1397
6561	1397
6610	1397
938	1397
299	1397
4968	1396
7294	1396
919	1396
6383	1396
4139	1396
7029	1396
6902	1395
5756	1395
6912	1395
6214	1395
7118	1395
6982	1395
5329	1395
7197	1395
6388	1395
7151	1395
5610	1395
5251	1395
5619	1395
6732	1395
5582	1395
6414	1394
7126	1394
4252	1394
2083	1394
128	1394
7266	1394
6453	1394
1576	1394
6948	1394
2136	1393
4907	1393
4856	1393
335	1393
1802	1393
6118	1393
6946	1393
5870	1393
5689	1393
6187	1393
7128	1393
5122	1393
6053	1392
7203	1392
6925	1392
7167	1392
5989	1392
2855	1392
6923	1392
7238	1392
6449	1392
4641	1392
5214	1391
3821	1391
2847	1391
5316	1391
4649	1391
7291	1391
5949	1391
5135	1391
6725	1390
5255	1390
2197	1390
7164	1390
3739	1390
6938	1389
7106	1389
6557	1389
7063	1389
5189	1389
6463	1389
2676	1389
6545	1389
2339	1389
2246	1388
3323	1388
7176	1388
6881	1388
3800	1388
4938	1388
3617	1388
1677	1388
6764	1388
3054	1388
7191	1388
5744	1388
371	1388
5557	1388
6894	1388
5290	1388
4954	1387
3104	1387
7102	1387
2529	1387
2950	1387
5667	1387
5049	1387
2161	1386
6145	1386
4137	1386
4093	1385
3282	1385
6554	1385
5201	1385
6873	1385
4455	1385
4802	1385
3278	1385
6531	1385
6940	1385
3673	1384
4734	1384
7227	1384
4023	1384
6814	1384
6092	1384
2429	1384
6943	1384
5616	1384
6676	1384
6240	1383
4682	1383
6864	1383
5734	1383
4371	1383
5564	1383
3291	1383
5840	1383
171	1383
3370	1383
6319	1382
692	1382
6965	1382
7168	1382
4701	1381
6983	1381
6066	1381
7077	1381
1743	1381
5875	1381
2988	1381
3515	1381
6692	1381
3632	1381
5762	1381
5215	1381
6826	1380
3367	1380
3344	1380
590	1380
3876	1380
6748	1380
5445	1380
5474	1379
2874	1379
6533	1379
6399	1379
6884	1379
7180	1379
6601	1378
2844	1378
7083	1378
6937	1378
2477	1378
4051	1378
6430	1378
1828	1378
3834	1378
5490	1378
6739	1377
4777	1377
1791	1377
4335	1377
6007	1377
6376	1377
2265	1377
4792	1377
6636	1377
6135	1377
5107	1377
5166	1377
4480	1377
6977	1377
5473	1377
4110	1377
6654	1376
7184	1376
7143	1376
5607	1376
7205	1376
7183	1376
522	1376
1448	1376
6981	1375
7170	1375
6304	1375
7323	1375
3981	1375
3525	1375
1922	1375
5878	1375
6635	1375
7313	1375
6213	1375
6327	1374
7033	1374
280	1374
1739	1374
5965	1374
7263	1374
653	1374
7136	1374
7161	1374
6975	1374
1700	1374
6363	1374
3181	1373
1350	1373
5821	1373
6997	1373
204	1373
5562	1373
7247	1373
1601	1373
5586	1373
6339	1373
3743	1373
6303	1372
3694	1372
5408	1372
7134	1372
7163	1372
6811	1372
7268	1372
6967	1372
5430	1372
6600	1372
6460	1372
6665	1371
7172	1371
1786	1371
7273	1371
7249	1371
5765	1371
6658	1370
7286	1370
6181	1370
5288	1370
6667	1370
4562	1370
6102	1370
5679	1370
5279	1370
6680	1370
6691	1370
7209	1370
3794	1370
4801	1369
4283	1369
7228	1369
2897	1369
6620	1369
5447	1369
6891	1369
1899	1369
1502	1369
2441	1369
240	1369
1761	1368
7130	1368
6736	1368
7309	1368
6180	1368
7150	1368
4739	1368
3295	1368
3527	1368
81	1368
7246	1367
7027	1367
5156	1367
6959	1367
5726	1367
4583	1367
5926	1367
6716	1366
7196	1366
4733	1366
2969	1366
5198	1366
6218	1366
5033	1365
6594	1365
6837	1365
6652	1365
7095	1365
7141	1365
7208	1365
6195	1365
3756	1364
3215	1364
6089	1364
7156	1364
6333	1364
5027	1364
6505	1363
6568	1363
6631	1363
6583	1363
1528	1363
7321	1362
7045	1362
6605	1362
4783	1362
6137	1362
6553	1362
1085	1362
6589	1362
6513	1361
6017	1361
1954	1361
6308	1361
937	1361
7290	1360
5527	1360
5298	1360
7050	1359
6532	1359
4408	1359
5706	1358
3712	1358
5239	1358
6074	1358
5705	1358
3845	1358
4625	1358
6029	1358
771	1358
4179	1358
6064	1358
3336	1357
1598	1357
6529	1357
3393	1357
6812	1357
6627	1357
6011	1356
4811	1356
6057	1356
4661	1356
5869	1356
7145	1356
2944	1356
5977	1356
7302	1355
4286	1355
2224	1355
5532	1355
7288	1355
7125	1355
5559	1354
4203	1354
6580	1354
6599	1354
3028	1354
7037	1354
1529	1354
6584	1353
2898	1353
6012	1353
6147	1353
647	1353
5871	1352
7147	1352
7052	1352
5500	1352
6905	1352
2673	1352
5858	1352
6232	1351
7314	1351
7133	1351
3634	1351
6960	1351
6906	1351
6162	1351
2357	1350
7075	1350
6999	1350
6592	1349
6461	1349
5902	1349
7113	1349
6301	1349
5065	1348
6715	1348
4827	1348
7210	1348
6428	1347
1580	1347
7009	1347
6021	1347
6954	1347
2070	1347
5197	1346
6139	1346
6045	1346
4319	1346
7324	1346
7171	1346
4380	1346
539	1345
3993	1345
2926	1345
585	1345
5888	1344
6355	1344
5257	1344
4764	1344
7096	1343
6552	1343
7204	1343
6638	1342
6711	1342
5710	1342
7194	1342
5944	1342
6688	1340
5225	1340
6528	1340
6751	1340
3532	1339
6639	1339
7270	1337
538	1337
6222	1337
6821	1337
5308	1336
151	1336
5991	1336
4681	1336
7330	1335
4291	1335
6136	1334
5848	1334
5556	1333
6987	1333
7032	1333
3302	1332
3049	1331
6971	1331
6096	1330
4840	1330
87	1329
4784	1329
3110	1329
5070	1328
5605	1327
3773	1327
4390	1327
6863	1326
7242	1326
6049	1326
773	1326
4949	1324
617	1324
1827	1323
7215	1323
4902	1323
3458	1322
3985	1322
5448	1322
6345	1322
6893	1321
6862	1319
7311	1319
7057	1319
6541	1319
5666	1318
1554	1318
7024	1318
3714	1318
352	1317
7000	1316
7073	1316
4382	1315
1973	1314
7020	1314
5732	1314
5231	1313
7042	1313
6563	1313
2151	1312
7259	1312
136	1311
5651	1311
6018	1309
7292	1308
2785	1305
6372	1305
6714	1305
4929	1304
6426	1304
5954	1302
6611	1299
5828	1298
6152	1297
5642	1297
6633	1295
3340	1294
6895	1291
5664	1290
3237	1286
6114	1285
322	1284
4816	1280
3586	1280
6595	1275
4195	1274
6707	1262
Reply With Quote
  #35   Spotlight this post!  
Unread 05-04-2018, 11:28 AM
osofdez osofdez is offline
Registered User
FRC #3478
 
Join Date: Feb 2017
Location: Mexico
Posts: 12
osofdez is an unknown quantity at this point
Re: paper: FRC Elo 2008-2016

Amazing Job, I have a question in the "End of Season" Elos there are no team numbers column could you please add it to be able to find teams more easily?

Thaks
Reply With Quote
  #36   Spotlight this post!  
Unread 05-04-2018, 01:01 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by osofdez View Post
Amazing Job, I have a question in the "End of Season" Elos there are no team numbers column could you please add it to be able to find teams more easily?

Thaks
I have uploaded a v2 which is scrolled all the way to the left on the "End of Season Elos" sheet to show the team numbers. Let me know if that wasn't the issue you were experiencing.

I also added 2018 to the legend on the graph in the "Team Lookup" sheet.
Reply With Quote
  #37   Spotlight this post!  
Unread 05-04-2018, 06:50 PM
Matt G's Avatar
Matt G Matt G is offline
365 --> 4039
AKA: Mattward
FRC #4039 (Makeshift)
Team Role: Mentor
 
Join Date: Apr 2015
Rookie Year: 2009
Location: Ontario
Posts: 62
Matt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud of
Re: paper: FRC Elo 2008-2016

A few questions:

Any thoughts on the increased standard deviation in the last two years? seems like a large jump from the values of the other years (although 2014 is close). I suspect this reflects more on game design than your model, but I'd still be interested in hearing your thoughts.

I assume the 70/30 weighting on the past two year's elo was chosen due to it having the strongest predictions for future performance, do you think that there's any value in a different weighting scheme for an All-Time elo rank? If so, how might you weight the previous years? Would you keep the current 70/30 system, have an equal average of all years, weight years by the predictive strength of elo? Have you thought of making a GOAT elo-rank and do you think there would be any value to an elo system that didn't have such a hard reset after each season?
Reply With Quote
  #38   Spotlight this post!  
Unread 05-05-2018, 06:40 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by CVR View Post
A few questions:

Any thoughts on the increased standard deviation in the last two years? seems like a large jump from the values of the other years (although 2014 is close). I suspect this reflects more on game design than your model, but I'd still be interested in hearing your thoughts.
I think the standard deviations are almost entirely a function of the game design. I just use them as a rough way to normalize winning margins between years. What I find more interesting is the standard deviation of Elo ratings for each year:


These give a loose upper bound on how "dominant" a team can possibly be for each year. 1114 at the end of 2008 had an Elo nearly 200 points higher (or more than two standard deviations higher) than the next highest team, in a year where it was relatively difficult to be dominant.

2018 in contrast was one of the easiest years to be dominant, likely due to the positive feedback loops for scoring described in other threads. It's because of years like this that I try to make as few assumptions as possible in my Elo model, since my Elo predictions this year were great at the same time as some other predictive methods were poor because they didn't know how to deal with a game like this. I'm a little concerned that 254 might have been so good this year that some of the assumptions Elo made broke down and under-rated them. I only gave them a 0.012% chance of going undefeated (if you look at pre-match win probabilities), which might have been low, but it's hard to tell since this is such a rare event, and I might be engaging in hindsight bias.

Quote:
I assume the 70/30 weighting on the past two year's elo was chosen due to it having the strongest predictions for future performance
Yes, this is correct.

Quote:
...do you think that there's any value in a different weighting scheme for an All-Time elo rank? If so, how might you weight the previous years? Would you keep the current 70/30 system, have an equal average of all years, weight years by the predictive strength of elo?
I mean, sure, I think there are alternative systems out there that have value, and parameters in my current model could be tweaked to maximize performance in regards to something other than predictive power. I don't know how I'd weight previous years if building a system like this. I can't think of any metric to try to optimize like I do with predictive power, so I'd probably just pick values that "feel right" to me, which I am loathe to do since I can't really objectively justify why I made the decisions I did.

Quote:
Have you thought of making a GOAT elo-rank and do you think there would be any value to an elo system that didn't have such a hard reset after each season?
Yeah, I've certainly thought about trying to get more into the GOAT conversation, but I think when you get down to it, these conversations rest more on personal preference than anything objective. You can certainly use statistics to back up your claims, but unless you have a single statistic which encapsulates everything about a team and everyone agrees this is the best metric, people will pick and choose which metrics to use to decide who is the GOAT.

Personally, I don't put much weight in 254's undefeated season. I don't think they have many great wins in quals, and they had some awesome playoff partners who provided a lot of help to them when they faced tough opponents in the playoffs. It kind of stinks for them, but it may not have even been possible for them to win the GOAT position in my mind this year just because it was so easy for other teams to also be dominant. All of this is just my opinion though, other people value other things in the GOAT conversation and that's fine.

If I had to make a GOAT list, I would probably use the Elo standard deviations above to find out which teams were the most dominant relative to the rest of the field in their year. It would probably look something like this, with the top 3 or so in serious contention:


Sometime I'll probably extend Elos back to 2002, which would be cool because then we could include the legendary 71 in 2002.

All in all, I'm much more interested in maximizing predictive power than almost anything else. Anyone can choose to rank teams however they would like, and that's totally fine. However, at some point I would encourage people to put their ideas to the test and actually take a shot at predicting things. Otherwise you are very likely getting caught up in hindsight bias.
Reply With Quote
  #39   Spotlight this post!  
Unread 05-07-2018, 09:10 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by Caleb Sykes View Post
Personally, I don't put much weight in 254's undefeated season. I don't think they have many great wins in quals, and they had some awesome playoff partners who provided a lot of help to them when they faced tough opponents in the playoffs. It kind of stinks for them, but it may not have even been possible for them to win the GOAT position in my mind this year just because it was so easy for other teams to also be dominant. All of this is just my opinion though, other people value other things in the GOAT conversation and that's fine.
Just for fun, I went and found what kind of scores 254 would have needed to end the season as strong as 1114 did in 2008 relative to the next best team. To do this, 254 would have needed an ending Elo of 2293, which they could have achieved if they had won all of their matches by around 100 more points than they did. Obviously, there are other ways they could have gotten their Elo this high, but this gives a sense of how much better they would have needed to be to get my GOAT title.
Reply With Quote
  #40   Spotlight this post!  
Unread 05-07-2018, 09:45 AM
Bryce2471's Avatar
Bryce2471 Bryce2471 is offline
Alumnus
AKA: Bryce Croucher
FRC #2471 (Team Mean Machine)
Team Role: Mentor
 
Join Date: Feb 2013
Rookie Year: 2006
Location: Camas, WA
Posts: 664
Bryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by Caleb Sykes View Post
Just for fun, I went and found what kind of scores 254 would have needed to end the season as strong as 1114 did in 2008 relative to the next best team. To do this, 254 would have needed an ending Elo of 2293, which they could have achieved if they had won all of their matches by around 100 more points than they did. Obviously, there are other ways they could have gotten their Elo this high, but this gives a sense of how much better they would have needed to be to get my GOAT title.
I love the idea of using standard deviation of ELO to gauge the dominance of proformances accross the years. I don't like the idea of comparing to the next best team though, because the ability of the second team seems fairly arbitrary and two extremely outlier robots could easily be built in the same year. I think it would be much better to to use the standard deviations above the average to compare across years.

I also personally see more recent achievements in a brighter light, (This what you were talking about with persal opinions) because all teams are getting much better at building robots over the years. For example, 1114 was incredibly dominant at the time in 2008, but if FRC gave us the 2008 game again next year, I believe their would be hundreds of similarly capable teams.
__________________
2006-2011:
FLL Team Future imagineers
2011-????:
FRC Team Mean Machine
Reply With Quote
  #41   Spotlight this post!  
Unread 05-07-2018, 10:17 AM
Jared Russell's Avatar
Jared Russell Jared Russell is offline
4933T15
FRC #0254 (The Cheesy Poofs), FRC #0341 (Miss Daisy)
Team Role: Engineer
 
Join Date: Nov 2002
Rookie Year: 2001
Location: San Francisco, CA
Posts: 3,540
Jared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond reputeJared Russell has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

2012, 2016, 2017, and 2018 have featured bonus RPs during qualification matches.
Many successful teams in these seasons often sacrificed winning margin for RPs to varying degrees, and I'm not sure how to quantify that in a summary statistic. Is there a way to incorporate these in the Elo model that captures all the nuances (some fraction of RPs correlate to whether you win the match, but the other fraction is independent of opponent)? Maybe some combination of Elo + ranking point OPR?
Reply With Quote
  #42   Spotlight this post!  
Unread 05-07-2018, 10:40 AM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,058
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Another reason why 254 in 2018 may seem less dominant than 1114 in 2008 might be due to how Caleb's Elo factors in winning margin. Scores in 2018 were time-based, so it was probably harder for 254 to stand out (statistically) in terms of scores and winning margins as it was for 1114 in 2008.
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #43   Spotlight this post!  
Unread 05-07-2018, 10:51 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by Jared Russell View Post
2012, 2016, 2017, and 2018 have featured bonus RPs during qualification matches.
Many successful teams in these seasons often sacrificed winning margin for RPs to varying degrees, and I'm not sure how to quantify that in a summary statistic. Is there a way to incorporate these in the Elo model that captures all the nuances (some fraction of RPs correlate to whether you win the match, but the other fraction is independent of opponent)? Maybe some combination of Elo + ranking point OPR?
I'll have to think about this more. It's really a tough problem that I've wanted to address but can't think of a great way to do so. My best idea at the moment would be to try to quantify the opportunity cost (in terms of winning margin) of each different kind of ranking point in quals matches. You could then create adjusted quals scores by adding this cost to the total score if the team gets the ranking point.

For example, this year I would guess the opportunity cost of choosing to go for the auto RP to be ~5 points, since the autos don't vary a ton between quals and playoffs, but teams will be more likely to go for riskier (but on average higher scoring) autos in playoffs since the cost of one robot not crossing the line or no one getting a cube in the switch is so much lower in quals than playoffs. Likewise, I would estimate that going for the climb RP has a much higher opportunity cost, maybe ~20 points. Even without the climbing RP, many teams still climb in the playoffs because the time based scoring makes switch/scale scoring not as valuable in the last 30 seconds. However in playoffs, teams seemed to spend a few seconds less prepping for the climb, and would use that time to score other points or play defense.

Those are just guesses though, sometime I might try making "adjusted" quals scores that incorporate the opportunity costs described above, and I can select the opportunity costs that provide the most predictive power for future team performance.
Reply With Quote
  #44   Spotlight this post!  
Unread 05-07-2018, 10:54 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

Quote:
Originally Posted by Eugene Fang View Post
Another reason why 254 in 2018 may seem less dominant than 1114 in 2008 might be due to how Caleb's Elo factors in winning margin. Scores in 2018 were time-based, so it was probably harder for 254 to stand out (statistically) in terms of scores and winning margins as it was for 1114 in 2008.
This is very true. If my model were based on WLT instead of winning margin, it would be very plausible that 254 in 2018 would look the best.
Reply With Quote
  #45   Spotlight this post!  
Unread 08-18-2018, 04:37 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is online now
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: FRC Elo 2008-2016

I've uploaded a new book, this is the Elo model I am currently planning to use going into the 2019 season (although that is subject to change). The only key change in this book compared to my previous books is the addition of 2002-2004 matches, which has negligible impact on 2018+ Elos.

I advised caution when looking at 2005-2007 due to poor data quality, so I need to caution twice as hard about this when looking at 2002-2004 data. Even for events in which the data quality is solid, the incentive structure in those years can potentially cause Elo to be misleading. The predictive power in these years is worse than basically every other year on record, however, they are worth including in the model if only to improve predictive power in the following years.

Interestingly, the key model parameters for 2002-2004 are essentially the same as the parameters for 3-team alliance years. I could have made them a bit different to improve predictive power a touch, but I have opted to keep them the same for simplicity.

I didn't make any major model changes other than including 2002-2004, but it certainly wasn't for a lack of trying. Here are the big changes I attempted but ended up opting not to include:
  • Varying kQuals and kPlayoffs for 2-team alliances in 2002-2004: I mentioned this above, but the predictive power increase was small enough that this change didn't merit inclusion.
  • Using the log of match winning margins instead of the raw match winning margin (stolen from xkcd and initiated by conversations with Bob Croucher): This was to hopefully counteract the annoying effects of ridiculous penalty matches and/or red cards. Unfortunately, this didn't improve predictive power in every year I applied it, although some years did see improvement. In a similar vein, I tried taking the log of the difference between the predicted match score and the actual match score, this had similarly disappointing results.
  • Updating Elo values differently for the highest, second highest, and lowest Elo teams (stolen from Eugene Fang): The idea here is that each team doesn't contribute equally to matches, so it might make more sense to, for example, weight the Elo change more for the "better" teams than the worse teams, or vice versa. Unfortunately, this had negligible impact on predictive power while also being inconsistent between years.
  • Using a weighted sum of Elos instead of just taking a raw sum when predicting match results (stolen from Kevin Leonard on discord): This is a similar idea to the one above, but it would be applied before the match instead of after the match. For example, I found that, in 2018, it would have been best to take 1.5*(best team's Elo) + 1.0*(second best team's Elo) + 0.5*(worst team's Elo) when determining the alliance's strength instead of just taking a raw sum. This actually provided a very large amount of predictive power increase when applied to 2018, but the optimal weights varied drastically across years. For example, 2017 had weights very close to 1.0, 1.0, and 1.0, and 2016 was close to 1.5, 1.0, and 1.0. Because of this, I had to leave this improvement out of the model.
  • Using a geometric mean instead of an arithmetic mean when predicting match results: This is tied very closely to the above change. The idea would be to have a weighting of sorts for teams of varying skills, since geometric means (depending on where you place the zero) will either be higher or lower than arithmetic means but will also factor in alliance composition. The results were very similary to the different weightings, the optimal placement for the zero varied drastically year-to-year, but could potentially provide drastic predictive power increase in years like 2018.
  • Setting some cap on the amount a team's Elo rating can change between matches: This was another approach to dealing with the ridiculous penalty matches. Unfortunately, the predictive power increase with this change was negligible. Perhaps waiting until after a team has played a few matches to implement the cap could be valuable? Since team's ratings move around more early in the season. I'm not too hopeful though.
  • Using a variant of the fivethirtyeight Elo update formula: The model currently only looks at the winning margin when updating Elos after matches. This seems a bit counterintuitive since, for example, it will cause a team's Elo to drop if they win a match by 190 points if they were expected to win by 210. I've tried this change in the past unsuccessfully, but I thought I'd take another crack at it since I think I'm at least a little smarter than I was two years ago. Nothing much came out of it though. I could be misunderstanding how to implement it, but I manually tweaked all of the parameters and couldn't get anything even slightly more predictive than my current model. I'm starting to believe that, in most matches, perhaps teams aren't actually trying to win, but are rather trying to maximize the match winning margin. This makes intuitive sense to me since matches are so short, there is no perfect information about the game state, any team trying to strictly win has two partners they don't have direct control over, and there are frequently alternative incentives in a match instead of winning (showing off for alliance selection, going for other RPs, etc...). If this is true, then it would make more sense for a model to not incorporate who wins or loses, but instead to look only at the winning margin.
  • Adjusting the Elo distribution to be "standard" between years: I analyzed Elo distributions in different years here. Using the 2008-2018 average distribution, I was able to get a really solid predictive performance increase in most years in that range. Unfortunately, there were a couple years that saw their predictive power drop with this change. There's certainly potential here though if I ever decide to update parameters each year.

It's a bit frustrating that I don't really have anything concrete to show from all of these investigations other than the addition of 2002-2004 data. I think I'm getting close to maximizing the predictive power I can get using the strict restrictions I have placed on myself, which means it might be time to branch out and break some of those restrictions. Roughly, here are the key restrictions I may loosen in the future to improve my model:
  • Demanding that model parameters stay the same every year: With the exception of the standard deviations, all of my model parameters are the same in 2018 as they were in 2008 or even 2002. The huge advantage of this approach is that I have a model that will probably work pretty darn well in 2019 and beyond, even though we know basically nothing about those games at the moment. The drawback is that I can't incorporate things that clearly improve predictive power in some years if they reduce predictive power in other years. If I loosened this restriction, I would probably restrict myself to measurements I could take using only week 1 data (like stdevs). Possible additions after loosening this restriction would be different team weights or geometric mean usage as well as looking at the Elo distribution change between years.
  • Restricting my analyses only to raw match scores and match type: For much of the history of FRC data, this has been basically the only type of data we have had available, so looking historically this is about as good as we're going to get. However, the addition of twitter score breakdowns (~2009?) opened up the possibility to use other match data, and the detailed API score breakdowns in 2015 expanded on this data drastically. There's a ton of potential to use these data sets to improve upon predictions. I've played around with incorporating these data into my Elo ratings in the past, maybe it's time to officially add some things like this. Looking forward, I think the TBA live scoring breakdowns will also add a whole new dimension to the metrics we can use.
  • Demanding a team's rating be restricted to a single number (my current Elo rating): There are lots of supplementary metrics that I have envisioned that could be added onto raw Elo ratings to provide better predictions in certain scenarios. For example, giving teams a playoff Elo boost to improve playoff predictions or rating the region a team hails from to make better champs predictions. Additions such as these could certainly improve predictive power, but would require that I track extra metrics instead of just the simple Elo rating.

I'm not strictly opposed to loosening any of the above restrictions, although I do think I might make two separate ratings if I do, one for the tighter restrictions and one for the looser restrictions. It's important to me that my model can be quickly applied in a new year, which my current model does very well, and loosening these restrictions will only make the transition to a new year more difficult, but certainly manageable.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 07:28 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi