Championship District Points by Region

FIRST General Forum
1

Following on my obsession with using district points to rank teams at non-district competitions, I wrote a small script to calculate* each teams’ district points at the championships. I also recorded what region (district, state, or country) each team was from and made some graphs from that data. I thought it was interesting, I hope other people do too.

Raw Data:


Team #		DP		Region
254		108		California
148		102		Texas
27		92		FIM
2767		92		FIM
4911		88		PNW
3707		88		FIM
1678		87		California
2910		85		PNW
1619		84		Colorado
1323		83		California
195		83		NE
1241		83		ONT
2056		83		ONT
217		82		FIM
3357		80		FIM
4003		79		FIM
868		75		IN
4488		73		PNW
3476		68		California
1574		66		ISR
494		65		FIM
865		65		ONT
4499		63		Colorado
118		61		Texas
2869		60		New York
4028		59		Ohio
2614		59		West Virginia
3310		58		Texas
1756		58		Illinois
56		58		MAR
1533		57		NC
4541		57		CHS
4967		57		FIM
179		56		Florida
3538		56		FIM
4917		56		ONT
987		55		Nevada
1640		55		MAR
3339		54		ISR
1706		53		Missouri
3646		53		Turkey
2122		53		Idaho
1747		52		IN
141		52		FIM
5987		51		ISR
2655		51		NC
333		51		New York
2590		51		MAR
341		50		MAR
4944		50		Colorado
1296		49		Texas
842		49		Arizona
1114		49		ONT
3674		48		PNW
4237		48		FIM
67		48		FIM
225		48		MAR
1690		47		ISR
1806		47		Missouri
1325		47		ONT
5406		47		ONT
368		46		Hawaii
71		46		IN
3641		46		FIM
3316		45		ISR
5050		45		FIM
2521		44		PNW
694		44		New York
3452		44		FIM
2013		44		ONT
3098		44		FIM
4265		43		Tennessee
4522		43		Missouri
2557		43		PNW
2054		43		FIM
230		43		NE
176		43		NE
340		43		New York
2976		42		PNW
4061		42		PNW
3512		42		California
25		42		MAR
175		41		NE
5572		41		Texas
1648		41		PCH
133		41		NE
16		40		Arkansas
1577		40		ISR
4613		40		Australia
624		40		Texas
3620		40		FIM
1629		40		CHS
2877		40		NE
177		40		NE
687		39		California
3478		39		Mexico
2706		39		ONT
319		39		NE
1986		38		Missouri
359		38		Hawaii
2052		38		Minnesota
1024		38		IN
188		38		ONT
4539		38		Minnesota
469		38		FIM
2481		38		Illinois
772		38		ONT
1817		37		Texas
330		37		California
4451		37		South Carolina
2990		37		PNW
399		37		California
2642		37		NC
610		37		ONT
125		37		NE
1421		36		Mississippi
846		36		California
971		36		California
5254		36		New York
708		36		MAR
3128		35		California
3853		35		Arizona
2383		35		Florida
1731		35		CHS
3536		35		FIM
3005		34		Texas
3374		34		Wyoming
294		34		California
7039		34		ISR
2708		34		ONT
3539		34		FIM
2337		34		FIM
2485		33		California
2974		33		PCH
2992		33		Louisiana
3489		33		South Carolina
973		33		California
3309		33		California
5554		33		ISR
2102		33		California
2471		33		PNW
2502		33		Minnesota
33		33		FIM
4362		33		FIM
1414		32		PCH
4587		32		Texas
1684		32		FIM
3683		32		ONT
3542		32		FIM
5817		31		California
4188		31		PCH
5437		31		Arkansas
6328		31		NE
5813		31		NE
3546		31		FIM
1425		30		PNW
1197		30		California
5190		30		NC
649		30		California
180		30		Florida
5199		30		California
968		30		California
2522		30		PNW
4980		30		PNW
1538		30		California
4776		30		FIM
4253		30		Chinese Taipei
20		30		New York
2826		30		Wisconsin
2451		30		Illinois
1712		30		MAR
6800		29		Texas
744		29		Florida
6025		29		Turkey
1072		29		California
3250		29		California
4191		29		Turkey
5472		29		Florida
862		29		FIM
4855		29		FIM
234		29		IN
4678		29		ONT
5026		28		California
2403		28		Arizona
4513		28		PNW
5818		28		California
2200		28		ONT
3928		28		Iowa
1918		28		FIM
4004		28		FIM
4405		28		FIM
1391		28		MAR
314		28		FIM
5172		28		Minnesota
6502		27		NC
2682		27		NC
3937		27		Arkansas
5496		27		California
3959		27		Alabama
4786		27		Wisconsin
384		27		CHS
3940		27		IN
4946		27		ONT
2168		27		NE
291		27		Pennsylvania
342		26		South Carolina
5803		26		PNW
1156		26		Brazil
3397		26		Missouri
5104		26		California
1730		26		Missouri
1778		26		PNW
6072		26		California
2630		26		ISR
3847		26		Texas
3015		26		New York
4476		26		ONT
5687		26		NE
1519		26		NE
1477		25		Texas
1011		25		Arizona
1102		25		PCH
2075		25		FIM
237		25		NE
3667		25		FIM
5499		24		California
456		24		Mississippi
2046		24		PNW
364		24		Mississippi
7179		24		Texas
3656		24		FIM
1718		24		FIM
3314		24		MAR
910		24		FIM
1305		24		ONT
1796		24		New York
238		24		NE
5675		24		FIM
6429		23		Turkey
1902		23		Florida
2848		23		Texas
2137		23		FIM
2771		23		FIM
1023		23		FIM
2175		23		Minnesota
379		23		Ohio
5460		23		FIM
2791		23		New York
2834		23		FIM
1676		23		MAR
5567		23		FIM
3132		22		Australia
2486		22		Arizona
604		22		California
1710		22		Kansas
2073		22		California
6996		22		Australia
1318		22		PNW
2231		22		ISR
4146		22		Arizona
1259		22		Wisconsin
1559		22		New York
2220		22		Minnesota
302		22		FIM
365		22		MAR
525		22		Iowa
3965		21		California
3284		21		Missouri
1982		21		Kansas
3164		21		Florida
5654		21		ISR
1983		21		PNW
488		21		PNW
4561		21		NC
1923		21		MAR
4557		21		NE
4027		21		Pennsylvania
1625		21		Illinois
7152		20		Florida
1218		20		MAR
2016		20		MAR
870		20		New York
2729		20		MAR
4481		20		Netherlands
2194		20		Wisconsin
2659		19		California
6348		19		Mexico
2202		19		Wisconsin
1885		19		CHS
7198		19		IN
346		19		CHS
747		19		MAR
85		19		FIM
3238		18		PNW
418		18		Texas
5805		18		California
2468		18		Texas
4256		18		Missouri
6106		18		Mexico
6822		18		California
422		18		CHS
5712		18		FIM
5561		18		FIM
836		18		CHS
2415		17		PCH
1939		17		Missouri
4910		17		PCH
3223		17		PNW
6705		17		PCH
7034		17		PNW
1723		17		Missouri
6844		17		Utah
7102		17		Mexico
1322		17		FIM
6875		17		ONT
103		17		MAR
1736		17		Illinois
5505		17		FIM
558		17		NE
70		17		FIM
5517		17		FIM
6868		17		New York
3130		17		Minnesota
1360		17		ONT
2474		17		FIM
4618		17		ONT
343		16		South Carolina
580		16		California
6988		16		Turkey
5630		16		Canada
6672		16		Texas
957		16		PNW
1251		16		Florida
2367		16		California
3158		16		Mexico
3481		16		Texas
3792		16		Missouri
6024		16		India
3663		16		PNW
5584		16		Australia
2377		16		CHS
4392		16		FIM
894		16		FIM
1507		16		New York
3990		16		Canada
4361		16		MAR
5846		16		NE
1741		16		IN
2067		16		NE
236		16		NE
3244		16		Minnesota
401		16		CHS
7048		16		North Dakota
5036		16		ONT
3770		16		FIM
5114		16		FIM
597		16		California
6911		16		New York
1410		15		Colorado
2928		15		PNW
3229		15		NC
6357		15		Texas
6886		15		Missouri
1261		15		PCH
2583		15		Texas
3711		15		PNW
6050		15		Australia
6907		15		China
4468		15		PCH
5663		15		Australia
1002		15		PCH
3882		15		Hawaii
4159		15		California
5429		15		Nevada
120		15		Ohio
2059		15		NC
2478		15		Arizona
4189		15		PCH
6131		15		Arkansas
5407		15		MAR
5724		15		CHS
7329		15		ONT
3276		15		Minnesota
3624		15		New York
88		15		NE
1071		15		NE
263		15		New York
316		15		MAR
5205		15		FIM
3534		15		FIM
663		15		NE
1018		15		IN
3461		15		NE
4198		15		Minnesota
6869		15		Canada
4531		15		Wisconsin
6077		15		FIM
6090		15		FIM
7226		15		FIM
1622		14		California
2152		14		Florida
4795		14		NC
7111		14		Alabama
932		14		Oklahoma
4635		14		Mexico
5145		14		Florida
6831		14		PNW
3495		14		California
6353		14		China
6702		14		Mexico
6885		14		California
2980		14		PNW
3140		14		Tennessee
6424		14		Missouri
6510		14		Australia
1540		14		PNW
5410		14		Florida
5678		14		California
3218		14		PNW
5045		14		Tennessee
5413		14		Ohio
6329		14		NE
1100		14		NE
1816		14		Minnesota
6909		14		Japan
7225		14		FIM
126		14		NE
2638		14		New York
5553		14		France
5603		14		FIM
145		14		New York
245		14		FIM
548		14		FIM
578		14		New York
5883		14		Poland
5914		14		Minnesota
6936		14		Ohio
1792		14		Wisconsin
5528		14		Canada
108		13		Florida
3753		13		Louisiana
3008		13		Hawaii
3653		13		Florida
6813		13		India
4400		13		Mexico
4576		13		Tennessee
6474		13		California
6508		13		Australia
6871		13		Arizona
1178		13		Missouri
5212		13		Texas
5985		13		Australia
972		13		California
2130		13		Idaho
2341		13		Oklahoma
360		13		PNW
5986		13		Texas
6952		13		China
303		13		MAR
447		13		IN
5436		13		FIM
5913		13		Minnesota
2846		13		Minnesota
4130		13		FIM
4384		13		FIM
4814		13		ONT
1189		13		FIM
3236		13		NE
611		13		CHS
623		13		CHS
3844		13		Kentucky
4458		13		New York
5234		13		FIM
1506		13		FIM
533		13		New York
6802		13		CHS
100		12		California
2429		12		California
4192		12		Texas
4501		12		California
7123		12		Arkansas
801		12		Florida
2789		12		Texas
3933		12		Mexico
4010		12		Mexico
6817		12		Missouri
3970		12		California
59		12		Florida
6919		12		PCH
3737		12		NC
2576		12		Chile
2584		12		California
668		12		California
6957		12		Nevada
4984		12		California
6314		12		Arizona
6941		12		China
5216		12		FIM
555		12		MAR
6121		12		FIM
7160		12		FIM
2338		12		Illinois
5546		12		CHS
6334		12		CHS
6548		12		FIM
1599		12		CHS
4903		12		ONT
5822		12		Illinois
6933		12		NE
2170		12		NE
614		12		CHS
888		12		CHS
247		12		FIM
3572		12		FIM
51		12		FIM
6193		12		FIM
6823		12		Wisconsin
1729		12		NE
3623		12		NE
5980		12		FIM
6618		12		FIM
2556		11		Florida
5458		11		California
6413		11		Arizona
6832		11		Mexico
6985		11		Turkey
1311		11		PCH
3598		11		California
5960		11		Texas
6898		11		Texas
3506		11		NC
5291		11		ISR
5414		11		Texas
585		11		California
7173		11		Canada
3473		11		California
5997		11		Louisiana
6377		11		Texas
6645		11		Texas
7134		11		Turkey
1775		11		Missouri
4401		11		Mexico
6652		11		Mexico
6887		11		PCH
3593		11		Oklahoma
4698		11		California
5344		11		Mexico
5417		11		Texas
7287		11		Canada
2177		11		Minnesota
4011		11		Wisconsin
4521		11		Ohio
5674		11		FIM
190		11		NE
3936		11		IN
467		11		NE
5404		11		MAR
1025		11		FIM
3324		11		Ohio
5907		11		FIM
1468		11		New York
2783		11		Kentucky
337		11		West Virginia
3274		11		CHS
3535		11		FIM
4905		11		NE
5443		11		Canada
6947		11		Chinese Taipei
78		11		NE
1073		11		NE
1720		11		IN
2609		11		ONT
4472		11		CHS
573		11		FIM
1369		10		Florida
5006		10		Arkansas
6846		10		Tennessee
4201		10		California
4610		10		Texas
5857		10		California
6824		10		Nevada
7277		10		Canada
1828		10		Arizona
2283		10		Mexico
3075		10		ISR
5801		10		Missouri
980		10		California
4125		10		PNW
4270		10		Hawaii
6814		10		California
7137		10		California
2352		10		Oklahoma
3880		10		Hawaii
6390		10		Canada
5449		10		China
5453		10		China
1660		10		New York
3767		10		FIM
612		10		CHS
6237		10		Illinois
2225		10		Minnesota
2537		10		CHS
4230		10		Minnesota
5123		10		New York
2601		10		New York
4607		10		Minnesota
503		10		FIM
5804		10		CHS
7103		10		Wisconsin
3175		10		FIM
4466		10		CHS
5030		10		New York
5148		10		Wisconsin
6753		10		FIM
1418		10		CHS
3695		10		Illinois
5740		10		Pennsylvania
3038		10		Minnesota
6574		10		Wisconsin
1466		9		Tennessee
3145		9		Idaho
3991		9		Louisiana
6902		9		Brazil
3526		9		Mexico
4091		9		Dominican Republic
6026		9		Oklahoma
1810		9		Kansas
5242		9		Texas
5809		9		Missouri
7157		9		California
1700		9		California
5802		9		California
5849		9		China
832		9		PCH
1515		9		California
3239		9		Utah
4802		9		Australia
1744		9		Florida
2658		9		California
5522		9		China
7079		9		ISR
2830		9		Wisconsin
4909		9		NE
191		9		New York
2987		9		Minnesota
6880		9		New York
167		9		Iowa
3082		9		Minnesota
4976		9		ONT
2512		9		Minnesota
4308		9		ONT
4327		9		FIM
157		9		NE
250		9		New York
2501		9		Minnesota
5434		9		Minnesota
1310		9		ONT
3197		9		Wisconsin
4039		9		ONT
1378		8		Hawaii
3834		8		Texas
4171		8		California
6838		8		Turkey
3103		8		Texas
3230		8		Utah
4205		8		PNW
6168		8		ISR
5839		8		China
6060		8		California
6803		8		China
7293		8		Turkey
2905		8		Turkey
5098		8		Missouri
5700		8		California
7303		8		Texas
812		8		California
2906		8		PNW
3288		8		Wyoming
7081		8		China
4603		8		Mexico
5468		8		PNW
568		8		PNW
1262		8		CHS
2549		8		Minnesota
3381		8		Wisconsin
4096		8		Illinois
2832		8		FIM
4145		8		Ohio
5484		8		IN
7028		8		Minnesota
2265		8		New York
6401		8		New York
7021		8		Wisconsin
3637		8		MAR
7153		8		NE
74		8		FIM
810		8		New York
111		8		Illinois
5339		8		Minnesota
5492		8		Kentucky
977		8		CHS
2169		8		Minnesota
3550		8		Canada
527		8		New York
5829		7		Texas
7301		7		Idaho
4293		7		Colorado
6304		7		China
701		7		California
2096		7		ISR
3019		7		Arizona
5854		7		NC
1868		7		California
5726		7		Texas
6891		7		Oklahoma
7193		7		Florida
7312		7		Texas
4941		7		PCH
6445		7		PNW
6888		7		NC
2081		7		Illinois
2228		7		New York
58		7		NE
2586		7		FIM
3044		7		New York
3604		7		FIM
4342		7		MAR
45		7		IN
4655		7		Illinois
3171		7		New York
3950		7		New York
5962		7		NE
4122		7		New York
4645		7		Illinois
321		7		MAR
4680		7		FIM
5090		7		FIM
1769		6		Kansas
2811		6		PNW
3006		6		Utah
6999		6		Turkey
7106		6		Arkansas
4733		6		Canada
6014		6		Turkey
7140		6		Turkey
3501		6		California
7067		6		ISR
7190		6		Canada
7018		6		Mexico
7327		6		California
4627		6		Canada
5716		6		Mexico
1493		6		New York
3618		6		FIM
5422		6		NE
2604		6		FIM
272		6		MAR
6806		6		New York
4099		6		CHS
6175		6		Minnesota
7250		6		FIM
1076		6		FIM
329		6		New York
5243		6		CHS
3504		6		Pennsylvania
6590		6		New York
4810		6		FIM
4939		6		ONT
743		6		New York
2723		5		Oklahoma
4306		5		Tennessee
647		5		Texas
6833		5		Arizona
1984		5		Kansas
2840		5		Arizona
7128		5		Australia
1763		5		Missouri
2080		5		Louisiana
2637		5		California
5908		5		Texas
6860		5		MAR
6968		5		Netherlands
2234		5		MAR
4811		5		FIM
6964		5		Ohio
1711		5		FIM
857		5		FIM
2500		5		Minnesota
4391		5		FIM
4924		4		Tennessee
2854		4		California
3472		4		Mexico
3390		4		Turkey
7063		4		Texas
5531		4		FIM
5016		4		New York
6032		4		Ohio
1739		4		Illinois
3258		4		CHS
4013		3		Florida
5515		3		China
4775		3		Mexico
6459		3		Turkey
7057		3		California
2620		3		FIM
3359		3		CHS
1555		3		IN
369		3		New York
3117		3		Canada
7068		3		Minnesota

*by calculate I mean pull the pre-calculated values for the sub-divisions from TBA then add 5 points for each match [strike]played[/strike]won on Einstein

2

That makes sense, judging that 40% of Colorado teams that attended Houston made Einstein.

3

Match played or match won?

Also did you remember the Chairmans Award finalists and winners points?

4

Matches won, fixed on original post

…No I’m pretty sure I forgot about them. If I end up redoing the calculation for some reason, I’ll add them.

5

How does a Einstein finalist alliance team have 13?

Sent from my iPhone using Tapatalk

6

According to the rules for district points,

In a bit clearer English, that means if you advance to the next round you get 5 points for every playoff match where you played and won. In playoffs with backups and 4th robots, things get a bit more complicated than 10 points per round. Since 4130 was picked as a 4th robot (not worth any district points) and only played in one playoff match, you only get 5 extra district points despite being an Einstein finalist.

7

So where does the 13 come from if it’s only 5 points?

Sent from my iPhone using Tapatalk

8

This method returned my perception of a reasonable ranking of the teams this year. I think FIRST should be using district points to seed the divisions. (They also could be used to set match schedules.)

9

Probably their qualification ranking.

10

That is a calculated number. I guess it was 8.

Sent from my iPhone using Tapatalk

11

Mississippi exists? :thinking::thinking::thinking:

12

The attached thumbnail is a plot of CMP points vs. Caleb’s winning margin Elo, for 804 teams that competed at the Houston and Detroit 2018 CMP events.

Balancing match schedules based on team strength (ala the 2007 Algorithm of Death) might be more effective if points are used rather than team number, but the shape of the trendline in the attached plot suggests that it might be more effective in balancing “A” flight teams, and less so as a means of ranking the “B” and “C” flight teams.

points v Elo.PNG
points v Elo.PNG1265×729 64.9 KB

13

So I had some time to run some more statistical analyses. First of all, I recalculated all the District Points (DP) data to include points for Chairman’s and Chairman’s finalists. Thanks to “Johnny 4130”, I also found a bug in my calculation of 3rd and 4th robots on Einstein, so I am reposting the correct raw data below (with a number ranking this time). I also calculated DPs just for sub-divisions (i.e. not including Einstein) to make champs DP a bit more comparable to pre-champs DP (this I called DivDP). I then added a new metric for pre-champs events - Adjusted DP (Adj. DP) which is Avg DP*(# Events)^0.3 (to account for the fact that winning 2 events and finalisting at a third is more impressive than winning one event). I also pulled OPR, CCWM, and EPR for pre-champs events from Ether’s excel sheets, and OPR and CCWM for champs sub-divisions from TBA.

With all this new data, I did a bunch of analyses. The first thing I did was to make a box-and-whisker plots for all of the divisions using their pre-champs DP scores. This doesn’t show anything about DP at champs, just another way to show the relative strengths of the sub-divisions going into champs.
https://i.imgur.com/TRTKzypl.jpg](https://i.imgur.com/TRTKzyp.jpg)

I think most people agree that DP is a pretty good indicator of how a team did at that event. I’d like to show that it’s a decent predictor of future events. To do that, I first ran best-fit lines to see how well Adj. DP, OPR, CCWM, EPR predict DP at champs. For those, I got R2 values of 0.398, 0.371, 0.321, and 0.389, respectively.
https://i.imgur.com/Bj1QM3al.jpg](https://i.imgur.com/Bj1QM3a.jpg)

Next, I did the same calculations but this time predicting DivDP. Since this doesn’t add points for Einstein performance, it’s more comparable to the DP values at pre-champs events. The R2 values for those were 0.403, 0.373, 0.322, and 0.389, respectively.
https://i.imgur.com/0UHei06l.jpg](https://i.imgur.com/0UHei06.jpg)

Finally, I looked at using pre-champs Adjusted DP as a predictor for other measures of performance at champs. I ran best-fit lines to see how Adj. DP predicts champs DP, DivDP, OPR, and CCWM. The R2 values for those calculations were 0.398, 0.403, 0.253, and 0.236, respectively.
https://i.imgur.com/SFf7UTdl.jpg](https://i.imgur.com/SFf7UTd.jpg)

In conclusion, though some of the calculations were very close, Adjusted DP was the best predictor of both Champs DP and Champs Division DP. It, however, was not a great predictor of Champs OPR or CCWM. Overall, I think that this new metric is a decent way of predicting future event performance for events where all robots have played at least one event previously (e.g. CMPs and DCMPs).

And as a little bonus, I’m throwing in this graph that shows something that we all already know: the more events you play at, the better you do at champs.
https://i.imgur.com/J8YZIoHl.jpg](https://i.imgur.com/J8YZIoH.jpg)

Here’s the raw data ranking for anyone interested:


Rank		Team #		Champs DP
1		254		108
2		148		102
3		27		92
4		2767		92
5		4911		88
6		3707		88
7		1678		87
8		2910		85
9		1619		84
10		1323		83
11		195		83
12		1241		83
13		2056		83
14		217		82
15		3357		80
16		4003		79
17		3476		78
18		868		75
19		4488		73
20		2976		72
21		1574		66
22		494		65
23		865		65
24		4499		63
25		118		61
26		2869		60
27		4028		59
28		2614		59
29		3310		58
30		1756		58
31		56		58
32		4061		57
33		1533		57
34		4541		57
35		4967		57
36		179		56
37		3538		56
38		4917		56
39		987		55
40		1640		55
41		3339		54
42		1706		53
43		3646		53
44		2122		53
45		1747		52
46		141		52
47		5987		51
48		2655		51
49		333		51
50		2590		51
51		341		50
52		4944		50
53		1296		49
54		842		49
55		1114		49
56		2708		49
57		3674		48
58		4237		48
59		67		48
60		225		48
61		1690		47
62		1806		47
63		1325		47
64		5406		47
65		368		46
66		71		46
67		3641		46
68		3316		45
69		5050		45
70		2521		44
71		694		44
72		3452		44
73		2013		44
74		3098		44
75		4265		43
76		4522		43
77		2557		43
78		2054		43
79		230		43
80		176		43
81		340		43
82		3512		42
83		25		42
84		175		41
85		5572		41
86		1648		41
87		133		41
88		16		40
89		1577		40
90		4613		40
91		624		40
92		3620		40
93		1629		40
94		2877		40
95		177		40
96		3374		39
97		687		39
98		3478		39
99		2706		39
100		319		39
101		1986		38
102		359		38
103		2052		38
104		1024		38
105		188		38
106		4539		38
107		469		38
108		2481		38
109		772		38
110		1817		37
111		330		37
112		4451		37
113		2990		37
114		399		37
115		2642		37
116		610		37
117		125		37
118		1421		36
119		846		36
120		971		36
121		5254		36
122		708		36
123		3128		35
124		3853		35
125		2383		35
126		1731		35
127		3536		35
128		3005		34
129		294		34
130		7039		34
131		1072		34
132		3539		34
133		2337		34
134		2485		33
135		2974		33
136		2992		33
137		3489		33
138		973		33
139		3309		33
140		5554		33
141		2102		33
142		2471		33
143		2502		33
144		33		33
145		4362		33
146		1414		32
147		4587		32
148		1684		32
149		3683		32
150		3542		32
151		5817		31
152		1778		31
153		4188		31
154		5437		31
155		6328		31
156		5813		31
157		4027		31
158		3546		31
159		1425		30
160		1197		30
161		5190		30
162		649		30
163		180		30
164		5199		30
165		968		30
166		2522		30
167		4980		30
168		1538		30
169		4776		30
170		4253		30
171		20		30
172		2826		30
173		2451		30
174		1712		30
175		6800		29
176		744		29
177		6025		29
178		3250		29
179		4191		29
180		5472		29
181		862		29
182		4855		29
183		234		29
184		4678		29
185		5026		28
186		2403		28
187		4513		28
188		5818		28
189		2200		28
190		3928		28
191		1918		28
192		4004		28
193		4405		28
194		1391		28
195		314		28
196		5172		28
197		6502		27
198		2682		27
199		3937		27
200		5496		27
201		3959		27
202		4786		27
203		384		27
204		3940		27
205		4946		27
206		2168		27
207		291		27
208		342		26
209		5803		26
210		1156		26
211		3397		26
212		5104		26
213		1730		26
214		6072		26
215		2630		26
216		3847		26
217		3015		26
218		4476		26
219		5687		26
220		1519		26
221		1477		25
222		1011		25
223		1102		25
224		2075		25
225		237		25
226		3667		25
227		5499		24
228		456		24
229		2046		24
230		364		24
231		7179		24
232		3656		24
233		1718		24
234		3314		24
235		910		24
236		1305		24
237		1796		24
238		238		24
239		5675		24
240		6429		23
241		1902		23
242		2848		23
243		2137		23
244		2771		23
245		1023		23
246		2175		23
247		379		23
248		5460		23
249		2791		23
250		2834		23
251		1676		23
252		5567		23
253		3132		22
254		2486		22
255		604		22
256		1710		22
257		2073		22
258		6996		22
259		1318		22
260		2231		22
261		4146		22
262		1259		22
263		1559		22
264		2220		22
265		302		22
266		365		22
267		525		22
268		3965		21
269		3284		21
270		1982		21
271		3164		21
272		5654		21
273		1983		21
274		488		21
275		4561		21
276		1923		21
277		4557		21
278		1625		21
279		7152		20
280		1218		20
281		2016		20
282		870		20
283		2729		20
284		4481		20
285		2194		20
286		2659		19
287		6348		19
288		2202		19
289		1885		19
290		7198		19
291		346		19
292		747		19
293		85		19
294		3238		18
295		418		18
296		5805		18
297		2468		18
298		4256		18
299		6106		18
300		6822		18
301		4130		18
302		422		18
303		5712		18
304		5561		18
305		836		18
306		2415		17
307		1939		17
308		4910		17
309		3223		17
310		6705		17
311		7034		17
312		1723		17
313		6844		17
314		7102		17
315		1322		17
316		6875		17
317		103		17
318		1736		17
319		5505		17
320		558		17
321		70		17
322		5517		17
323		6868		17
324		3130		17
325		1360		17
326		2474		17
327		4618		17
328		343		16
329		580		16
330		6988		16
331		5630		16
332		6672		16
333		957		16
334		1251		16
335		2367		16
336		3158		16
337		3481		16
338		3792		16
339		6024		16
340		3663		16
341		5584		16
342		2377		16
343		4392		16
344		894		16
345		1507		16
346		3990		16
347		4361		16
348		5846		16
349		1741		16
350		2067		16
351		236		16
352		3244		16
353		401		16
354		7048		16
355		5036		16
356		3770		16
357		5114		16
358		597		16
359		6911		16
360		1410		15
361		2928		15
362		3229		15
363		5006		15
364		6357		15
365		6886		15
366		1261		15
367		2583		15
368		3075		15
369		3711		15
370		6050		15
371		6907		15
372		4468		15
373		5663		15
374		1002		15
375		3882		15
376		4159		15
377		5429		15
378		120		15
379		2059		15
380		2478		15
381		4189		15
382		6131		15
383		5407		15
384		5724		15
385		7329		15
386		3276		15
387		3624		15
388		88		15
389		1071		15
390		263		15
391		316		15
392		5205		15
393		3534		15
394		663		15
395		1018		15
396		3461		15
397		4198		15
398		6869		15
399		4531		15
400		6077		15
401		6090		15
402		7226		15
403		1622		14
404		2152		14
405		4795		14
406		7111		14
407		932		14
408		4635		14
409		5145		14
410		6831		14
411		3495		14
412		6353		14
413		6702		14
414		6885		14
415		2980		14
416		3140		14
417		6424		14
418		6510		14
419		1540		14
420		5410		14
421		5678		14
422		3218		14
423		5045		14
424		5413		14
425		6329		14
426		1100		14
427		1816		14
428		6909		14
429		7225		14
430		126		14
431		2638		14
432		5553		14
433		5603		14
434		145		14
435		245		14
436		548		14
437		578		14
438		5883		14
439		5914		14
440		6936		14
441		1792		14
442		5528		14
443		108		13
444		3753		13
445		3008		13
446		3653		13
447		6813		13
448		4400		13
449		4576		13
450		6474		13
451		6508		13
452		6871		13
453		1178		13
454		5212		13
455		5985		13
456		972		13
457		2130		13
458		2341		13
459		360		13
460		5986		13
461		6952		13
462		303		13
463		447		13
464		5436		13
465		5913		13
466		2846		13
467		4384		13
468		4814		13
469		1189		13
470		3236		13
471		611		13
472		623		13
473		3844		13
474		4458		13
475		5234		13
476		1506		13
477		533		13
478		6802		13
479		100		12
480		2429		12
481		4192		12
482		4501		12
483		7123		12
484		801		12
485		2789		12
486		3933		12
487		4010		12
488		6817		12
489		3970		12
490		59		12
491		6919		12
492		3737		12
493		2576		12
494		2584		12
495		668		12
496		6957		12
497		4984		12
498		6314		12
499		6941		12
500		5216		12
501		555		12
502		6121		12
503		7160		12
504		2338		12
505		5546		12
506		6334		12
507		6548		12
508		1599		12
509		4903		12
510		5822		12
511		6933		12
512		2170		12
513		614		12
514		888		12
515		247		12
516		3572		12
517		51		12
518		6193		12
519		6823		12
520		1729		12
521		3623		12
522		5980		12
523		6618		12
524		2556		11
525		5458		11
526		6413		11
527		6832		11
528		6985		11
529		1311		11
530		3598		11
531		5960		11
532		6898		11
533		3506		11
534		5291		11
535		5414		11
536		585		11
537		7173		11
538		3473		11
539		5997		11
540		6377		11
541		6645		11
542		7134		11
543		1775		11
544		4401		11
545		6652		11
546		6887		11
547		3593		11
548		4698		11
549		5344		11
550		5417		11
551		7287		11
552		2177		11
553		4011		11
554		4521		11
555		5674		11
556		190		11
557		3936		11
558		467		11
559		5404		11
560		1025		11
561		3324		11
562		5907		11
563		1468		11
564		2783		11
565		337		11
566		3274		11
567		3535		11
568		4905		11
569		5443		11
570		6947		11
571		78		11
572		1073		11
573		1720		11
574		2609		11
575		4472		11
576		573		11
577		1369		10
578		6846		10
579		4201		10
580		4610		10
581		5857		10
582		6824		10
583		7277		10
584		1828		10
585		2283		10
586		5801		10
587		980		10
588		4125		10
589		4270		10
590		6814		10
591		7137		10
592		2352		10
593		3880		10
594		6390		10
595		5449		10
596		5453		10
597		1660		10
598		3767		10
599		612		10
600		6237		10
601		2225		10
602		2537		10
603		4230		10
604		5123		10
605		2601		10
606		4607		10
607		503		10
608		5804		10
609		7103		10
610		3175		10
611		4466		10
612		5030		10
613		5148		10
614		6753		10
615		1418		10
616		3695		10
617		5740		10
618		3038		10
619		6574		10
620		1466		9
621		3145		9
622		3991		9
623		6902		9
624		3526		9
625		4091		9
626		6026		9
627		1810		9
628		5242		9
629		5809		9
630		7157		9
631		1700		9
632		5802		9
633		5849		9
634		832		9
635		1515		9
636		3239		9
637		4802		9
638		1744		9
639		2658		9
640		5522		9
641		7079		9
642		2830		9
643		4909		9
644		191		9
645		2987		9
646		6880		9
647		167		9
648		3082		9
649		4976		9
650		2512		9
651		4308		9
652		4327		9
653		157		9
654		250		9
655		2501		9
656		5434		9
657		1310		9
658		3197		9
659		4039		9
660		1378		8
661		3834		8
662		4171		8
663		6838		8
664		3103		8
665		3230		8
666		4205		8
667		6168		8
668		5839		8
669		6060		8
670		6803		8
671		7293		8
672		2905		8
673		5098		8
674		5700		8
675		7303		8
676		812		8
677		2906		8
678		3288		8
679		7081		8
680		4603		8
681		5468		8
682		568		8
683		1262		8
684		2549		8
685		3381		8
686		4096		8
687		2832		8
688		4145		8
689		5484		8
690		7028		8
691		2265		8
692		6401		8
693		7021		8
694		3637		8
695		7153		8
696		74		8
697		810		8
698		111		8
699		5339		8
700		5492		8
701		977		8
702		2169		8
703		3550		8
704		527		8
705		5829		7
706		7301		7
707		4293		7
708		6304		7
709		701		7
710		2096		7
711		3019		7
712		5854		7
713		1868		7
714		5726		7
715		6891		7
716		7193		7
717		7312		7
718		4941		7
719		6445		7
720		6888		7
721		2081		7
722		2228		7
723		58		7
724		2586		7
725		3044		7
726		3604		7
727		4342		7
728		45		7
729		4655		7
730		3171		7
731		3950		7
732		5962		7
733		4122		7
734		4645		7
735		321		7
736		4680		7
737		5090		7
738		1769		6
739		2811		6
740		3006		6
741		6999		6
742		7106		6
743		4733		6
744		6014		6
745		7140		6
746		3501		6
747		7067		6
748		7190		6
749		7018		6
750		7327		6
751		4627		6
752		5716		6
753		1493		6
754		3618		6
755		5422		6
756		2604		6
757		272		6
758		6806		6
759		4099		6
760		6175		6
761		7250		6
762		1076		6
763		329		6
764		5243		6
765		3504		6
766		6590		6
767		4810		6
768		4939		6
769		743		6
770		2723		5
771		4306		5
772		647		5
773		6833		5
774		1984		5
775		2840		5
776		7128		5
777		1763		5
778		2080		5
779		2637		5
780		5908		5
781		6860		5
782		6968		5
783		2234		5
784		4811		5
785		6964		5
786		1711		5
787		857		5
788		2500		5
789		4391		5
790		4924		4
791		2854		4
792		3472		4
793		3390		4
794		7063		4
795		5531		4
796		5016		4
797		6032		4
798		1739		4
799		3258		4
800		4013		3
801		5515		3
802		4775		3
803		6459		3
804		7057		3
805		2620		3
806		3359		3
807		1555		3
808		369		3
809		3117		3
810		7068		3

As always, I’d love to hear what other people think about these new metrics or my analysis thereof.

14

Ari asked me to comment on these metrics and how they compare to my Elo ratings. Here is a chart showing pre-champs Elo vs champs DP:

I’m not surprised that CCWM is the worst predictor of the metrics you tried, I’ve consistently found CCWM to be a worse metric for performance than OPR/EPR. OPR and EPR have similar R^2 values to each other which makes sense since they are strongly correlated. Elo is a much better predictor of champs DP than OPR at R^2=0.465, which is in line with what I have found throughout the year. I don’t know that I would have guessed that pre-champs DP is a better predictor of champs DP than OPR, but it doesn’t really shock me. It’s not really an apples-to-apples comparison. Pre-champs DP actually factors in the auto and climb ranking points, while OPR only looks at scores. That isn’t bad necessarily, it’s just not really a fair comparison in my opinion.

In conclusion, though some of the calculations were very close, Adjusted DP was the best predictor of both Champs DP and Champs Division DP. It, however, was not a great predictor of Champs OPR or CCWM. Overall, I think that this new metric is a decent way of predicting future event performance for events where all robots have played at least one event previously (e.g. CMPs and DCMPs).
As always, I’d love to hear what other people think about these new metrics or my analysis thereof.

Yes, I agree that if you are interested in future district points, past district points is a reasonably good predictor. A better predictor though would probably be to make some kind of metric that incorporates both RP strength and competitive strength.

15

Thanks Caleb!

I agree that the reason pre-champs DP is a better predictor than OPR of champs DP is probably because DP factors in more things than just match score (e.g. extra RPs in quals, defendability and back-to-back matches in elims, and awards). I think in general DP for one competition is a pretty good measure of how good a team did at that competition (arguably better than average match score), so it’s something useful to be predicting. Elo seems to be the best prediction method tested so far by a decent amount, but for an easier to calculate predictor that’s already calculated for every event by TBA anyways, I think using DP as both the predictor and prediction isn’t half bad.

16

I think DP is easily better than average match score as an indicator of team performance. It is also nice to have a metric that is relatively straightforward to calculate. It is really cool to see that this would probably be an even better way to seed divisions than OPR, so we can cross our fingers that FIRST takes a cue from the FIM and Ontario districts and does something like this in the future at the champs level.

I’m kind of interested now in playing around with the district point system to see what changes could be made to it in order to better capture the best performing teams (probably as measured by Elo). From what I’ve seen it’s pretty darn good, and any improvements might come at the expense of simplicity. For example, 1st pick points should probably be worth more than 1+the second pick points, but finding optimal district points for each pick individually would be way more confusing than having the points just be 17-(pick #).

17

So I did a bit more analysis, this time looking at using DP to predict champs quals matches. The method I used was simple: the alliance with the higher total pre-champs adjusted DP wins. This method correctly predicted:

  • 74% on Carver
  • 68% on Galileo
  • 68% on Hopper
  • 75% on Newton
  • 64% on Roebling
  • 80% on Turing
  • 72% on Archemedies
  • 71% on Carson
  • 71% on Curie
  • 73% on Daly
  • 74% on Darwin
  • 72% on Tesla

That’s an average of 72% predicted correctly.

Compare this to a pure OPR prediction, which predicted:

  • 68% on Carver
  • 72% on Galileo
  • 69% on Hopper
  • 76% on Newton
  • 68% on Roebling
  • 79% on Turing
  • 71% on Archimedes
  • 69% on Carson
  • 64% on Curie
  • 75% on Daly
  • 71% on Darwin
  • 72% on Tesla

That’s an average of 71% predicted correctly by OPR.

Overall, adjusted DP predicted a comparable percent of matches to OPR, bust still falls behind some other prediction methods. I also don’t have a good method for calculating win probabilities yet (only W-L-T), so ideas for that would be appreciated.

18

That’s actually very impressive assuming you used pre-champs DP and not champs DP. My earlier caveats about not comparing apples-to-apples don’t apply as much to this analysis since you are just looking at wins and not also the other RPs.

Overall, adjusted DP predicted a comparable percent of matches to OPR, bust still falls behind some other prediction methods. I also don’t have a good method for calculating win probabilities yet (only W-L-T), so ideas for that would be appreciated.

Sum up the DPs for each alliance and subtract the blue alliance’s sum from the red alliance’s to get DP_diff. A positive DP_diff indicates that the red alliance is stronger and a negative DP_diff indicates that the blue alliance is stronger. The further from zero DP_diff is, the more confident we should be that the favorite will win.

There are a couple of ways to use DP_diff to extract win probabilities. My method of choice is a logistic equation: blue win probability = 1/(1+10^(DP_diff/scale))

The trick now is to find the best value for scale to map DP_diff onto win probabilities. If you want to share your data I can show you how to set up a regression. Otherwise you can just try values of scale until you get something that looks right to you, and you’ll probably get within around 30% of the optimal value. My shot in the dark guess for scale would be 1.5*(stdev of DP_diff values over all matches).

19

Yup, that 72% is only using DP from regionals, district events, and DCMPs. Honestly, I was also surprised by how effective the DP predictions were.

I attached a spreadsheet with my predictions, including the true result of the match and a calculation of DP_diff. It’d be great if you could show me how to set up a regression using that data to get a good value for scale to see what kind of brier scores I can get with this model. I also wonder, how do you find a good scale before you have the match results? (Since my model really only works at champs once all teams have played before)

2018_champs_predictions.csv (148 KB)

20

Here is a google sheet which has your values, along with a spot on the right to play around with different scale values (click File->Make a Copy). The optimal value appears to be around 80. You can look at my calculations, but there’s really nothing complicated going on. For every match, I plug in the formula above with the scale value in the sheet. I then find the squared error between the prediction and the match result, and average all of these squared errors to get the MSE. This is exactly the same formula as is used to find Brier scores, although I avoid calling the MSEs Brier scores because we are fitting our formula to old data, not actually predicting anything.

Your model would have stood up well in the Detroit match prediction contest. If you had just calibrated scale based on Houston results, you would have gotten a scale of around 80. Using this value at Detroit would have given you a Brier score of 0.1799, which would have gotten you fourth in the match prediction contest.

I also wonder, how do you find a good scale before you have the match results? (Since my model really only works at champs once all teams have played before)

Well, you have to make an estimate somehow. I already mentioned using Houston to calibrate for Detroit. You could also calibrate using district champs prior to world champs since all teams at these events will be guaranteed to have only teams which have already competed. If you looked back at previous years, I’m sure you would find that scale is relatively (±40%) the same year to year, so you could just use 80 next year and you’ll probably be alright. If you want to use this method earlier in the season, you’ll have to come up with some kind of estimate of DPs for teams before they compete. Last year’s DP would be a solid starting point for this, and you could add in mean reversion or a weighted average of previous season’s DPs if you wanted. I suspect if you have good DP estimates, that scale will stay relatively constant throughout the season.