View Single Post
  #10   Spotlight this post!  
Unread 16-05-2015, 16:31
James Kuszmaul James Kuszmaul is offline
NEFIRST CSA
FRC #0971 (Spartan Robotics)
 
Join Date: Jan 2012
Rookie Year: 2011
Location: Worcester, MA
Posts: 61
James Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud ofJames Kuszmaul has much to be proud of
Re: "standard error" of OPR values

Quote:
Originally Posted by wgardner View Post
Here's a poor-man's approach to approximating the error of the OPR value calculation (as opposed to the prediction error aka regression error):

1. Collect all of a team's match results.

2. Compute the normal OPR.

3. Then, re-compute the OPR but excluding the result from the first match.

4. Repeat this process by removing the results from only the 2nd match, then only the 3rd, etc. This will give you a set of OPR values computed by excluding a single match. So for example, if a team played 6 matches, there would be the original OPR plus 6 additional "OPR-" values.

5. Compute the standard deviation of the set of OPR- values. This should give you some idea of how much variability a particular match contributes to the team's OPR. Note that this will even vary team-by-team.

Thoughts?
Using Ether's data, I just did essentially this, where I randomly* selected 200 (I just chose this because it excludes enough matches to ensure variation in OPRs, but should include enough matches to keep the system sufficiently over-determined) of the 254 alliance scores to use for the OPR calculation. I ran this 200 times and got the following:
Code:
Team	Original OPR	Mean OPR	Standard Deviation	StdDev / Mean
1023	119.9222385	120.0083320153	11.227427964	0.0935554038
234	73.13049299	72.801129356	8.9138064084	0.1224404963
135	71.73803792	72.0499437529	7.953512079	0.1103888728
1310	68.29454232	69.3467152712	14.1978070751	0.2047365476
1538	66.51660956	65.739882921	10.0642899215	0.1530926049
1640	63.89355804	63.1124212044	12.5486944006	0.1988308191
4213	59.83218159	60.3799737845	9.7581954471	0.1616131117
2383	59.3454496	58.4390556944	8.8170835924	0.1508765583
5687	58.89565276	58.0801454327	8.5447703278	0.1471203328
2338	57.52050487	57.8998084926	9.9345796042	0.1715822533
68	57.31570571	57.5000280561	7.3734953486	0.1282346391
2342	56.91016998	57.2987212179	6.6038945531	0.115253786
2974	55.52108592	57.1342122847	8.3752237419	0.1465885921
857	56.58983207	56.5258351411	7.2736015551	0.1286774718
2619	55.87939909	55.7690519681	8.4202867997	0.150984937
314	54.93283739	54.2189755764	9.2781646413	0.1711239385
4201	54.36868175	53.4393101098	10.5474638148	0.1973727541
2907	52.20131966	52.8528874425	7.542822466	0.1427135362
360	50.27624758	50.4115562132	7.0992892482	0.1408266235
5403	50.29915841	50.3683881678	6.7117433122	0.133253089
201	45.9115291	44.7743914139	8.4846178186	0.189497111
2013	44.91032156	44.6243506137	6.8765159824	0.1540978387
3602	44.27190346	44.0845482182	9.1690079569	0.2079868872
207	43.76003325	43.534273676	9.6975195297	0.2227559739
1785	42.88695283	43.4312399486	8.2699452851	0.1904146714
1714	43.01192386	42.548981107	10.4744349747	0.2461735793
2848	42.09926229	42.3315382699	5.5963086425	0.1322018729
5571	41.52437471	41.7434170692	9.1647109829	0.2195486528
3322	41.46602143	41.5494849767	7.1743838875	0.1726708259
4334	40.44991373	41.05033774	8.7102627815	0.2121849237
5162	40.45440709	40.9929568271	8.2624477928	0.2015577414
5048	39.89000748	40.3308767357	11.0199899828	0.2732395344
2363	39.94545778	40.1152579819	6.6177263936	0.1649678134
280	39.5619946	39.5341268065	7.3717432763	0.1864653117
4207	38.2684727	39.4991498122	6.9528849981	0.1760261938
5505	39.67352888	38.9668291926	11.3348728596	0.2908851732
217	36.77649547	37.4492632177	6.4891284445	0.1732778668
836	36.43648963	37.0437210956	12.1307341233	0.3274707228
503	36.81699351	36.7802949819	7.9491833149	0.2161261436
1322	36.38199798	36.7254993257	8.5268395114	0.2321776332
4451	35.19372256	35.3483644749	9.807710599	0.2774586815
623	34.52165055	35.1189107974	7.930898959	0.2258298671
1648	35.50610406	35.0638323174	10.815198205	0.3084431304
51	34.66010328	34.6703806244	5.4485310273	0.157152328
122	34.32806143	33.5962803896	7.5092149942	0.223513285
115	31.91437124	31.3399395607	8.4108320311	0.2683742263
5212	30.01729221	30.4525516362	8.9862156315	0.2950890861
1701	29.87650404	30.3212455768	6.3833025833	0.2105224394
3357	29.17742219	29.6022237315	6.381280757	0.2155676146
1572	29.88934385	29.5148636895	7.882621955	0.2670729582
3996	29.80296599	29.071104692	12.1221539603	0.4169829144
2655	26.12997208	26.8414199039	8.2799141902	0.3084752677
3278	27.75400612	26.676383757	8.7090459236	0.3264702593
2605	26.77170149	26.4416718205	7.2093344642	0.2726504781
2914	25.16358084	25.6405460981	8.2266061339	0.3208436397
5536	25.12712518	25.537683706	8.9692243899	0.3512152666
108	25.12900331	24.9994393089	8.1059495087	0.3242452524
4977	23.84091367	24.1678220977	8.8309117942	0.3653995697
931	20.64386303	20.6395850124	9.7862519781	0.4741496485
3284	20.6263851	20.3004828941	7.7358872421	0.3810691244
5667	20.24853487	20.2012572648	10.5728126478	0.5233739915
188	19.63432177	19.5009951172	8.527091207	0.4372644142
5692	17.52522898	16.9741593261	9.9533189003	0.5863806689
1700	15.35451961	15.0093164719	7.5208523959	0.5010789405
4010	12.26210563	13.9952121466	9.8487154699	0.7037203414
1706	12.6972477	11.7147928015	6.1811481569	0.5276361487
3103	12.14379904	11.6822069225	8.4008681879	0.7191165371
378	11.36567533	11.6581748916	8.2483175766	0.7075136248
3238	8.946537399	9.2298154231	9.6683698675	1.0475149745
5581	9.500192257	8.7380812257	8.2123397521	0.9398333044
5464	4.214298451	5.4505495437	7.2289498778	1.326279088
41	5.007828439	4.3002816244	9.0353666405	2.1011104457
2220	4.381189923	4.2360658386	6.880055327	1.6241615662
4364	4.923793169	3.504087428	8.6917749423	2.4804674886
1089	1.005273551	0.9765385053	6.9399339807	7.1066670109
691	-1.731531162	-1.2995295456	11.9708242834	9.2116599609
Original OPR is just copied straight from Ether's OPR.csv; Mean OPR is just the mean OPR that a team received across the 200 iterations; Standard Deviation is the Standard Deviation of all the OPRs a team recieved and the final column is just the standard deviation divided by the Mean OPR. The data is sorted by Mean OPR.

In terms of whether this is a valid way of looking at it, I'm not sure--the results seem to have some meaning, but I'm not sure how much of it is just that only looking at 200 scores is even worse than just 254, or if there is something more meaningful going on.

*Using python's random.sample() function. This means that I did nothing to prevent duplicate runs (which are extremely unlikely; 254 choose 200 is ~7.2 * 10^55) and nothing to ensure that a team didn't "play" <3 times in the selection of 200 scores.
__________________
FRC971 (Student) 2011-2014
FRC190 (College Mentor-ish) 2014
WPILib Development 2014-present
Reply With Quote