FRCTop25 Week 1 Poll Open. Closes 3/5 7:00pm Eastern

[Bryce2471]

Instead of OPR, I’d like to see a metric similar to but not exactly the same as CCWM. Pardon my crude math, but imagine a metric that approximately solved a matrix of equations for each match along the lines of MatchPointSpread = Red1 + Red2 + Red3 - Blue1 - Blue2 - Blue3.

Unlike OPR / DPR, the other alliance (and the other alliance’s score) is included, so it would take into account how the strength of the opponents could depress the margin.

---

I’ve been thinking of making this metric for a long time. (By that I mean several years) But I haven’t had the time to do it myself. However, some statistitions have assured me that it would come out the same as CCWM. I highly encourage you to try it and find out for sure.

[Tyler_Olds]

To build off of others on this. FUN is 100% on board with having the FRC Top 25 weighted by some sort of objective metric. Really the question is what would be accepted by the majority and how can I have a report ran instantly so it isn’t any more of a hindrance when compiling the FRC Top 25.

I will say that while teams will always slip through the cracks each week that overall this is one of the more accurate community polls IMHO and I believe having around 300 submissions helps with this immensely.

Instead of OPR, I’d like to see a metric similar to but not exactly the same as CCWM. Pardon my crude math, but imagine a metric that approximately solved a matrix of equations for each match along the lines of MatchPointSpread = Red1 + Red2 + Red3 - Blue1 - Blue2 - Blue3.

Unlike OPR / DPR, the other alliance (and the other alliance’s score) is included, so it would take into account how the strength of the opponents could depress the margin.

---

As for the actual topic of this thread:

• I don’t like using ELO for rankings since it picks up where last year left off (right?) rather than making new numbers each year.
• 2791 is dope as hell and definitely a top team, pardon my bias
• People aren’t mentioning the traditional powerhouses because they don’t need to mention them in order for others to pay attention

[jboynton6329]

I just wanted to say how much I enjoyed the set up of the region recaps last night! It was a great addition to an already great program. My kids were beyond thrilled to get a mention. Keep up the great work FUN!

[jtrv]

Instead of OPR, I’d like to see a metric similar to but not exactly the same as CCWM. Pardon my crude math, but imagine a metric that approximately solved a matrix of equations for each match along the lines of MatchPointSpread = Red1 + Red2 + Red3 - Blue1 - Blue2 - Blue3.

Unlike OPR / DPR, the other alliance (and the other alliance’s score) is included, so it would take into account how the strength of the opponents could depress the margin.

edit: i may have done ccwm by accident. looking into code now ¯_(ツ)_/¯

Here’s the results for CNY for this metric (quals only):

340   118.06
5254  112.06
2791  108.15
319   107.2
639   89.16
4253  74.25
694   69.02
287   42.46
145   41.56
27    37.06
3003  33.39
20    25.55
3044  18.88
1518  13.89
6422  11.85
527   -1.22
378   -3.99
4027  -4.07
191   -5.51
5484  -12.22
2016  -18.25
810   -20.83
4122  -29.23
514   -32.12
250   -45.66
358   -54.19
6300  -59.72
7081  -59.93
223   -63.0
2053  -63.32
5030  -69.22
3173  -72.68
6621  -73.48
1450  -91.81
1665  -122.09

Or, including elims:

340   136.36   
2791  126.27   
5254  84.56    
319   74.65    
4253  64.63    
694   56.95    
639   54.75    
287   32.87    
27    26.39    
145   25.65    
1518  23.59    
3044  9.29     
20    5.4      
6422  -0.25    
527   -4.24    
4027  -4.66    
3003  -7.62    
191   -8.78    
378   -16.49   
5484  -19.86   
4122  -26.1    
514   -31.09   
810   -38.58   
6300  -42.17   
2016  -45.29   
223   -47.96   
5030  -48.03   
250   -50.78   
2053  -55.12   
7081  -55.7    
6621  -62.17   
3173  -63.79   
1450  -74.58   
358   -83.73   
1665  -106.12

[Caleb_Sykes]

Sorry for the long post, but I have a lot of thoughts on below topics.

My concern with OPR this year is essentially similar to Kevin’s, there just not enough sample size. Similar to other games in which teams can contribute meaningfully to the score without obvious quantitative interactions (2009, 2014), there is some inherent value to a metric that seeks to infer the offensive impact of teams beyond the quantity of game pieces they score. However, also similar to 2009 and 2014, so much of the scoring potential of a team depends on their alliance partners and opponents in a given match. I feel there’s simply too much noise in qualification schedule to get a truly accurate OPR measurement.

I’ll be looking in the next few weeks at comparing the predictive power of OPR to Elo for 2018, and compare the predictive power to previous years. People are free to use any metric they like to rank robots, but essentially the only rankings I care about are metrics that are backed by predictive power. My Elo model was purely designed for this purpose. To my knowledge, OPR was not designed with predictive power in mind, but it has turned out to have by far the best predictive power of any common metric, so it is still the gold standard in my mind. That doesn’t mean it isn’t better or worse in some years than others, but I found it’s predictive power was strictly greater than that of the other metrics for every year in 2008-2016. It’s a safe bet it will also be the best metric this year.

As for the actual topic of this thread:

• I don’t like using ELO for rankings since it picks up where last year left off (right?) rather than making new numbers each year.

I could reset my Elo ratings every year, but nearly everything in my model is done to maximize predictive power for matches, and doing this would severely limit the predictive power of the model, particularly in early weeks of each season. Each team’s Elo in my current model is found by taking 70% of their 2017 end of season Elo and adding 30% of their 2016 end of season Elo, and then reverting this sum by 20% toward 1550 (the pseudo-average Elo). If it were more predictive to revert everyone 100% after each season I would have found this during model tuning. I get that some people might view this as unfair, but my goal is to maximize predictive power, no matter where that takes me, not to make a metric that seems intuitively fair.

I’ve been thinking of making this metric for a long time. (By that I mean several years) But I haven’t had the time to do it myself. However, some statistitions have assured me that it would come out the same as CCWM. I highly encourage you to try it and find out for sure.

We just had a conversation on this very topic here. I didn’t actually go and check for equivalence, but a few years back wgardner did and found the measurements to be distinct. I’m seriously concerned with overfitting with this metric though and it’s predictive power was very poor from my testing, I like EPR a lot better if you want to incorporate who opponents are.

To build off of others on this. FUN is 100% on board with having the FRC Top 25 weighted by some sort of objective metric. Really the question is what would be accepted by the majority and how can I have a report ran instantly so it isn’t any more of a hindrance when compiling the FRC Top 25.

I will say that while teams will always slip through the cracks each week that overall this is one of the more accurate community polls IMHO and I believe having around 300 submissions helps with this immensely.

The only metric that would qualify for this privilege is OPR in my opinion. I’ve been working hard to get Elo to be as good or better of a metric than OPR, but considering I’m the only one who calculates it and it doesn’t have nearly as strong of a track record as OPR, it would be a poor choice to use it this year. As I stated above, OPR has consistently been the most predictive metric in FRC. I think it would be cool if the weightings were weighted half by OPR and half by polling. Neither method is anywhere close to perfect, but I think they could complement each other well. I often feel bad when amazing teams with little name recognition attending obscure events don’t make the list, when they are probably better than some teams that did make it.

[WTFogland]

Coming off the win from MVR, I would definitely agree that 4028 was a robot to contend with. They definitely have a great design, and compared to my team’s, a very similar design haha. I think 379 also has a great robot, with that triple climb working its way into the line-up. 302, being my team’s alliance captain, played smart throughout quals and definitely should be recognized for a great single climb that was fast and sturdy.

Although I haven’t spoke about my team, don’t count us out for coming in strong for our Michigan Events Ohio was just a warm-up.

Overall, had a great time at MVR and I think that a chunk of robots from that regional should not be forgotten.

[Bryce2471]

Sorry for the long post, but I have a lot of thoughts on below topics.

I’ll be looking in the next few weeks at comparing the predictive power of OPR to Elo for 2018, and compare the predictive power to previous years. People are free to use any metric they like to rank robots, but essentially the only rankings I care about are metrics that are backed by predictive power. My Elo model was purely designed for this purpose. To my knowledge, OPR was not designed with predictive power in mind, but it has turned out to have by far the best predictive power of any common metric, so it is still the gold standard in my mind. That doesn’t mean it isn’t better or worse in some years than others, but I found it’s predictive power was strictly greater than that of the other metrics for every year in 2008-2016. It’s a safe bet it will also be the best metric this year.

I could reset my Elo ratings every year, but nearly everything in my model is done to maximize predictive power for matches, and doing this would severely limit the predictive power of the model, particularly in early weeks of each season. Each team’s Elo in my current model is found by taking 70% of their 2017 end of season Elo and adding 30% of their 2016 end of season Elo, and then reverting this sum by 20% toward 1550 (the pseudo-average Elo). If it were more predictive to revert everyone 100% after each season I would have found this during model tuning. I get that some people might view this as unfair, but my goal is to maximize predictive power, no matter where that takes me, not to make a metric that seems intuitively fair.

We just had a conversation on this very topic here. I didn’t actually go and check for equivalence, but a few years back wgardner did and found the measurements to be distinct. I’m seriously concerned with overfitting with this metric though and it’s predictive power was very poor from my testing, I like EPR a lot better if you want to incorporate who opponents are.

The only metric that would qualify for this privilege is OPR in my opinion. I’ve been working hard to get Elo to be as good or better of a metric than OPR, but considering I’m the only one who calculates it and it doesn’t have nearly as strong of a track record as OPR, it wouldn be a poor choice to use it this year. As I stated above, OPR has consistently been the most predictive metric in FRC. I think it would be cool if the weightings were weighted half by OPR and half by polling. Neither method is anywhere close to perfect, but I think they could complement each other well. I often feel bad when amazing teams with little name recognition attending obscure events don’t make the list, when they are probably better than some teams that did make it.

Thanks for the post. It was highly informative. You make it sound like you have done some serious investigation of the predictive power of different metrics. If that is the case, could you point me in the direction of your findings? I am particularly interested in the relative power of OPR, WMPR, and EPR.

[Caleb_Sykes]

Thanks for the post. It was highly informative. You make it sound like you have done some serious investigation of the predictive power of different metrics. If that is the case, could you point me in the direction of your findings? I am particularly interested in the relative power of OPR, WMPR, and EPR.

https://www.chiefdelphi.com/media/papers/3315

[rjtbob]

here. I didn’t actually go and check for equivalence, but a few years back wgardner did and found the measurements to be distinct. I’m seriously concerned with overfitting with this metric though and it’s predictive power was very poor from my testing, I like EPR a lot better if you want to incorporate who opponents are.

The only metric that would qualify for this privilege is OPR in my opinion. I’ve been working hard to get Elo to be as good or better of a metric than OPR, but considering I’m the only one who calculates it and it doesn’t have nearly as strong of a track record as OPR, it wouldn be a poor choice to use it this year. As I stated above, OPR has consistently been the most predictive metric in FRC. I think it would be cool if the weightings were weighted half by OPR and half by polling. Neither method is anywhere close to perfect, but I think they could complement each other well. I often feel bad when amazing teams with little name recognition attending obscure events don’t make the list, when they are probably better than some teams that did make it.

Took a stab at gleaning the top 25 OPR scores for week one (done by hand, so apologies in advance if I’ve missed someone or a typo crept in). Given Caleb’s plug for OPR, I will be interested in seeing how this list compares with the popular polling tonight (noting the hesitation by many to use OPR comparisons across events).

OPR week 1 Team
311.46 148 272.81 610 234.78 2046 228.24 4513 226.31 3478 223.24 118 216.25 3005 215.59 1574 212.00 238 210.47 1360 209.45 1918 208.92 4476 201.04 3990 200.27 359 198.98 829 196.73 1325 195.58 1339 195.49 6705 195.08 3750 194.68 379 192.86 2848 189.70 5472 188.88 1772 188.34 1058 187.37 3538

[Richard_McCann]

I posted this also in another thread: while the OPR WITHIN an event likely as significant predictive power, the OPRs ACROSS events are likely almost meaningless. This year’s score is highly dependent on the relative strengths of each competition. One regional might be dominated by one bot that can put up only 2 scale cubes, while another might have several that can put up an average of 4 cubes, but they split up the ownership time. Which bots have the highest offensive output? The former likely has a higher OPR.

[rjtbob]

I posted this also in another thread: while the OPR WITHIN an event likely as significant predictive power, the OPRs ACROSS events are likely almost meaningless. This year’s score is highly dependent on the relative strengths of each competition. One regional might be dominated by one bot that can put up only 2 scale cubes, while another might have several that can put up an average of 4 cubes, but they split up the ownership time. Which bots have the highest offensive output? The former likely has a higher OPR.

Not in disagreement with this in general. But I have nevertheless been impressed with the overall effectiveness of bots this early in the season, and would suggest that the dominant bot in your example might only be putting up two scale cubes for strategic reasons, not necessarily due to lack of ability. Despite the limitations of OPR, it will still be interesting to see how many of its top 25 rank in the top 25 of the popular poll.

[Tyler_Olds]

Starting the show in a bit! Come join us.

[rjtbob]

For whatever it’s worth, 12 of the 25 in the popular poll also had top 25 OPR scores week one (188 and above). The other 13 had OPR scores ranging from 118 to 183, or 161 on average (118 being the outrider). I don’t know whether that would affect anyone’s opinion about the relevance (most thinking irrelevant) of OPR across events.

[Ryan_Swanson]

For whatever it’s worth, 12 of the 25 in the popular poll also had top 25 OPR scores week one (188 and above). The other 13 had OPR scores ranging from 118 to 183, or 161 on average (118 being the outrider). I don’t know whether that would affect anyone’s opinion about the relevance (most thinking irrelevant) of OPR across events.

You’re going to cause a Ruckus if you keep talking about 118 like that.