2015 Championship division simulated rankings

[themccannman]

Originally Posted by Spoam

I had the exact same idea actually. The problem with the OPR residual, however, is that it gives you information about the accuracy of the regression with regard to each match, not each robot.

If you took the set of residuals from the matches a robot played in, it makes intuitive sense that that data should contain some level of information about that robot's deviation from their OPR. But is a data set of only 8-12 elements enough for this value to dominate the noise generated by their alliance partners' deviations (and therefore produce a meaningful standard deviation itself)? I dunno.

If some statistics wiz would like to chime in on this, I'd love to hear it.

The problem with trying to use variation or standard deviation with OPR is that the number it spits out pretty much just tells you what their match schedule was like. OPR is already a calculation of how much an alliances score tends to change when certain teams are playing, calculating standard deviation for that basically just going backwards. OPR tries to determine how one robot affects an alliances score, where as SD (with unique alliances) would give you how each alliance affected that robots score.

Unfortunately it's not very useful unless you have actual scouted data for each team to use, in which case you can make much more accurate predictions about rankings. Our scouting system had a little less than an 80% success rate guessing the winners of each match in our division the last two years, and those games were very defense heavy. I would bet on this system approaching a 95% success rate guessing match results this year since the game is much more consistent.

[Lidor51]

Very intersting, I like this idea. One problem I can see is that there are some teams that their last regional was early in the season (week 1-3), and I think the OPR of those teams won't represent the amount of points they will score at the Championship (they got a lot of time to practice, but it wasn't in an official competition so there isn't any recorded data of their improvement).

Originally Posted by themccannman

The problem with trying to use variation or standard deviation with OPR is that the number it spits out pretty much just tells you what their match schedule was like. OPR is already a calculation of how much an alliances score tends to change when certain teams are playing, calculating standard deviation for that basically just going backwards. OPR tries to determine how one robot affects an alliances score, where as SD (with unique alliances) would give you how each alliance affected that robots score.

Unfortunately it's not very useful unless you have actual scouted data for each team to use, in which case you can make much more accurate predictions about rankings. Our scouting system had a little less than an 80% success rate guessing the winners of each match in our division the last two years, and those games were very defense heavy. I would bet on this system approaching a 95% success rate guessing match results this year since the game is much more consistent.

Pretty high percentages. Can you tell more about the system? What data it's based on, and what are the calculations it does?

[tr6scott]

Originally Posted by Lidor51

Very intersting, I like this idea. One problem I can see is that there are some teams that their last regional was early in the season (week 1-3), and I think the OPR of those teams won't represent the amount of points they will score at the Championship (they got a lot of time to practice, but it wasn't in an official competition so there isn't any recorded data of their improvement).

So can you curve fit the the overall week to week improvement in OPR of the population, and then use that to bias the random factor up for these teams.

Our data for TORC is from Week 7 MSC, so our random is the standard +-10, but team X, data is from week 3 and we know since week OPR overall saw a 20% increase (for example, not actual data) so let the random from team X range from +12 to -8...

Or we could just play the match next week.

[CVR]

I think that your calculation method, which is essentially the following:

Red score = Red1_OPR + Red2_OPR + Red3_OPR

Greatly overestimates qual scores.

I think it might be more accurate to seperate the co-op and auto scores from OPR. In a single match, only one team can do co-op, and only one team can do auto (not entirely true, but pretty close). By counting all 3 team's auto and co-op scores, you're triple-weighting those scores.
Example: Qual 24 has three red teams, each of which have a co-op OPR of 20 (100% consistent 3 tote stack) and an auto OPR of 40 (100% consistent co-op). However, their tote, RC, and litter OPRs are each zero, for a total OPR for each team of 60. The score for this match would be 60, as they would get one auto stack and complete co-op. However, your method predicts the score being 180 points. That's an extreme example, but it illustrates the issue well.
I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)
+ Red2_(toteOPR + binOPR + litterOPR)
+ Red3_(toteOPR + binOPR + litterOPR)
+ MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
+ MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

[Citrus Dad]

Originally Posted by CVR

I think that your calculation method, such is essentially the following:

Red score = Red1_OPR + Red2_OPR + Red3_OPR

Greatly overestimates qual scores.

I think it might be more accurate to seperate the co-op and auto scores from OPR. In a single match, only one team can do co-op, and only one team can do auto (not entirely true, but pretty close). By counting all 3 team's auto and co-op scores, you're triple-weighting those scores.
Example: Qual 24 has three red teams, each of which have a co-op OPR of 20 (100% consistent 3 tote stack) and an auto OPR of 40 (100% consistent co-op). However, their tote, RC, and litter OPRs are each zero, for a total OPR for each team of 60. The score for this match would be 60, as they would get one auto stack and complete co-op. However, your method predicts the score being 180 points. That's an extreme example, but it illustrates the issue well.
I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)
+ Red2_(toteOPR + binOPR + litterOPR)
+ Red3_(toteOPR + binOPR + litterOPR)
+ MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
+ MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

I concur this is correct method. However, it's also important this year to use the Max OPR, not average, as teams improved dramatically through the season. I've got a message into Ed Law on a method to extract the Max Auto and Coop OPRs from this database.

[Spoam]

Originally Posted by CVR

Red Score =
Red1_(toteOPR + binOPR + litterOPR)
+ Red2_(toteOPR + binOPR + litterOPR)
+ Red3_(toteOPR + binOPR + litterOPR)
+ MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
+ MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

Originally Posted by Citrus Dad

I concur this is correct method. However, it's also important this year to use the Max OPR, not average, as teams improved dramatically through the season. I've got a message into Ed Law on a method to extract the Max Auto and Coop OPRs from this database.

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We'll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

[forbes]

Originally Posted by Spoam

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We'll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Thanks!

[Citrus Dad]

Originally Posted by Spoam

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We'll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Wow, thanks!

[forbes]

Originally Posted by Spoam

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We'll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Just to verify: the "Highest OPR (no co-op)" column includes auto?

[Ubiquity]

I see our team is ranked last in Newton. :-( Well, with 118, 1671, and 1678 in Newton, we are definitely a division to watch. All these powerful alliances are going to be looking for a match proven cheese caked can burglar (I hope).

Top alliances with a landfill stacker, a human side stacker and a can burglar are going to be fun to watch. A bonus is a can burglar who can add functionality by stacking or manipulating flipped over totes and cans, or fill in if a top seed malfunctions.

[Spoam]

Originally Posted by forbes

Just to verify: the "Highest OPR (no co-op)" column includes auto?

Yes it does; If you'd like to do what CVR suggested you'd take the "Highest OPR (no coop)" and subtract the auto OPR.

[forbes]

Originally Posted by Spoam

Yes it does; If you'd like to do what CVR suggested you'd take the "Highest OPR (no coop)" and subtract the auto OPR.

Perfect. I updated my website to use this new algorithm: championship.evanforbes.net

[Abhishek R]

Can I just say, thanks to Jeremy for the simulation, 955 for the really nice applet and breakdown OPR for every team and event, and to Evan for his awesome Championship website. All this stuff is really cool to look at, and we appreciate it.

[nuclearnerd]

Originally Posted by CVR

I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)
+ Red2_(toteOPR + binOPR + litterOPR)
+ Red3_(toteOPR + binOPR + litterOPR)
+ MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
+ MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

There's an issue with this method that may skew predictions for alliances with more than one team that does a lot of co-op. Lets say:

R1 has a "platform OPR of 40, and a co-op OPR of 30
R2 has a "platform OPR of 40 and a co-op OPR of 28
R3 has OPRs of 0 for the sake of argument

By the above method, the red alliance would be predicted to score just 110 points, since we would use R1's co-op average, but not R2's. But if R2 can usually score 40 platform points *and do co-op most of the time*, surely they wouldn't put up their 40 and then spend the rest of the match twiddling their thumbs. They would use the time they normally use on co-op to score more platform points!

(There's also the issue of whether the opposing alliance has a high enough Co-op OPR to ensure co-op will be successful, but now we're getting really complicated.)

[Louisiana Jones]

Originally Posted by nuclearnerd

There's an issue with this method that may skew predictions for alliances with more than one team that does a lot of co-op. Lets say:

R1 has a "platform OPR of 40, and a co-op OPR of 30
R2 has a "platform OPR of 40 and a co-op OPR of 28
R3 has OPRs of 0 for the sake of argument

By the above method, the red alliance would be predicted to score just 110 points, since we would use R1's co-op average, but not R2's. But if R2 can usually score 40 platform points *and do co-op most of the time*, surely they wouldn't put up their 40 and then spend the rest of the match twiddling their thumbs. They would use the time they normally use on co-op to score more platform points!

(There's also the issue of whether the opposing alliance has a high enough Co-op OPR to ensure co-op will be successful, but now we're getting really complicated.)

Is the max coopOPR used for the same event as the Max OPR? If not I see some problems where it may appear that teams score more than they really do, if they coop more at some events, and then stacked more at other events. They would be getting the max points from both.