2015 Championship division simulated rankings

[themccannman]

I had the exact same idea actually. The problem with the OPR residual, however, is that it gives you information about the accuracy of the regression with regard to each match, not each robot.

If you took the set of residuals from the matches a robot played in, it makes intuitive sense that that data should contain some level of information about that robot’s deviation from their OPR. But is a data set of only 8-12 elements enough for this value to dominate the noise generated by their alliance partners’ deviations (and therefore produce a meaningful standard deviation itself)? I dunno.

If some statistics wiz would like to chime in on this, I’d love to hear it.

The problem with trying to use variation or standard deviation with OPR is that the number it spits out pretty much just tells you what their match schedule was like. OPR is already a calculation of how much an alliances score tends to change when certain teams are playing, calculating standard deviation for that basically just going backwards. OPR tries to determine how one robot affects an alliances score, where as SD (with unique alliances) would give you how each alliance affected that robots score.

Unfortunately it’s not very useful unless you have actual scouted data for each team to use, in which case you can make much more accurate predictions about rankings. Our scouting system had a little less than an 80% success rate guessing the winners of each match in our division the last two years, and those games were very defense heavy. I would bet on this system approaching a 95% success rate guessing match results this year since the game is much more consistent.

[Lidor51]

Very intersting, I like this idea. One problem I can see is that there are some teams that their last regional was early in the season (week 1-3), and I think the OPR of those teams won’t represent the amount of points they will score at the Championship (they got a lot of time to practice, but it wasn’t in an official competition so there isn’t any recorded data of their improvement).

The problem with trying to use variation or standard deviation with OPR is that the number it spits out pretty much just tells you what their match schedule was like. OPR is already a calculation of how much an alliances score tends to change when certain teams are playing, calculating standard deviation for that basically just going backwards. OPR tries to determine how one robot affects an alliances score, where as SD (with unique alliances) would give you how each alliance affected that robots score.

Unfortunately it’s not very useful unless you have actual scouted data for each team to use, in which case you can make much more accurate predictions about rankings. Our scouting system had a little less than an 80% success rate guessing the winners of each match in our division the last two years, and those games were very defense heavy. I would bet on this system approaching a 95% success rate guessing match results this year since the game is much more consistent.

Pretty high percentages. Can you tell more about the system? What data it’s based on, and what are the calculations it does?

[MechEng83]

I had the exact same idea actually. The problem with the OPR residual, however, is that it gives you information about the accuracy of the regression with regard to each match, not each robot.

If you took the set of residuals from the matches a robot played in, it makes intuitive sense that that data should contain some level of information about that robot’s deviation from their OPR. But is a data set of only 8-12 elements enough for this value to dominate the noise generated by their alliance partners’ deviations (and therefore produce a meaningful standard deviation itself)? I dunno.

If some statistics wiz would like to chime in on this, I’d love to hear it.

The problem with trying to use variation or standard deviation with OPR is that the number it spits out pretty much just tells you what their match schedule was like. OPR is already a calculation of how much an alliances score tends to change when certain teams are playing, calculating standard deviation for that basically just going backwards. OPR tries to determine how one robot affects an alliances score, where as SD (with unique alliances) would give you how each alliance affected that robots score.

Unfortunately it’s not very useful unless you have actual scouted data for each team to use, in which case you can make much more accurate predictions about rankings. Our scouting system had a little less than an 80% success rate guessing the winners of each match in our division the last two years, and those games were very defense heavy. I would bet on this system approaching a 95% success rate guessing match results this year since the game is much more consistent.

Good points. Thanks for pointing out the flaw in my idea. what I surmise is that this calculation of stdev would be marginally useful at best. This reminds me of a mantra I hear at work quite often: “All models are wrong. Some models are useful.”

[tr6scott]

Very intersting, I like this idea. One problem I can see is that there are some teams that their last regional was early in the season (week 1-3), and I think the OPR of those teams won’t represent the amount of points they will score at the Championship (they got a lot of time to practice, but it wasn’t in an official competition so there isn’t any recorded data of their improvement).

So can you curve fit the the overall week to week improvement in OPR of the population, and then use that to bias the random factor up for these teams.

Our data for TORC is from Week 7 MSC, so our random is the standard ±10, but team X, data is from week 3 and we know since week OPR overall saw a 20% increase (for example, not actual data) so let the random from team X range from +12 to -8…

Or we could just play the match next week.

[Matt_G]

I think that your calculation method, which is essentially the following:

Red score = Red1_OPR + Red2_OPR + Red3_OPR

Greatly overestimates qual scores.

I think it might be more accurate to seperate the co-op and auto scores from OPR. In a single match, only one team can do co-op, and only one team can do auto (not entirely true, but pretty close). By counting all 3 team’s auto and co-op scores, you’re triple-weighting those scores.
Example: Qual 24 has three red teams, each of which have a co-op OPR of 20 (100% consistent 3 tote stack) and an auto OPR of 40 (100% consistent co-op). However, their tote, RC, and litter OPRs are each zero, for a total OPR for each team of 60. The score for this match would be 60, as they would get one auto stack and complete co-op. However, your method predicts the score being 180 points. That’s an extreme example, but it illustrates the issue well.
I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)

• Red2_(toteOPR + binOPR + litterOPR)
• Red3_(toteOPR + binOPR + litterOPR)
• MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
• MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

[Joe_Ross]

What probability distribution did you use for the random terms?

[Richard_McCann]

I think that your calculation method, such is essentially the following:

Red score = Red1_OPR + Red2_OPR + Red3_OPR

Greatly overestimates qual scores.

I think it might be more accurate to seperate the co-op and auto scores from OPR. In a single match, only one team can do co-op, and only one team can do auto (not entirely true, but pretty close). By counting all 3 team’s auto and co-op scores, you’re triple-weighting those scores.
Example: Qual 24 has three red teams, each of which have a co-op OPR of 20 (100% consistent 3 tote stack) and an auto OPR of 40 (100% consistent co-op). However, their tote, RC, and litter OPRs are each zero, for a total OPR for each team of 60. The score for this match would be 60, as they would get one auto stack and complete co-op. However, your method predicts the score being 180 points. That’s an extreme example, but it illustrates the issue well.
I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)

• Red2_(toteOPR + binOPR + litterOPR)
• Red3_(toteOPR + binOPR + litterOPR)
• MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
• MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

I concur this is correct method. However, it’s also important this year to use the Max OPR, not average, as teams improved dramatically through the season. I’ve got a message into Ed Law on a method to extract the Max Auto and Coop OPRs from this database.

[Spoam]

Red Score =
Red1_(toteOPR + binOPR + litterOPR)

• Red2_(toteOPR + binOPR + litterOPR)
• Red3_(toteOPR + binOPR + litterOPR)
• MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
• MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I concur this is correct method. However, it’s also important this year to use the Max OPR, not average, as teams improved dramatically through the season. I’ve got a message into Ed Law on a method to extract the Max Auto and Coop OPRs from this database.

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We’ll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

[forbes]

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We’ll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Thanks!

[Richard_McCann]

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We’ll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Wow, thanks!

[forbes]

We have all this data available here if anyone wants to use it. Click the download button to get a local copy in csv format. We’ll run our own monte carlo analysis later today using various models and post it here, along with some regression analysis on which model gives the most accurate predictions, just for fun.

Just to verify: the “Highest OPR (no co-op)” column includes auto?

[Ubiquity]

I see our team is ranked last in Newton. Well, with 118, 1671, and 1678 in Newton, we are definitely a division to watch. All these powerful alliances are going to be looking for a match proven cheese caked can burglar (I hope).

Top alliances with a landfill stacker, a human side stacker and a can burglar are going to be fun to watch. A bonus is a can burglar who can add functionality by stacking or manipulating flipped over totes and cans, or fill in if a top seed malfunctions.

[Spoam]

Just to verify: the “Highest OPR (no co-op)” column includes auto?

Yes it does; If you’d like to do what CVR suggested you’d take the “Highest OPR (no coop)” and subtract the auto OPR.

[forbes]

Yes it does; If you’d like to do what CVR suggested you’d take the “Highest OPR (no coop)” and subtract the auto OPR.

Perfect. I updated my website to use this new algorithm: championship.evanforbes.net

[Abhishek_R]

Can I just say, thanks to Jeremy for the simulation, 955 for the really nice applet and breakdown OPR for every team and event, and to Evan for his awesome Championship website. All this stuff is really cool to look at, and we appreciate it.

[nuclearnerd]

I think a better method would be to use the following:

Red Score =
Red1_(toteOPR + binOPR + litterOPR)

• Red2_(toteOPR + binOPR + litterOPR)
• Red3_(toteOPR + binOPR + litterOPR)
• MAX(Red1_autoOPR, Red2_autoOPR, Red3_autoOPR)
• MAX(Red1_coopOPR, Red2_coopOPR, Red3_coopOPR)

I think that method, while slightly more complex, will give more accurate results.

There’s an issue with this method that may skew predictions for alliances with more than one team that does a lot of co-op. Lets say:

R1 has a "platform OPR of 40, and a co-op OPR of 30
R2 has a "platform OPR of 40 and a co-op OPR of 28
R3 has OPRs of 0 for the sake of argument

By the above method, the red alliance would be predicted to score just 110 points, since we would use R1’s co-op average, but not R2’s. But if R2 can usually score 40 platform points and do co-op most of the time, surely they wouldn’t put up their 40 and then spend the rest of the match twiddling their thumbs. They would use the time they normally use on co-op to score more platform points!

(There’s also the issue of whether the opposing alliance has a high enough Co-op OPR to ensure co-op will be successful, but now we’re getting really complicated.)

[Louisiana_Jones]

There’s an issue with this method that may skew predictions for alliances with more than one team that does a lot of co-op. Lets say:

R1 has a "platform OPR of 40, and a co-op OPR of 30
R2 has a "platform OPR of 40 and a co-op OPR of 28
R3 has OPRs of 0 for the sake of argument

By the above method, the red alliance would be predicted to score just 110 points, since we would use R1’s co-op average, but not R2’s. But if R2 can usually score 40 platform points and do co-op most of the time, surely they wouldn’t put up their 40 and then spend the rest of the match twiddling their thumbs. They would use the time they normally use on co-op to score more platform points!

(There’s also the issue of whether the opposing alliance has a high enough Co-op OPR to ensure co-op will be successful, but now we’re getting really complicated.)

Is the max coopOPR used for the same event as the Max OPR? If not I see some problems where it may appear that teams score more than they really do, if they coop more at some events, and then stacked more at other events. They would be getting the max points from both.

[Richard_McCann]

On the coop OPR I think you need to take a sum of the maxBlue(coopOPR,20)+maxRed(coopOPR,20) as the closest approximation and apply the total to both alliances.

And watching the regionals, the coop points increased with overall scores so that the max coop OPRs should be in an event close to the max OPR.

[Woolly]

On the coop OPR I think you need to take a sum of the maxBlue(coopOPR,20)+maxRed(coopOPR,20) as the closest approximation and apply the total to both alliances.

I think you would want to take a sum of the min(max(BluecoopOPRs),20)+min(max(RedcoopOPRs),20) , as what you wrote would just give every alliance 40 points.

[ceb515]

I would like to get a copy of this source code before CMP if possible, so I can play with it while on the plane.