This is correct, and is generally why I remove the foulPoints component when doing any sort of OPR analysis. I would like to take a look at seeing if applying regularization methods (i.e. ridge regression or LASSO) would help to enforce smoothness and limit the impact of components with extreme outliers.
Instead of constantly uploading new spreadsheets, I added a live viewer to our website: http://viperbotsvalor6800.com/scout/
I’ll be making changes throughout the next couple weeks but this is a good start
Looking at the 2019 Google Sheets COPR calculator, I noticed a few things:
Creator chose to add in penalties into the overall OPR. I don’t like doing that since penalties are not consistent
In the google sheet calculator, there are some teams that have climbing averages of over 12. Wasn’t possible during week 1 because no one had a double level 3 yet.
District event I was comparing (miket) had 0s for all cargo scored in rocket level 3. I know for a fact that a team scored a cargo game piece in the top level of the rocket at that event.
Edit: No bugs! Just me not paying close enough attention. See discussion below
So although that google docs generator is an awesome idea, looks like there are still a few bugs in it (not saying mine doesn’t have bugs in it either!)
I will have to look at the event simulator in a bit to do a comparison on that as well.
An OPR calculation is statistical, not direct. It’s the effective contribution, not the actual point totals, that are calculated. That’s how some teams will end up with a negative OPR–they hurt the performance of the other teams on the alliance. The component contribution OPRs in the past have often shown values higher than the possible contribution of a single robot. For example in this case, a team may both contribute 12 points AND increase the likelihood that the other teams reach a L1 climb.
Are you, by any chance, including playoff matches in your OPR calculations? The code I use to calculate OPRs (which exclude playoff matches) produces numbers similar to TBA, and when modified to use all matches produces numbers similar to yours. It’s important to note that OPR is only defined over qualification matches due to the lack of mixing of alliances in playoff rounds.
Analytically speaking, the OPR model assumes that the interaction terms between specific teams are insignificant (i.e. have a mean of zero), which is generally a safe assumption for qualification matches, because teams rarely play with or against each other more than once or twice. However, as teams develop synergy in the playoffs and assume roles on alliances this becomes an increasingly poor model. In addition, since a playoff alliance will have the same vector for each match, you can potentially run into issues with your ß-matrix (from the y=ßx linear model) not being of full rank when it comes time to solve, especially at small events. This is why it’s generally important to remove playoff matches from regression-type CC algorithms.
Ahhh we found the culprit! Thanks for that! I totally meant to remove elimination matches (I have been doing that for years but forgot this year). I will modify that ASAP.
Yup on the same page! This year, the FMS keeps track of individual robots for endgame. That means we get exact averages for climbing this year!
Looks like the google sheets creator is calculating an OPR for climbing rather than using the FMS data for climbing. I opted to avoid calculating climb OPR and instead, use FMS data to calculate the exact average (similar for starting configuration).
I include penalties because I believe OPR should be used mainly as a very rough gauge to compare teams, so I try to adjust the total scores as little as possible. While penalties are certainly inconsistent, so are the alliances teams are facing, and that is not factored into OPR either. This is also to stay consistent with Caleb’s and TBA’s calculations, which I use to check against for bugs. (and last I checked, all 3 match)
If this is team 27’s climbing OPR at miket, I agree with what @Richard_McCann said. Seeing as they had the most consistent L3 climber there, I think it’s reasonable to assume they would try to ensure at least one alliance partner made it to L1. Alliances unable to get the endgame RP may not have cared as much for those 3 points.
I would also suggest using the endgame % values instead of endgame OPR for evaluating climbing performance, since those are tracked per team instead of per alliance and are likely more accurate.
I’m not sure what you’re talking about here. Match 10 and 63 both have cargo listed as scored, which matches TBA. If you’re talking about cargo L3 OPR, the teams in those matches have very small (0.08 to 0.25) but non-zero OPRs, which makes sense.
Please let me know if you find bugs, but I don’t think what you’ve listed are issues and are just side effects of linear regression on small data sets.
That is actually a pretty cool assumption that I did not think of. I can imagine that high-tier level 3 climbers actually get more than 12 points due to what you said cause they are guaranteeing the level 1 climb as well.
Same page. I just multiplied the % by the point value to get their OPR instead of separating the columns out like you did. Smart though - gives you multiple datasets to view