OPR vs Record at championship

[Ivan Malik]

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pool elims play each other to determine the field champion. The field champion would then represent the field on Einstein.

[Bongle]

The qualification ranking used in FRC is pretty poor. Even if you had some very good mathematical evaluation of robot performance (which in certain years, including this year, OPR is), it wouldn't correlate well because the standings are very often not a good evaluation of robot performance.

So really, there's a corollary to Lil Lavery's post:
-A bad performance indicator (OPR in his opinion) doesn't correlate well with another bad performance indicator (standard qual rankings with 8 matches played)
-Also, a good performance indicator (OPR in other's opinions) doesn't correlate well with a bad performance indicator.

Given the definition of OPR: "the average number of points a robot's presence adds to its Alliance's score", the real question about OPR is not whether it predicts final standings, but rather whether it accurately predicts an upcoming match score given the input of matches played before.

[Citrus Dad]

Originally Posted by Bongle

The qualification ranking used in FRC is pretty poor. Even if you had some very good mathematical evaluation of robot performance (which in certain years, including this year, OPR is), it wouldn't correlate well because the standings are very often not a good evaluation of robot performance.

So really, there's a corollary to Lil Lavery's post:
-A bad performance indicator (OPR in his opinion) doesn't correlate well with another bad performance indicator (standard qual rankings with 8 matches played)
-Also, a good performance indicator (OPR in other's opinions) doesn't correlate well with a bad performance indicator.

Given the definition of OPR: "the average number of points a robot's presence adds to its Alliance's score", the real question about OPR is not whether it predicts final standings, but rather whether it accurately predicts an upcoming match score given the input of matches played before.

Based on my statistical analysis, the OPR this year was a better predictor than last year's. I was surprised, but teams improved more over last season than they did this season. I haven't yet run a full analysis on the predictive capability this year, but last year using average OPRs, I could predict qualifying match outcomes 75% of the time. I switched to max OPRs this year (and made some adjustments based on our own scouting data) and I think the prediction rate improved.

Standings can only be predicted by totaling the OPRs for all alliances in all matches and computing the win-loss record. Remember this is a team+teams sport, not just an individual or single sport, so an individual OPR won't carry many matches.

[Bongle]

Originally Posted by Ivan Malik

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pools play each other to determine the field champion. The field champion would then represent the field on Einstein.

The answer to this is to consider what would happen if you took it to the extreme: what if you had pools of 6 robots that played 8 matches against each other, and they had a single round playoff of one alliance against another to determine the pool winner, then gradually played against the other pool winners, tournament-style?

The answer: as you reduce the pool size, you reduce the chance of the objectively "best" robot actually winning the entire competition and increase the effect of luck - the best robot might have 4-5 mediocre robots in their pool to pick from, and so can't win the whole thing against an alliance of 3 medium robots from another pool. Put another way: the pool results would be very accurate (the best robot would tend to win all the time), but the overall results would be less accurate since the best robot might not have good robots to take to inter-pool play with it.

The main way to make championships more "accurate" is to play more matches, which really comes down to field-hours: you either need more fields for the same hours, or to simply use the 4 fields you've got for longer hours in the days you've got. Or you need to use Einstein to play more matches.

[artdutra04]

IMHO, the 2013 ranking system is the least problematic ranking system ever devised by FIRST. Very straight forward, entirely based upon robot performance, no weirdness from coopertition or losing score.

The only way this system can be improved is with more matches. The more matches a team has, the better the odds are that luck will filter out and skill will filter in.

[bduddy]

Originally Posted by Ivan Malik

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pool elims play each other to determine the field champion. The field champion would then represent the field on Einstein.

Hmmm... this sounds like it could be one of those statistical traps, but I've thought it over a couple of different ways and I can't think of any way that would improve the variability. Not to mention that, even if it did, it would still weaken the eliminations alliances...

When I was in the TBA chat, we talked about what I thought was a really cool idea: Have 6 divisions, with the same number of teams (I think there's room in the dome for 2 more fields, don't you?) That would dramatically increase the number of matches, and make for a more regional-like environment. But what do you do with the 6 division winners at Einstein, you ask? This was the really good idea someone suggested: have a round-robin to decide the 2 alliances for the finals. Yes, it would take more time - 13-15 matches before the Einstein finals, instead of 4-6. But if you cut down on the hour before Einstein and all the speeches, you could almost fit them without expanding the schedule at all... not to mention that, with the smaller divisions, you could start eliminations earlier. Any feedback? I think a round-robin would be really fun...