OPR vs Record at championship

[MrRiedemanJACC]

One item that is amazing to me thus far is the discrepancy of OPR vs the record of the teams. At Michigan State Championship, it was a shoot out for most matches and OPR matched your ranking fairly well. So far from what I've seen, not so much so at St. Louis. But then again I am not there.

I can think of two reasons:
1) With 100 teams per division the field is not as deep as Michigan.
2) There are only 8 matches per team so the match difficulty is not equal. Or there is a lot left to the "luck of the draw"

So my questions are:
1) do you agree with these observations?
2) is it that way every year because the issues are inherent to the way the system is set up?
3) Am I biased because I am in Michigan?

[IKE]

8 matches to sort 100 teams is a much coarser function than 12 matches drying 64.

[Abhishek R]

Well, several teams have stepped up their game since their last regional, so sometimes it's not "technically" a discrepancy, just problems getting fixed.

[Lil' Lavery]

How well does one metric poor at evaluating robot performance correlate to another metric poor at evaluating robot performance?

[MikeE]

Originally Posted by Lil' Lavery

How well does one metric poor at evaluating robot performance correlate to another metric poor at evaluating robot performance?

Poorly?

[Ivan Malik]

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pool elims play each other to determine the field champion. The field champion would then represent the field on Einstein.

[Bongle]

The qualification ranking used in FRC is pretty poor. Even if you had some very good mathematical evaluation of robot performance (which in certain years, including this year, OPR is), it wouldn't correlate well because the standings are very often not a good evaluation of robot performance.

So really, there's a corollary to Lil Lavery's post:
-A bad performance indicator (OPR in his opinion) doesn't correlate well with another bad performance indicator (standard qual rankings with 8 matches played)
-Also, a good performance indicator (OPR in other's opinions) doesn't correlate well with a bad performance indicator.

Given the definition of OPR: "the average number of points a robot's presence adds to its Alliance's score", the real question about OPR is not whether it predicts final standings, but rather whether it accurately predicts an upcoming match score given the input of matches played before.

[Bongle]

Originally Posted by Ivan Malik

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pools play each other to determine the field champion. The field champion would then represent the field on Einstein.

The answer to this is to consider what would happen if you took it to the extreme: what if you had pools of 6 robots that played 8 matches against each other, and they had a single round playoff of one alliance against another to determine the pool winner, then gradually played against the other pool winners, tournament-style?

The answer: as you reduce the pool size, you reduce the chance of the objectively "best" robot actually winning the entire competition and increase the effect of luck - the best robot might have 4-5 mediocre robots in their pool to pick from, and so can't win the whole thing against an alliance of 3 medium robots from another pool. Put another way: the pool results would be very accurate (the best robot would tend to win all the time), but the overall results would be less accurate since the best robot might not have good robots to take to inter-pool play with it.

The main way to make championships more "accurate" is to play more matches, which really comes down to field-hours: you either need more fields for the same hours, or to simply use the 4 fields you've got for longer hours in the days you've got. Or you need to use Einstein to play more matches.

[artdutra04]

IMHO, the 2013 ranking system is the least problematic ranking system ever devised by FIRST. Very straight forward, entirely based upon robot performance, no weirdness from coopertition or losing score.

The only way this system can be improved is with more matches. The more matches a team has, the better the odds are that luck will filter out and skill will filter in.

[bduddy]

Originally Posted by Ivan Malik

I forgot just how bad 8 matches is at sorting... add in the 40ish extra teams and its even worse... Smaller divisions are a must in the future.

For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort...

You could keep the same number of fields; heck there could be two "pools" per field and have the winners of the pool elims play each other to determine the field champion. The field champion would then represent the field on Einstein.

Hmmm... this sounds like it could be one of those statistical traps, but I've thought it over a couple of different ways and I can't think of any way that would improve the variability. Not to mention that, even if it did, it would still weaken the eliminations alliances...

When I was in the TBA chat, we talked about what I thought was a really cool idea: Have 6 divisions, with the same number of teams (I think there's room in the dome for 2 more fields, don't you?) That would dramatically increase the number of matches, and make for a more regional-like environment. But what do you do with the 6 division winners at Einstein, you ask? This was the really good idea someone suggested: have a round-robin to decide the 2 alliances for the finals. Yes, it would take more time - 13-15 matches before the Einstein finals, instead of 4-6. But if you cut down on the hour before Einstein and all the speeches, you could almost fit them without expanding the schedule at all... not to mention that, with the smaller divisions, you could start eliminations earlier. Any feedback? I think a round-robin would be really fun...

[shrkeatsman]

i think that if there is an OPR there should also be a DPR defensive power rating,
because like my team we were all most always playing defense and were put up against teams that we had a hard time playing defense against. So our ranking defiantly didn't show our potential.

[Ether]

Originally Posted by shrkeatsman

i think that if there is an OPR there should also be a DPR defensive power rating

If you want a number for DPR, there already is one. The DPR analogy to OPR is given by the following simple relation:

DPR = OPR - CCWM

But it's not very useful, for reasons discussed elsewhere. [edit]fixed broken link[/edit]

[shrkeatsman]

Originally Posted by Ether

If you want a number for DPR, there already is one. The DPR analogy to OPR is given by the following simple relation:

DPR = OPR - CCWM

But it's not very useful, for reasons discussed elsewhere.

thank you but can you translate the ccwm?
thanks

[DampRobot]

Originally Posted by Lil' Lavery

How well does one metric poor at evaluating robot performance correlate to another metric poor at evaluating robot performance?

I know your user title has said something to this effect for a while, but I really have to disagree. OPR is never perfect, but I found that it worked pretty well this year. I used it to predict the Saturday seedlings of SVR Friday night with a fair degree of accuracy. It was a pretty good measurement of how much we contributed to our alliances, and we found that it represented other teams' performance fairly well too.

Of course, OPR can't predict exactly how many points you (or anyone else) will score in a given match. But in my experience, it does provide a fairly accurate way of predicting which alliance will win a given match.

Why did champs seeding not correlate well to OPR? Because 8 matches is way too small of a statistical sample.

[PayneTrain]

Originally Posted by Lil' Lavery

How well does one metric poor at evaluating robot performance correlate to another metric poor at evaluating robot performance?

I can tell you 422 went the opposite of the OPR projection of 2-6 and posted a 6-2 record and everyone will agree that CMP was by far nowhere near as good on the whole as DCR, and the actual robot scores show that.

OPR is becoming less reliable as teams in CMP divisions are coming in with pre-CMP match differences of up to 50.