One item that is amazing to me thus far is the discrepancy of OPR vs the record of the teams. At Michigan State Championship, it was a shoot out for most matches and OPR matched your ranking fairly well. So far from what I’ve seen, not so much so at St. Louis. But then again I am not there.
I can think of two reasons:
With 100 teams per division the field is not as deep as Michigan.
There are only 8 matches per team so the match difficulty is not equal. Or there is a lot left to the “luck of the draw”
So my questions are:
do you agree with these observations?
is it that way every year because the issues are inherent to the way the system is set up?
I forgot just how bad 8 matches is at sorting… add in the 40ish extra teams and its even worse… Smaller divisions are a must in the future.
For those who are way better at statistics than I am, would doubling the number of divisions, but halving the teams in each division result in better sorting? Same number of matches (more would be helpful) but a smaller group to sort…
You could keep the same number of fields; heck there could be two “pools” per field and have the winners of the pool elims play each other to determine the field champion. The field champion would then represent the field on Einstein.
The qualification ranking used in FRC is pretty poor. Even if you had some very good mathematical evaluation of robot performance (which in certain years, including this year, OPR is), it wouldn’t correlate well because the standings are very often not a good evaluation of robot performance.
So really, there’s a corollary to Lil Lavery’s post:
-A bad performance indicator (OPR in his opinion) doesn’t correlate well with another bad performance indicator (standard qual rankings with 8 matches played)
-Also, a good performance indicator (OPR in other’s opinions) doesn’t correlate well with a bad performance indicator.
Given the definition of OPR: “the average number of points a robot’s presence adds to its Alliance’s score”, the real question about OPR is not whether it predicts final standings, but rather whether it accurately predicts an upcoming match score given the input of matches played before.
The answer to this is to consider what would happen if you took it to the extreme: what if you had pools of 6 robots that played 8 matches against each other, and they had a single round playoff of one alliance against another to determine the pool winner, then gradually played against the other pool winners, tournament-style?
The answer: as you reduce the pool size, you reduce the chance of the objectively “best” robot actually winning the entire competition and increase the effect of luck - the best robot might have 4-5 mediocre robots in their pool to pick from, and so can’t win the whole thing against an alliance of 3 medium robots from another pool. Put another way: the pool results would be very accurate (the best robot would tend to win all the time), but the overall results would be less accurate since the best robot might not have good robots to take to inter-pool play with it.
The main way to make championships more “accurate” is to play more matches, which really comes down to field-hours: you either need more fields for the same hours, or to simply use the 4 fields you’ve got for longer hours in the days you’ve got. Or you need to use Einstein to play more matches.
IMHO, the 2013 ranking system is the least problematic ranking system ever devised by FIRST. Very straight forward, entirely based upon robot performance, no weirdness from coopertition or losing score.
The only way this system can be improved is with more matches. The more matches a team has, the better the odds are that luck will filter out and skill will filter in.
Hmmm… this sounds like it could be one of those statistical traps, but I’ve thought it over a couple of different ways and I can’t think of any way that would improve the variability. Not to mention that, even if it did, it would still weaken the eliminations alliances…
When I was in the TBA chat, we talked about what I thought was a really cool idea: Have 6 divisions, with the same number of teams (I think there’s room in the dome for 2 more fields, don’t you?) That would dramatically increase the number of matches, and make for a more regional-like environment. But what do you do with the 6 division winners at Einstein, you ask? This was the really good idea someone suggested: have a round-robin to decide the 2 alliances for the finals. Yes, it would take more time - 13-15 matches before the Einstein finals, instead of 4-6. But if you cut down on the hour before Einstein and all the speeches, you could almost fit them without expanding the schedule at all… not to mention that, with the smaller divisions, you could start eliminations earlier. Any feedback? I think a round-robin would be really fun…
i think that if there is an OPR there should also be a DPR defensive power rating,
because like my team we were all most always playing defense and were put up against teams that we had a hard time playing defense against. So our ranking defiantly didn’t show our potential.
I know your user title has said something to this effect for a while, but I really have to disagree. OPR is never perfect, but I found that it worked pretty well this year. I used it to predict the Saturday seedlings of SVR Friday night with a fair degree of accuracy. It was a pretty good measurement of how much we contributed to our alliances, and we found that it represented other teams’ performance fairly well too.
Of course, OPR can’t predict exactly how many points you (or anyone else) will score in a given match. But in my experience, it does provide a fairly accurate way of predicting which alliance will win a given match.
Why did champs seeding not correlate well to OPR? Because 8 matches is way too small of a statistical sample.
I can tell you 422 went the opposite of the OPR projection of 2-6 and posted a 6-2 record and everyone will agree that CMP was by far nowhere near as good on the whole as DCR, and the actual robot scores show that.
OPR is becoming less reliable as teams in CMP divisions are coming in with pre-CMP match differences of up to 50.
Another reason for that is that OPR doesn’t compare well across events. Suppose regional A had a ton of defense, and regional B had wide-open scoring. Two robots of equal quality could each go to their regionals and have vastly different OPRs because at regional A, nobody could score well and at regional B, everyone was free to score. Projections based on OPR from each regional might show that the teams that went to regional A were much worse than the teams from regional B, but they’re actually pretty strong.
It also ignores robot, team, strategy, and luck upgrades that might occur between regionals and championships.
I’m okay with the qualifying system as it exists… I think it matches up operational requirements of the tournament with the goal of correctly sorting the teams. Its not perfect, of course, but it is as good as a round-robin is likely to get.
Round robin tournaments aren’t the only way to organize a tournament however. I think a Swiss System type tournament could be organized where all the teams start at the same level, but either advance or drop down based on their record.
This way, while teams play their first match in random alliances, it would quickly work out so that top teams ended up playing against/with other top teams, while weaker teams would play against/with other weak teams.
It would take some planning, and I’m certainly not saying that change is needed… I’m just pointing out that change is possible.
Based on my statistical analysis, the OPR this year was a better predictor than last year’s. I was surprised, but teams improved more over last season than they did this season. I haven’t yet run a full analysis on the predictive capability this year, but last year using average OPRs, I could predict qualifying match outcomes 75% of the time. I switched to max OPRs this year (and made some adjustments based on our own scouting data) and I think the prediction rate improved.
Standings can only be predicted by totaling the OPRs for all alliances in all matches and computing the win-loss record. Remember this is a team+teams sport, not just an individual or single sport, so an individual OPR won’t carry many matches.
Note that defense is much more important at the championships than at any regional. Robots are much more beaten up after 8 matches than after 10-12 at regionals. That means that individual robot scores will drop.
And also, in most regionals I think, a team is more likely to play with or against every other team at least once - 9 to 12 matches with 35 to 60 teams, depending on the regional - which make the OPR more accurate than in a championship division, where you get to play with 40 to 45 teams - 8 or 9 matches, depending on the year - out of 100 teams.
OPR is meant to be more effective if a team plays against or with every other team at least once.