OPR vs Record at championship

[Bongle]

Originally Posted by PayneTrain

I can tell you 422 went the opposite of the OPR projection of 2-6 and posted a 6-2 record and everyone will agree that CMP was by far nowhere near as good on the whole as DCR, and the actual robot scores show that.

OPR is becoming less reliable as teams in CMP divisions are coming in with pre-CMP match differences of up to 50.

Another reason for that is that OPR doesn't compare well across events. Suppose regional A had a ton of defense, and regional B had wide-open scoring. Two robots of equal quality could each go to their regionals and have vastly different OPRs because at regional A, nobody could score well and at regional B, everyone was free to score. Projections based on OPR from each regional might show that the teams that went to regional A were much worse than the teams from regional B, but they're actually pretty strong.

It also ignores robot, team, strategy, and luck upgrades that might occur between regionals and championships.

[dtengineering]

I'm okay with the qualifying system as it exists... I think it matches up operational requirements of the tournament with the goal of correctly sorting the teams. Its not perfect, of course, but it is as good as a round-robin is likely to get.

Round robin tournaments aren't the only way to organize a tournament however. I think a Swiss System type tournament could be organized where all the teams start at the same level, but either advance or drop down based on their record.

This way, while teams play their first match in random alliances, it would quickly work out so that top teams ended up playing against/with other top teams, while weaker teams would play against/with other weak teams.

It would take some planning, and I'm certainly not saying that change is needed... I'm just pointing out that change is possible.

Jason

[Citrus Dad]

Originally Posted by Bongle

The qualification ranking used in FRC is pretty poor. Even if you had some very good mathematical evaluation of robot performance (which in certain years, including this year, OPR is), it wouldn't correlate well because the standings are very often not a good evaluation of robot performance.

So really, there's a corollary to Lil Lavery's post:
-A bad performance indicator (OPR in his opinion) doesn't correlate well with another bad performance indicator (standard qual rankings with 8 matches played)
-Also, a good performance indicator (OPR in other's opinions) doesn't correlate well with a bad performance indicator.

Given the definition of OPR: "the average number of points a robot's presence adds to its Alliance's score", the real question about OPR is not whether it predicts final standings, but rather whether it accurately predicts an upcoming match score given the input of matches played before.

Based on my statistical analysis, the OPR this year was a better predictor than last year's. I was surprised, but teams improved more over last season than they did this season. I haven't yet run a full analysis on the predictive capability this year, but last year using average OPRs, I could predict qualifying match outcomes 75% of the time. I switched to max OPRs this year (and made some adjustments based on our own scouting data) and I think the prediction rate improved.

Standings can only be predicted by totaling the OPRs for all alliances in all matches and computing the win-loss record. Remember this is a team+teams sport, not just an individual or single sport, so an individual OPR won't carry many matches.

[Citrus Dad]

Originally Posted by PayneTrain

I can tell you 422 went the opposite of the OPR projection of 2-6 and posted a 6-2 record and everyone will agree that CMP was by far nowhere near as good on the whole as DCR, and the actual robot scores show that.

Note that defense is much more important at the championships than at any regional. Robots are much more beaten up after 8 matches than after 10-12 at regionals. That means that individual robot scores will drop.

[minhnhatbui]

Originally Posted by DampRobot

Why did champs seeding not correlate well to OPR? Because 8 matches is way too small of a statistical sample.

And also, in most regionals I think, a team is more likely to play with or against every other team at least once - 9 to 12 matches with 35 to 60 teams, depending on the regional - which make the OPR more accurate than in a championship division, where you get to play with 40 to 45 teams - 8 or 9 matches, depending on the year - out of 100 teams.

OPR is meant to be more effective if a team plays against or with every other team at least once.

[PayneTrain]

Originally Posted by Citrus Dad

Note that defense is much more important at the championships than at any regional. Robots are much more beaten up after 8 matches than after 10-12 at regionals. That means that individual robot scores will drop.

On the flip side of the coin, some teams have their robots at peak performance by championships.

[Ether]

Originally Posted by minhnhatbui

And also, in most regionals I think, a team is more likely to play with or against every other team at least once - 9 to 12 matches with 35 to 60 teams

In each match, a team plays with 2 other teams and against 3 other teams. So in 12 matches, any given team could play against at most 36 other teams, and with at most 24 other teams.

But this ideal is rarely realized.

Here's an analysis of the qual match scheduling in the 2013 season:

http://www.chiefdelphi.com/media/papers/2822

[MamaSpoldi]

Originally Posted by Citrus Dad

Note that defense is much more important at the championships than at any regional. Robots are much more beaten up after 8 matches than after 10-12 at regionals. That means that individual robot scores will drop.

REALLY?? We found far, FAR less defense at Championships than at either of the Regionals we attended this year (BAE - week 1, and CT - week 5).

[OrangeCataclysm]

Originally Posted by MamaSpoldi

REALLY?? We found far, FAR less defense at Championships than at either of the Regionals we attended this year (BAE - week 1, and CT - week 5).

More defense I would say was played in elims, especially in Curie. My team can attest to that

[MamaSpoldi]

Originally Posted by OrangeCataclysm

More defense I would say was played in elims, especially in Curie. My team can attest to that

I didn't really get to see division elims as I was in the pit helping pack up, but I was surprised at the general lack of defense on Einstein.

[OrangeCataclysm]

Originally Posted by MamaSpoldi

I didn't really get to see division elims as I was in the pit helping pack up, but I was surprised at the general lack of defense on Einstein.

I saw more defense in Curie v. Archimedes than any other matchup. To be honest, I wasn't expecting a whole lot of defense. Once the full court shooters stopped full courting or were eliminated, much of the defense ceased.

[Citrus Dad]

Originally Posted by OrangeCataclysm

More defense I would say was played in elims, especially in Curie. My team can attest to that

We selected 862 in large part for its defensive capabilities and they held off 67. 4814 demonstrated in Curie how important defense can be.

[Citrus Dad]

Originally Posted by MamaSpoldi

REALLY?? We found far, FAR less defense at Championships than at either of the Regionals we attended this year (BAE - week 1, and CT - week 5).

Maybe you needed to play in Curie. 4814 made it to the division finals through defense and counter defense. We beat 2056 through defense.

[minhnhatbui]

Originally Posted by MamaSpoldi

I didn't really get to see division elims as I was in the pit helping pack up, but I was surprised at the general lack of defense on Einstein.

As my students pointed out: defense was well planned on Curie, but went full speed to the landfill on Einstein field because, well, playing in front of 20,000 spectators is somewhat very stressful. I tend to agree with them on that one.

[Siri]

Originally Posted by Citrus Dad

Maybe you needed to play in Curie. 4814 made it to the division finals through defense and counter defense. We beat 2056 through defense.

Curie was great, but I didn't see anything in terms of defense there that hadn't happened at other events. Similar D and counter-D are exceedingly common this year (for good reason).