Quote:
Originally Posted by Chris is me
This is extremely difficult to do, even with full scouting data - no one / two / three stats correlates perfectly with overall robot quality particularly with a multiple role game. You can't even accurately say how many points a robot scored in a match - who gets the assist points? I have data, if you have an idea what good evidence would be for this claim, let me know and I'll work on it.
|
Quote:
Originally Posted by compwiztobe
This data may be difficult to come by, since the results have been inconsistent across events. In my experience, where the foul-heavy, inconsistent, or even incompetent refing is taking place, there are upsets and anomalous rankings. Where these are not a problem, rankings are accurate and high seeds go far in elims. But as you said this is anecdotal, and even the preliminary judgments of foul prevalence, inconsistency, inappropriate rankings, and upsets will be somewhat subjective unless backed up by data, before any such conclusion can be drawn with confidence. Some suggestions would be to look at, for each event, Winning Margin without foul points stats, stats on whether red or blue won elims matches, stats on declines in selections, what sort of actions determined upset results (clean match, fouls, etc.), and things of that sort. Perhaps a correlation can be drawn, perhaps not.
|
I agree that's it's difficult to do, and harder still to separate whatever stats we can create from noise (given that all the historical context is flawed at best because of the constantly changing games). But I don't think decisions should be made on anecdotes or small sample sizes. We have these types of gripes about many games. There was a point last year where people felt that Ultimate Ascent was an
upset friendly game as well. Commentary on the ranking system and serpentine draft happen for many FRC games, not just this one.
As for suggestions of how to measure the impact, there are a few ways you could go about it. None are foolproof, but they may at least begin to paint a picture of what's happening, and hopefully would be useful for the community as a whole in the future. First and foremost, tracking what alliance's win in each round, and which alliances win tournaments as a whole. There are some already compiled in
this thread and
this thread pertaining to partial data sets from 2012-2014. While it certainly measures the combined results of several different variables, I'm uncertain how much delta we'd see in the final evaluations.
As much as I loathe OPR for comparing teams, over a significantly large sample size it could provide some high level information as well. Tracking how well OPR correlates to rank for 2014 compared to other games would be interesting. If you have significant scouting data, picking any one or two meaningful stats to see if any correlation exists may also be interesting in terms of helping to establish a baseline, even if it can't be directly applied to the 2014 game. I'm not sure if the sample size would be large enough for tracking declines to be worth much, but it would be interesting.
Any single statistical analysis would probably be flawed. But if we saw that multiple methods suggested similar conclusions, there may be validity to it.