Quote:
Originally Posted by Kevin Leonard
Oh god subjective rating systems make me cringe.
For some reason, in 2013, we had a metric on some of our match scouting sheets called "Speed". It was a rating of 1-5 based on what they saw during the match. We never used it for strategy (so I don't know why we had it), but it was funny to see what different students rated different robots. Sometimes teams with 2-speed, aggressively geared drivetrains were given 1's and 2's, while some robots with single-speed, relatively slow drivetrains were given 4's and 5's. Most notably was the fact that somehow our single speed 12 fps tank drive from that year had the highest "average speed rating" at the event, due to obvious bias in the scouts.
We've tried since then to weed out poor, subjective rating systems like that.
|
Our solution is to rank robots within their alliance in an ordinal fashion (although there can be some cardinal ranking, e.g., no one is worth a 3 as the best), and then to pick the best of the match in each dimension, giving them a 4. We worked through the math and assuming the transitive property, the distribution of rankings fall quite close to a cardinal ranking system that relies on our "superscouts" keeping a constant metric across the entire tournament. The system worked extremely well in 2014. (As Mike has said, this system wasn't very important in this year's game.) We're looking at adjusting that ranking system to use the variance of the scoring distribution to standardize the metric.
I think this is where I take over from Mike. The next twist is that we use our quantitative scouting system results and the match scores from the previous competitoin to run predicted scores. We then add in our qualitative scores as defensive effects and minimize the squared error using Solver by varying the weights of those qualitative scores. We're then able estimate the defensive contribution expected for a given qualitative score and the relative weights for each dimension. For example, I think we found the 4814 contributed about 20 points a match (maybe higher?) in defense in the 2013 Curie Division which was multiples of the next robot.
Quote:
Originally Posted by gblake
However, it sounds like you are trying to tell me that in your method, at the end of quals, if you ranked the teams according to the on-field performance your scouts see, you would then also adjust those ranks non-trivially based on pit-scouting data. That sounds a bit odd. I can certainly see making a case for it because the number of qual matches played usually isn't enough to supply an excellent assessment of each teams abilities. But ... with that in mind, I think we might at least agree that as the number of qual matches increases (and for the sake of argument, lets assume everything else is constant), the value of pit-scouting data steadily declines.
What I was saying in support of what I think EricH was saying, is that by the end of a typical tournament, I would side with him and be unlikely to let early pit-scouting data significantly alter any ranking I had created using on-the-field scouting.
If you guys do let pit-scouting data significantly affect your end-of-quals rankings I'm surprised. And, if you do, maybe that has helped you win, or maybe you have won regardless of any possible harm done by those changes. Get out a ouija to answer that one.
Regardless, congrats on the wins.
Blake
PS: In all of this I am setting aside aspects of team performance that depend on how well any two teams get along when they need to communicate/cooperate. For the sake this discussion, let's assume everyone is equal in that regard, and in other similar characteristics.
|
We don't use early pit scouting data other than to get pictures, and probably the drive train configuration. In fact as the season goes on we use previous competition results to pre-seed our scouting data, and we progressively replace that pre-competition data with actual matches. We run a regression of our scouting data on the OPR metrics to estimate the relationship to our quantitative scouting parameters.
We get pit scouting and drive team information as the competition goes on. We've had specific task questions the last two years about robot configuration that we can't really see from the stands, and that our scouts probably can't discern. Our drive team and match tactician gives input about working with particular teams.
We do the quantitative ranking and then we use the pit scout and drive team info to move teams up and down. The fact is that 10-12 matches is not enough observations, and those observations are not independent of each other. Teams change performance over the tournament. The initial ranking is a starting point. Then we introduce the non quantifiable factors such as drive train configuration (no mecanum until this year), robot configuration and team cooperation. And we include our past experiences. We moved both 1671 and 5012 up our list because of positive experiences with their organizations.
So in the end, it may not be pit scouting that trumps our initial rankings, but it is qualitative assessments that are not feasible by our field scouts.