DISCLAIMER: The following analysis and data are based on a sample of one event, with incomplete scouting data (stats reported for a little over 5/9 of a team’s matches on average). Take it with a grain of salt, my true evil motive is to solicit more data to see if it checks out.
DISCLAIMER II: This post also got a little long :/, whoops
Recently (desperately avoiding college apps), I was looking back through our team’s scouting data from 2019mnmi and found something interesting. For that regional, I had decided to try something new and have scouts include on their report a qualitative rating from 1 to 4 of each robot.* We didn’t use the data for much at the time, but when I went back 8 months later and used conditional formatting to look at the average ratings, I noticed it mapped pretty well to the points scored we scouted for each team. Being a stats nerd (with no stats education), I thought “hmm, that’s pretty funky, I wonder how strong the correlation is?” So I went ahead and graphed the average rating and found the correlation coefficient:
The correlation coefficient was 0.8457. As mentioned above, I have no idea what I’m doing with statistics, and couldn’t judge how good that was, so I decided to graph OPR compared to our scouted scoring numbers in comparison:
OPR only correlated with real scoring with an R-value of 0.8117, worse than our qualitative ratings! (though not significantly) That blew my mind! I will note however, that since our scoring data was incomplete, OPR probably had a higher true correlation, and I don’t know if the same would be true of more qualitative data.
I think the application of this finding is limited in games like 2019 and 2020 where one can deterministically tell how many points a team scores, but could be very useful in other games (ie 2017, 2018) where contributions are more convoluted.
So… those were my observations, what are yours? I am 100% sure I’m not the first person to come up with the novel idea to have people rate robots, and I beg anyone else who’s collected data to share it. I have no idea if this is a legit thing or not, but I’m excited to find out.
*an idea primarily stolen from the Citrus Circuits, especially the insight to use an even number to force scouts to sort the robot as either above or below average.
Here are my data:
Qualitative Scouting Predictive Power.xlsx (14.8 KB)