[Rick TYler]

Impressive scouting method. Nice work, and you certainly picked good partners for the finals.

I wanted to take this opportunity to point out to the utes here that purely quantitative analyses have to be subjected to some critical thinking before they are accepted. Since we were at the PNW regional, I'm going to go ahead and use that as an example.

1. Your quant analysis draws some misleading conclusions. Using my own team as an example, your quant analysis shows that we scored between 0 and 4 times per round, and you averaged that out to 1.6 per round. This analysis using the mean (average) number neglects that fact that our scoring did not have a normal distribution. We had an arm fail in three rounds (twice due to a PWM cable being knocked loose before the match, and once when the arm operator jammed the arm by moving it into the stop when our software limit detector had been accidentally overridden by some code changes). This means that we scored 0 tetras in three early rounds, but scored between 2 and 4 times in the other rounds. A more useful measure might have been the median (center value when ranked) or even the mode (the most common result) rather than the mean for this particular measure. Using 1294 as an example, our average score was 1.6 capped tetras per round. Our median was probably 3, and our mode was probably 3 or 4. My records aren't complete, but this is pretty close. This means that when our robot hadn't been sabotaged by our own team, we reliably capped 3-4 tetras per round. Your quantitative analysis missed this. (Now you could make the argument that being unreliable should count against a team, and I would agree with you. That's not my point. My point is that, by its nature, a simple formula cannot take everything into account.)

Another way your Scout might have looked at an event like this would have been to qualitatively note that a robot which performed well at the Bellevue trials and on practice day failed to perform early Friday, and then worked fine after that. This is not an argument that we should have been chosen for the finals, but rather that I wanted to show you that a simple quantitative analyses won't always tell you what you think they will tell you.

2. You have no sense of time series in your data. Bots (like 997) that started off really strong ended up fairing comparatively worse as others learned to defend against them. Likewise, other teams became stronger as their driving teams got better. By treating all data the same, you probably over-emphasize early results. Try a weighting factor over time next year and see if it changes your analysis.

3. Have you done sensitivity analysis on your metrics? This means that you should play with your data to see if fairly minor input changes result in large outcome changes. I'm not asking you to tackle serious multiple-goal multi variable equations here, but you should know in advance if your model is robust enough to only reflect small ranking changes with small changes in the input data. Some models in the world fluctuate wildly with small input changes.

4. Some of your data are wrong, probably because of under-sampling. What process do you use to collect and quality-control it? You probably want to make sure to have different scouts evaluate each robot. As an example, in most of our matches we started off with a held tetra, yet you say "no" to this in your spreadsheet.

5. Unlike a baseball statistical analysis, your universe of measurements is too small to be statistically significant. This means that you should always apply human analysis before accepting the results. (You may, in fact, do this. I just wanted to encourage all teams not to blindly accept nice-looking quantitative results that may actually mean nothing.)

Your methods are proven by your results. You went through the finals like Patton through France. As I said up there, I am just encouraging all scouting teams to not just trust their numbers without fully understanding what those numbers mean and where they came from.

I think you did a nice job.