This thread has sparked some interesting discussion on quantitative/qualitative scouting. Since I see this as a spectrum rather than a binary decision, I thought I’d make a poll to guide some discussion. So, where is your team on the quantitative/qualitative scouting spectrum? Specifically in regards to alliance selection. I understand this is a difficult thing to quantify, so I tried to provide some guideposts. These are just to try to help give people a feel for what these categories represent, so feel free to reply with 7 even if your system isn’t exactly like the description.
The line between quantitative and qualitative is fuzzy, for the purposes of this thread, let’s define quantitative to be numerical or enumerable metrics found during match play that can be independently verified. Basically, let’s say objective=quantitative and subjective=qualitative. So for example, “vault cubes scored” would be quantitative since everyone collecting this metric will give the same answer and the “correctness” of the data can (at least theoretically) be objectively verified. However, “driving ability” on a 1-5 scale is qualitative because two scouts can reasonably put different answers, and neither will be objectively correct.
Why are you at the position you are at? Do you want to move more in one direction or the other next year? Does the optimal position on this spectrum vary depending on scouting team size?
I answered 7, which is the basic reason why I answered “yes” in the original thread. Quantitative measures definitely make up the bulk of our ranking, but we also adjust based on teams’ “intangibles”. Some stuff, like prior experience in playoffs, experienced drivers/coaches, and robot build quality just won’t show up in the scouting data, but can make or break an alliance. And now once you’ve started adding opinions to the mix, a team’s branding will undoubtedly influence the decision to some degree, consciously or subconsciously.
Edit: I forgot to mention, we do collect qualitative match scouting data (e.g. driving ability, defensive skill) but usually don’t use it for making pick lists. More often than not the value varies more based on the scout giving the grade than the team’s actual performance. Without a good way of standardizing across all scouts (which we haven’t found yet), the data is way too noisy to be really useful.
You will probably never come up with an effective system that removes qualitative choices. Even professional sports, which are probably the most analyzed activities in the world, rely on some qualitative observation by skilled individuals.
In FIRST robotics, you have a game that changes every year. The scouting systems do not have a chance to mature. The the amount of data on individual robots is not complete enough to make difficult decisions.
While our team looks at a number of quantifiable metrics, there are always the “intangibles”. How well our drive team and a potential allies drive team will work together always comes into play. And then there are those teams who we are ‘taking a break from’ because of comments or behaviors that we find objectionable. You can’t quantify that.
876 has gone to utilizing a qualitative ranking first, done by the lead scouts for 3-4 categories. After this we bring in our quantitative data to adjust/ validate and pull teams from that ranking if they fall on the NPL.
We seem to be in the minority with that one, but here is why: pick lists are an ordinal-level datatype (we only care where teams fall in relation to each other and not by how much). Qualitative methods are great for generating where a team falls on a spectrum, that can then be collapsed (along with the quantitative data) down to the ordinal level. Since you are looking for the best remaining option that fits your strategy when alliance selection is going on you pull the topped ranked team from the ordinal sub-list. (…and if your strategy involves more complex components such as what the alliance picking a team before you did you: then follow your decision trees/flowcharts, but that is a different matter that is functioning on nominal-level data.)
With smaller datasets you simply need to follow a more humanistic approach to team rankings, there simply isn’t enough quantitative data for the decisions to be made on that alone and be confident in the results.