Qualitative Scouting?

[evanperryg]

Originally Posted by Green Potato

Generally speaking, the problem with qualitative information is that different scouts have different reactions on the same event, and some form of training/ rubric is extremely helpful, even 100% necessary if your scouts are, like ours, younger team members.

If you go quantitative, you need to make sure that you can accurately interpret the data you get. If you don't attach any external meaning to a statistic you collect (shots, cycles, etc) it literally isn't a statistic any more, and looses its value as a result. Also, It's a good idea to add a "comments" area: some times, it's not obvious if action X should be counted in statistic Y, so allowing the individual scout to write it down is quite helpful.

You've brought up a couple really good points. Any team looking to do good qualitative scouting should record who is taking what notes. We can go back and see, "okay, this person said this about this team, and this person said this about the same team." This allows us to negate the effects of scouter bias, and discussion between disagreeing qualitative scouters can bring out amazing details on a team's strengths and weaknesses. Also, track the match numbers corresponding to specific notes, this allows you to easily explain anomalies in quantitative performance.

Remember, qualitative data is helpful alone, and so is quantitative data. However, when the two are combined efficiently and effectively, you can derive much more from the data than you could with either one alone.

[P.J.]

As many people have said, both quantitative and qualitative (or as we call it on 910, objective and subjective) scouting are incredibly useful in different ways. Both have their strengths and their weaknesses. On 910, we do both but the jobs are handled by different people.

The objective/quantitative data is gathered via paper and pencil scouting on sheets mainly involving tally marks in boxes. These sheets contain a box for notes so that these scouts can record things they notice that are not accounted for on the sheet, but this is not their primary task. This is what the majority of scouts are doing at a competition. (Anywhere from 6 to 24 people depending on how many of our team members are able to attend competitions)

The subjective/qualitative data is the responsibility of a much smaller group. These scouts use a legal pad to simply take notes on robot features, driving ability, general strategies, strengths, weaknesses, etc. These scouts earn their position, they take a test before our first competition and the head scouts and myself look over their answers and determine who will be given this position. We also factor in our observations regarding their abilities as an objective scout. In the past this group has been as small as two people but as our scouting system has become more refined and our team has grown we are lucky enough to have 6 subjective scouts this year, allowing them to each take a position (R1, R2, etc.) and focus on only one robot per match.

While I am lucky enough to be working on a fairly large team, our overall system is able to be scaled to as few as 4 people if absolutely necessary.

tl;dr In my experience both Qualitative and Quantitative scouting are important components of a successful scouting system.

[Citrus Dad]

How many scouts do you have available?

Our system uses 8 scouts. We started this in 2013. We believe that our standard scouts are too inexperienced to give good relative qualitative information. Also we've found asking them to keep track of that as well distracts them too much from inputting the key quantitative data that we use. So we have 6 scouts, one per robot, tracking scoring and other measurable metrics. This is the basis for our 1st pick list.

We then have 2 other "superscouts" who watch each alliance. They rank the teams within each alliance 1 to 3 across 4 parameters for qualitative measures such as evasion, blocking, speed and pushing ability. Across many matches, these rankings provide fairly good measures of relative abilities. We use this data primarily for our 2nd pick list.

We have mentors in the stands then taking some added notes on peculiar or particular aspects that we add in, but those are not extensive.

[CTbiker105]

Originally Posted by Citrus Dad

...we have 6 scouts, one per robot, tracking scoring and other measurable metrics. This is the basis for our 1st pick list. We then have 2 other "superscouts" who watch each alliance. They rank the teams within each alliance 1 to 3 across 4 parameters for qualitative measures such as evasion, blocking, speed and pushing ability. Across many matches, these rankings provide fairly good measures of relative abilities. We use this data primarily for our 2nd pick list.

This is the system we've found to work best. We have level I scouts, level II scouts, and defense coordinators.

Level I scouts track solely quantitative data on Windows tablets, 1 scout per tablet, 6 scouts per match. Level II scouts track qualitative data on laptops, usually set up behind the level I scouts in the stands, 1 scout per laptop, 3 scouts per match, each watching 2 robots. Level II scouts are normally pre-selected, as they must demonstrate the ability to watch 2 robots at a time and still record accurate qualitative data. Level I scouts can be any team member not on the drive team/pit crew.

We've worked out a system this season to be hardwired into the stands amongst each other, so all the data collected from each tablet/laptop will update in real-time across the system. Then, 2 or 3 matches prior to the one we're competing in, we print out strategy sheets, which are created to compile all the data collected throughout the day about the teams we're with and against in our match. This allows our drive team to get an overview of what they're in for, and lets us help them decide on a strategy.

Both our qualitative and quantitative data is pieced together to make up the strategy sheets, so our strategists get a complete picture of every robot in the match when they see them.

[Amy33Amy33]

I think I had a post about this last year, qualitative scouting can be good if it's detailed, and if scouters can properly recognize their own biases. If it's not detailed, like our team's qualitative notes last year... well... *war flashbacks of team captain standing at alliance selections for 10 minutes silent*

[wjordan]

2363 have done qualitative scouting since 2014, and it has resulted in a massive improvement in our ability to judge the quality of the rest of the teams at the competition. We had done the standard quantitative stuff for awhile, but the numbers gathered by quantitative methods are pretty much useless if presented without context, and qualitative scouting helps to confirm trends that are presented by the statistics.

Our system in 2014-15 involved 6 quantitative scouts on Kindle Fires, a coordinator who would manage the scouts and occasionally gather the tablets up to load them into a database, and 1-2 qualitative scouts (the strategy coordinator and someone else, usually) who would take notes on the teams currently playing in the match on index cards.

[Boltman]

With a smaller field in CVR coming off a successful SD scouting endeavor we plan to fine tune our qualitative techniques and get it down to a science...love qualitative scouting. It works.. I was watching the eliminations going "yup there it is". Almost like you can predict how these teams play.