Your scouts hate scouting and your data is bad: Here's why

Agreed!
We make suggestions through our drive mentor. We will ask the drive why they did something unexpected or unusual, but direction for that team must come from one source.

Hey Katie, first off, awesome post. Matches many of my observations but also had a few insights and ideas I’d never considered before.

I wanted to ask about this. I’ve been a part of a lot of good picklist meetings, but not big ones. How do you manage a picklist meeting with a lot of people?

1 Like

We try to pre-prepare as much as possible. Have all your data entered into whatever database or spreadsheet system you use, whether collected by an app or transcribed from paper. Run whatever formulas so you can quickly see the data you care about for any given team.

What you want to avoid is this scenario:

“Ok what about team 9997?”
“Um which one are they again?”
“The green and black one with the swerve drive”
“Wait but aren’t they a cargo only robot?”
“That’s what my sheet says…”
“No I’m certain I saw them place a hatch in one of the matches I scouted.”
“Are you sure? My data says they’re cargo only.”
“Does anyone have a picture handy?”
“Here let’s pull up their reveal video on YouTube…”

Repeat for every single team at the event…

2 Likes

My team also has scouting meetings open to the whole team, and usually almost everyone (~20-25 kids) comes.

I think the biggest key is cracking down on side conversations. Before we start I emphasize that anything on-topic needs to be shared with the whole group, don’t break down into “what about that one team” whispering with the person next to you, and anything off-topic shouldn’t be said. Throughout the meeting if I start hearing whispers I’ll call them out.

Then we mostly turn it over to the scouting captain, who will pitch specific questions to the group, like “who were the top high-rocket robots” or “what do y’all think about team xxxx?”. We use a scouting app, so making our lists of the best hatch bots, best climbers, etc is usually pretty straightforward, and human input in the meeting mostly comes in the form of challenging the list that falls out of the data (such as “yeah, their average is good but they didn’t move in their last two matches”). Often a couple scouters do most of the talking, but I still think it’s good for morale to not make the meeting exclusive.

1 Like

Side conversations cannot be permitted, we had to go to the length of giving one member a time-out when on their 5th time they refused to stop the side discussions. There is a very specific goal in mind during scouting pick meetings and while everyone is welcome to contribute, the discussion must be directed and one at a time. Everybody needs to get back to a bed to sleep confident that the picklist is the best that could be derived given the time and data available.

Before speaking, we ask that contributors have a fully formed idea that they can express in a way that others can understand. Naturally junior team members are to be given some leeway.

Very much yes to this. Our head scouts do the following:

  1. winnow out highly ranked teams that are guaranteed not to be available based on where we finished or are likely to finish as well as ones that simply cannot play proficiently enough to be considered for picking based on our scouting data, so we are not even discussing those as they are pure time wasters and scouting meetings already go late enough
  2. Ensure that you have pictures and a link to the reveal video (if it exists) for each team. These are embedded right into the spreadsheet so there is no hunting around. Spreadsheet layout for us is: Column 1 - rank (internal scouting ranking, not event rankings Column 2 - team number & name Column 3 - robot picture. 4 onward - other data from your scouts

Humans are visual animals, and many people will remember a visual of a robot better than the number or name

  1. Make sure that any special capabilities or failings of the robot, team strategy or driving are noted on the scouting sheet to either bump up or down prioritization of a given team.

Finally, this is the only place where our team is permitted to speak negatively of other teams or robots, but it must be based on facts, even if it was an experience of being blown off during a pit scouting exercise or difficulty working with another strat team during quals.

There are a few tricks I’ve learned - snichols does hit the nail on the head in some spots:

  • Have a plan, even a loose one. When there is no plan there is chaos*.
  • Determine who will have executive power. Typically this is the strategy lead and the scouting lead. I announce at the beginning who will have final say and then enforce it throughout the meeting.
  • Begin the meeting with funny stories from the event - blow off some steam, get everyone in a good mood. Is this 100% required? No. Do I recommend it? Yes.
  • Anyone not contributing/actively detracting from the meeting will be asked to leave.
  • Give everyone a task
    - I parallel process during pick list meeting. Instead of the entire team going over each robot, sub teams go over a subset of robots. This engages more people and is generally less boring.
  • Have a way to get everyone’s attention. I like to use “if you can hear me clap/snap once”

*What I do

quick disclaimer: I’ve never won a regional outside of 2009 as a 3rd bot. I rarely get out of quarters (shout out to 253 at Monterrey Bay). My team(s) have done well (namely 1296 at champs in 2017 & 2018) and picking was a factor, but I don’t really have a lot of hardware as proof of my systems. I have lots of lessons learned; observations & experience; and fantastic mentors, role models, and peers. This is to say that while I am pretty dang confident in my knowledge and systems, there is definitely room for improvement. There is always room for improvement.

6 Likes

We’ve had some success with our process, so figured I’d share. It usually involves 75-80% of the students, especially a drive team member so we understand if there are any personality conflicts. It’s got many of the same steps.

  1. Set a (rough) picking goal so everyone knows what we’re trying to optimize. Everyone understands that everything is always tentative
  2. Go through each team with all present looking at raw data and consolidated metrics that vary based on the game, verbal vote of high / medium / low / no (including ourselves). We don’t spend much time (like 5 seconds) on the obvious high / no classifications.
  3. Provided there are at least 24 / 32 teams not in ‘no’, we split into 3 teams to stack rank each category. This takes the most time. If there aren’t 24/32, we grandfather “no” teams to the low list until we hit 24/32.
  4. Combine the lists. Quick debate on if anyone has any major issues, and if after further reflection the teams at the high / medium and medium / low junctures need to be moved up or down
  5. Scouting lead & scouting mentor make adjustments throughout the next day through the very last qualification match. We usually prioritize teams doing well right before alliance selection.
  6. If we’re in first, find and work with the chosen first pick to get their buy-in. The same goes if we’re another team’s first pick. Let teams know we’re going to decline if they ask and we’re planning on declining.
  7. Whiteboard or text our choice from the sidelines to the student rep depending on what the current situation is. The list fluctuates based on who other teams pick and who we get as our first pick.
2 Likes

I don’t really understand why y’all break up into small groups, if you need to combine the teams into one list. Would it not be better to have everyone’s perspective on where a certain team should be initially? This seems like it would create a lot of re-explaining reasoning for every time a group merged.

Know when to say no

We always discuss this in the scouting meeting; it’s usually a pretty short conversation but if there’s someone we’d plan to turn down that’s something we want everyone to have the opportunity to weigh in on.

Another note I forgot to mention: At the end of the scouting meeting we very sternly emphasize that everything we’ve talked about is confidential within the team. Don’t tell anyone they’re on our pick list, or not on our pick list, or talk about it with each other at the venue even if you think no other teams are listening. My last team learned this the hard way when we first opened up the pick list conversation to everyone; it may have gone without saying when the conversation was limited to drive team and scouting lead, but not everyone on your team will necessarily realize the importance of discretion unless you call it out.

2 Likes

This is part of why we do top-down groups instead of bottom-up. Everyone has a chance to speak up about every team to slot them into at least a wide band. Splitting into groups helps the meeting last 1.5-2 hours vs 4 hours.

We do this too. We have pretty strict rules about who can talk to other teams about our picking decisions to manage the flow of information, but the people who are cleared to release information will be as straightforward as they possibly can be with other teams, because we would want other teams to be that way with us.

1 Like

Good point that I’d forgotten in my post… even though I’m often present at the meetings it’s not my place to announce picks, extend offers or otherwise “negotiate” on behalf of the team so I just pretend like I am purely a spectator and not involved in the process whatsoever.

This is emphasized to all team personnel (everyone from scouts, hangers-in, other mentors, parents and additional teachers etc) who are present at the meeting - the scouting mentor and scouting team lead are the only people allowed to engage in any kind of discussions about who may be picked / declined etc with other teams.

99% of the time there is no contest because everyone can look at the card with the stats. Everyone going through every team is super long and super boring.

Additionally, the team is doing what is called a “merge sort” where by sorting the smaller piles, it makes sorting everything faster. What you’re suggesting would be an insertion sort which is considerably slower:

Additionally by running the smaller groups, we can parallel process the smaller sorts, which also speeds up the process.

4 Likes

Very interesting simulation, thank you for that. So this would assume a random data set where the merge sort is clearly the fastest. For making the pick sheets, wouldn’t an app or excel sheet be able to give closer to a “nearly sorted” data set, where insertion would be faster, because it would be comparing to it’s relative neighbors in the data set. Usually scouting data can ball park the ranking if teams purely based on averages and peaks, then it can be tweaked manually. This is much harder for picking for things like defense, but for cycle comparison would this not be faster?

1 Like

Sure and when its been a small amount of people in the meeting, I have gone that way because it is much faster. Basel asked about a large group, and when you have the man-power to scrutinize every team, then go for it.

2 Likes

Okay, I like most of what you said but <25 entries? For my money, have an input with as many as you need … more or less than 25. Understand, that a number of those will be rarely checked off … like fouls, or fell or died. These are important in determining the reliability of said robot. Leaving them off to keep your form more simple will hurt you. Its like a passport. Rarely used but boy, when you need it, it is vital. I’d never let a limit like this deter one’s design of a scouting entry sheet.

I’ll go ahead and disagree with you here. The biggest struggles I’ve encountered with scouting in FRC is balancing ease of use, clarity, data quality, and data depth/breadth. Separating objective and subjective scouting helps to ensure that your data is of higher quality and you still get notes on in-match performance. Sectioning/paginating your scouting system into categories will help the students focus on what’s important at that point in time. If you teach kids to use a system where 95% of the time they won’t have to hit a button, they’re going to habituate not hitting that button, especially when they should have.

In my experience, the struggle with data quality in FRC isn’t getting from 95 to 100% quality data, it’s getting to well under 95%. Simplifying the scouting system is more valuable in the long run for data quality than having 10 extra options that are going to confuse your scouts, and are seldom used.

4 Likes

I took over as scouting mentor for my team last year. All but one of our scouts had graduated the previous year, so we had to start over from scratch. When I took over, I quickly realized that while the head mentor/coach of the team valued scouting and knew it is an important component of competition, not everyone felt the way he did. After having an in depth discussion with my one veteran, we quickly decided to change the culture of scouting on the team. First we redefined the role of members on the Scouting Sub team. While our Scouts can and do scout, their roles are to assist other team members in scouting, collect and correlate the data, and to meet with the drive team to discuss game play and alliance selections based on the data. Next we implemented the policy that if you are on the team, you automatically were a part of Scouting as you would be expected to scout at some point during competitions. As expected, a few students complained that “the drive team and pit crew did not have to scout…” We then explained that just because the drive team (or the pit crew not on duty) may not be in the stands, they were still scouting and that the Scouting Sub Team had very intense discussions with them and included their insights in our scouting data. After passing this hurdle, we focused on the aspects to change the culture of scouting for the team. We did this by 1) making scouting just as important as all other sub teams/positions and 2) making it fun and rewarding! Anyone scouting was thanked and told how important what they were doing was crucial to the team. We requested their input on what were were doing - what should we scout, how should we scout, what will make it easier? meaningful? rewarding? Scouts were given special buttons to wear, and a special banner was placed on the door of our “Scout Cave” in our shop. We did a few other things to make it fun, but the change we made that had the most impact is that we created an award that was given to the team member (not a member of the Scout Sub Team unless they went above and beyond the call of duty - this did happen at Worlds when one of our students scouted the entire Blue Alliance for 7 straight matches and didn’t miss anything!) who best exemplified scouting in that competition. As far as scouting is concerned by changing the culture, we transformed the team!

5 Likes

Hundreds of matches scouting data won? Seems unlikely.
Scouting can be a force multiplier in some matches, sometimes you just don’t have the scoring and/or defense needed. To get to “hundreds” are you talking since First started?

Sometimes ,scouting can mitigate the lack of raw alliance capability, usually by a discussion involving re-purposing a usually weaker struggling team member off their normal game plan to gain in the game in front of them and pick up some RP’s. This also serves to thwart other scout teams that were basing their gameplan on same same .

I believe scheduling has a much larger effect than scouting and a team literally has no control over that. That’s another thread.

Yes, I agree. Our drive team talks to their partners and lets them embellish, then they point out our scouts report you didn’t do that in a game yet…from there the other teams know the gig is up and then with all the fake out of the way , the alliance cane game plan based on actual metrics.

For scouting to win “hundreds” that means in every one of those theoretical games the alliance was an underdog and pulled out a win

1 Like

But it’s not. Scouting data is how you can form a pre match strategy. Which leads to a higher likelihood of winning.

Absolutely, that does not guarantee a win and sometimes teams that use scouting are already favored in every game they enter so scouting itself had no effect on that win alone