Is Scouting Beneficial?

We already agreed that yes, it does have a measureable impact. Always picking the lowest ranked team will lead to less wins than always picking the highest ranked team. Since then, we’ve been discussing, as the thread states, “Is Scouting Beneficial?”

1 Like

And I’m totally fine with that. I don’t want anyone to stop doing what they’re doing, especially if they’re confident in it. I’m confident it makes very little difference (although we do it anyway). I’m just hesitant to push either idea as a difference maker to others based on confidence alone.

I don’t think anyone is saying otherwise.

]

We (maybe not we, we) agreed we think there are some bad strategies, not that anything could be measured as is.

Then yes.

I may be way off base here since I don’t know what team you’re from, but there’s situations that data is unlikely to make a difference. If you can’t reliably drive and score scouting probably isn’t going to save you.

What’s the phrase? I think it’s something along the lines of “scouting isn’t going to win the event alone, but it might cost you the banner.”

2 Likes

The metrics I use for this task, like the metrics I use for picking, depend on what the strategy of the alliance is. Some details about how I analyze scouting data are explained in this post that @Katie_UPS and I wrote: Gettin' Picky: A Guide to Alliance Selections (Part 1: The Picklist).

The first step in my process is asking why my alliance lost, which usually boils down to at least one of three reasons:

  1. We got outplayed by a better alliance (usually not the sole problem)
  2. We chose the wrong strategy
  3. We chose the wrong robots to execute our strategy

The first step is to develop a good understanding of how you lost and which of these is the most accurate explanation, and then you need to look at your data and figure out how you could have mitigated that weaknesses.

For example, at the 2018 Central New York Regional, we won the event, but our third robot 5030 was dead on the field for most of semifinals and finals. Upon reviewing the data, we concluded that we had the right idea choosing a third robot that focused on the switch and the vault, but that 5030 was broken in too many matches and was too risky for us as the #1 seed. We concluded that teams 4122 or 6621, who had lower ceilings on switch cubes and vault cubes but were much more reliable, would have been a better pick for our alliance. It didn’t matter because us and 340 were so far ahead of the rest of the field at the event, but if one of us broke, it very easily could have determined if we won or lost. The solution going forward was to better emphasize and highlight reliability-related data points and metrics in our spreadsheet and take a more skeptical approach to evaluating reliability in third robots during our picklisting meetings.

(Absolutely no offense intended to 5030, they’re a great team who I have a ton of respect for, but I think they’d agree with my assessment here.)

1 Like

You yada-yada’d over the best part.

If the other team hadn’t been broken down for the finals and semifinals would you have come to the same conclusion?

We concluded that coordinating three robots scoring on the scale was hard and would require three skilled drivers with excellent scale robots, and the supply of viable scale robots for such a role ran out long before our second pick as the #1 seed. Thus, we wanted the best third robot to maximize our points from scoring in our vault and protecting our switch. Is that helpful?

I forget the exact numbers, but 5030 had somewhere around twice as many issues during qualifications as these other two robots. We discussed this, as the possibility of over-correcting is also something worth considering, but concluded that with the information we had going into alliance selection, 5030 was a significantly riskier pick for a marginal benefit that didn’t matter, and that picking them indicated that we failed to properly analyze the risk involved.

3 Likes

Scouting is more than just numbers.
Scouting is more than just numbers.
Scouting is more than just numbers.
Scouting is more than just numbers.
Scouting is more than just numbers.

8 Likes

Without poisoning the well I think your description gets to the point of issue as well as it’s going to. Thank you for taking the time to respond.

I had to edit my post a few times and I still don’t think I got it right. I meant, if 5030 hadn’t broken down and you guys ended up winning like before, would you have still investigated whether 5030 was the best choice for your alliance, and do you feel confident you would have come to the same conclusion (ie. no there were better pick options based on the risk analysis) even though in this hypothetical they ended up contributing something more in the semi-finals and finals.

Indeed.

This thread feels like a troll. Is it another troll? Are you going to step in and tell us it’s all part of a social research project soon?

Assuming good faith (because good faith is GP), I’m not sure what you’re advocating Smoosh. Banning scouting and auto picking by some metric? If so, I think I can respond in a short sentence: Which metric?

FRC games are not (generally) just 3 robots shooting in parallel. Instead if you watch high level matches closely you’ll discover winning alliances tend to pick strategies that work as an alliance. We did pretty well last year in Detroit, and I can tell you what metrics we looked for:

  • for the first pick, we wanted a team with the highest minimum number of cycles (meaning they still score 10-14 game pieces, even under defense, and rarely broke down). They also needed a level 3 climb in case ours want available.
  • for the second pick we didn’t even look at scoring cycles!. Instead, we picked a team who had shown effective defense in quals, had an excellent track record, didn’t break down, and crucially, climbed in a way that might let us fit two robots on level 3.

If we had picked by scoring OPR or any other single metric, instead of really excellent scouting data, we would never have put together that awesome alliance.

2018 and 2019 were big shocks to a lot of middle tier teams. There really wasn’t much use in the “meta game” for any more than 16 high-goal scorers in playoffs, because there was only enough space on the field for two offensive robots per alliance. The best use of the third robot was defense, auto, and end-game. Which is awesome! It’s way more interesting to have multiple roles on each alliance! But plenty of decent scorers who weren’t among the 16 best, and weren’t paying attention to the meta game, found themselves not picked for playoffs, or asked to play a roll they never did in quals. Scouting and strategy are absolutely linked. That’s one reason FRC is so complex and interesting!

9 Likes

As best I can tell OPs position is the following…

  • Scouting data is no better than using rank or OPR to make decisions (specifically draft picks) or just watching matches and picking whomever stands out at the time
  • Teams can still scout if they feel like it, and OPs team does, even though OP thinks it doesn’t matter, but that it’s irresponsible to tell impressionable youth that scouting matters because nobody has any proof it does

That accurate @Smoosh?

Edit to add: OP doesn’t believe others’ confidence in their postmortem analysis or past success on the field in the form of winning events is more valid than his confidence that it doesn’t matter, and thus rejects anecdotes that suggest as such. Also, OP doesn’t believe there’s enough of a sample size to draw proof and believes it’s impossible to prove that scouting data makes a difference as opposed to other methods of decision making (such as using rank or OPR).

4 Likes

My pleasure. If you can articulate what additional information you’re seeking, I’m happy to do my best to provide it. I wish I could give you a more one-size-fits-all answer here, but every situation is unique and sometimes you just have to look at why you lost and your data and use the critical thinking skills this program is so good at sharpening.

Ah I see. We always go through this reflection process on both picks to identify any possible areas of improvement, even if we think we executed flawlessly at the event. I think we would still observe that 5030 was significantly riskier than what we needed to win and that we didn’t need it to win (we were winning by 100-300 points consistently). I think we still would have concluded that 4122 or 6621 would have been better suited for the alliance but we likely wouldn’t have viewed the 5030 pick as a process failure (we’ve had similar conclusions for other events along the lines of “X was an acceptable pick but Y would have better”).

4 Likes

No more than any questioning of the status quo is a troll.

Heck no! This thread started as a response to a poster insistent on getting RFID tags in everything because they didn’t have enough people to scout, and/or the scouting didn’t accurately reflect the goodness of their team. I was stating that lack of scouts or RFID tags in game pieces isn’t hurting them as much as they think.

And in the same post I mentioned that there doesn’t need to be any metric at all to how a captain picks. And I’m firmly in support of that. Alliance captains earned the right to pick however they want based on whatever metric they choose.

I think the confirmation bias is strong in CD.

I would say that captures it more or less.

Regardless of the hyperbole in some of my posts, I didn’t mean to give the impression that I think it’s irresponsible. I believe I said I was hesitant to advocate for it one way or another just based on some confidence. I don’t believe it’s irresponsible to tell another team that scouting is going to be a difference maker, even if I don’t agree. I don’t think CD should pull down pick strategy posts. I mean I can’t envision a case where a team throws all resources towards scouting at the expense of building a robot for example. It’s just not something I would personally push onto my team, or any other.

That seems like an accurate assessment.

Haha. You certainly know how to keep an argument smoldering. I’ve provided a specific example where we used scouting data to find deep insight into what choices would work with our strategy. It’s pithy to dismiss that as confirmation bias.

4 Likes

I’m sorry but this simply isn’t true.

Not only has scouting data been a huge part of why my team was successful strategically last year, but we ended up picked not because our rank or OPR (we were 35th) but because we had a climb.

At Jackson, 1023 picked the 4th seed, not the 2nd seed because the 4th seed could outscore anyone else on the field except 1023.

I think it’s a safe assumption that that event would have been far far different without scouting.

1 Like

100% this. Not to knock the scouting efforts of other teams, but look at the IN Bloomington District Event this season and explain to me how we were able to pickup 7657 on the number one alliance with the last pick.

They had a 3 ball high goal autonomous, moved off the line in auto, a reliable climb and a solid drive train for defense. For them to get to us, multiple teams had to completely miss this in their draft picks. Once we saw we were going to be able to get them I felt like unless something broke we could basically outplay anyone - and we did. Even in the final matches where we were slightly outscored on balls our climbs more than made up for it. Scouting 100% made that possible.

I’ve been on teams that did zero scouting and teams that did very intense scouting throughout my various FRC travels. I’ll say that not much feels worse than seeding high because your robot actually performed & getting to alliance selection having absolutely no idea who you want to pick. In those situations, it’s usually a quick quarterfinal exit.

I always tell our scouters in the stands doing the data collection work that our victories are as much theirs as they are the drive team’s. We form our strategies based on their data and recommendations every match as we can. Sometimes it’s as simple as knowing who to defend or where to score from based on our opponent’s tendencies. FRC is a game where the teams that consistently win squeeze out every ounce of advantage possible.

Do all teams have the capacity to scout? Absolutely not. This is where you can do some analysis using Blue Alliance data to at least see what teams have done. Having a few people take good notes is also way better than having nothing as well. I guess my point is data is awesome and it absolutely helps. I always want more data rather than no data.

10 Likes

I guess.

Yes, and based on the fact that you had an awesome alliance, your conclusion was that there was no other picking strategy that would have achieved comparable results. I realize there’s no way to prove it one way or another, which is kind of the point.