Understaffed scouting

[Siri]

I think there's a lot of conflation of ideas going on here. What you do about scouting when you're resource-limited depends almost entirely on what you're trying to achieve. (Much like everything in life.) Are you legitimately aiming to be an alliance captain and make a pick list that can reasonably win you this competition? Then your best and only reliable solution is probably to not be understaffed (or under-trained, for that matter). Realize that this is a two step scouting issue: become an alliance captain and make your pick list.

But scouting that way when it's not your main/achievable goal that weekend won't necessarily help you. If your goal is to play well and get picked, 'gut' scouting isn't so bad. A few well-trained scouts/strategists can give you a lot of insight into your allies and opponents without actually tallying game pieces. In fact, when your team is understaffed but not under-trained in this way, qualitative can be better than quantitative. I've always opted for good qualitative over bad quantitative, and it hasn't let me down yet.

On the third hand, if you're aiming for either of these two and/or to get deeper into the FRC community, joint-team scouting can be great. Just understand it has its pitfalls.

On the fourth hand, if you goal is just to get better at the game (including scouting), you probably want a mix of qualitative and quantitative for your own team. I and most of the coaches I play with will keep at least one top scout on 'gut' duty--usually more than one--whenever we're fully staffed. That means at some point those guys need gut scout training. On the other hand, good quantitative scouting also requires practice.

Much like everything in this business, your scouting strategy needs match your competition strategy. Be honest with yourself - don't discount a strategy that (probably) won't make you Championship Alliance Captain when that's not your team's goal at the given moment.

[j93785]

what do you mean by scouting apps? Has someone developed an application for scouting matches?

[Kevin Leonard]

Originally Posted by Siri

I think there's a lot of conflation of ideas going on here. What you do about scouting when you're resource-limited depends almost entirely on what you're trying to achieve. (Much like everything in life.) Are you legitimately aiming to be an alliance captain and make a pick list that can reasonably win you this competition? Then your best and only reliable solution is probably to not be understaffed (or under-trained, for that matter). Realize that this is a two step scouting issue: become an alliance captain and make your pick list.

But scouting that way when it's not your main/achievable goal that weekend won't necessarily help you. If your goal is to play well and get picked, 'gut' scouting isn't so bad. A few well-trained scouts/strategists can give you a lot of insight into your allies and opponents without actually tallying game pieces. In fact, when your team is understaffed but not under-trained in this way, qualitative can be better than quantitative. I've always opted for good qualitative over bad quantitative, and it hasn't let me down yet.

On the third hand, if you're aiming for either of these two and/or to get deeper into the FRC community, joint-team scouting can be great. Just understand it has its pitfalls.

On the fourth hand, if you goal is just to get better at the game (including scouting), you probably want a mix of qualitative and quantitative for your own team. I and most of the coaches I play with will keep at least one top scout on 'gut' duty--usually more than one--whenever we're fully staffed. That means at some point those guys need gut scout training. On the other hand, good quantitative scouting also requires practice.

Much like everything in this business, your scouting strategy needs match your competition strategy. Be honest with yourself - don't discount a strategy that (probably) won't make you Championship Alliance Captain when that's not your team's goal at the given moment.

I have to disagree with a few points here.
The problem with scouting under the assumption you won't be an alliance captain is that sometimes you still end up an alliance captain.
Every team should have a picklist going into Saturday morning, because a) they could end up an alliance captain, and b) they could get picked by a team with no pick list.

In my opinion and experience, quantitative scouting is almost always better than qualitative. That being said, if you're unable to put together the people in your team or through multiple teams to have a quantitative scouting group, qualitative is better than nothing (and is often a good supplement to hard data anyway.)

[Sam_Mills]

Siri put it right, whereas I lost track of the problem statement. How you scout should depend on your goals. In my time as a scout, the goal was to make a pick list, but if you are looking to get picked, a better goal may be to scout weaknesses in potential opponents (ex: finding what 2013 cyclers could not drive through the pyramid so you know how to play defence on them).

If you are in picking position however, I would be wary of using your gut for anything more than choosing between two candidates who are worth the same amount of points on paper, but maybe play differently.

[Siri]

Originally Posted by Kevin Leonard

I have to disagree with a few points here.
The problem with scouting under the assumption you won't be an alliance captain is that sometimes you still end up an alliance captain.
Every team should have a picklist going into Saturday morning, because a) they could end up an alliance captain, and b) they could get picked by a team with no pick list.

In my opinion and experience, quantitative scouting is almost always better than qualitative. That being said, if you're unable to put together the people in your team or through multiple teams to have a quantitative scouting group, qualitative is better than nothing (and is often a good supplement to hard data anyway.)

Please allow me to clarify: aiming for the realistic goal of being picked is not the same thing as assuming that you won't need your own pick list. Never assume you won't need your own list. At a 2008 off-season, I had to pick a guy because I could read the number on his shirt. (We made semifinals.) Always make a pick list.

But pick list strategy and scouting strategy are not necessarily the same. In the best case, you want them to be the same, but if that's not realistic it doesn't necessarily make sense to conflate them:
Pick List: Every team from 1st to nth should write a pick list: you might have to use it, and it's a necessary skill to acquire should you wish to improve. But make no mistake, a winning pick strategy is very, very difficult to make (speaking as someone who's done it both right and wrong before). Like everything, Captaining takes practice.
My Point: What my argument does mean is you shouldn't be surprised your list isn't as good as one backed by an experienced, comprehensive scouting system. It can't be. Your [used as a general pronoun] list would not have been as good as theirs even if you tried to mimic their system, because you're just not ready for it. In fact, it would almost certainly be much worse that you actual list. Because there is such as thing a bad/badly used quantitative data, and it shows up a lot in untrained scouting systems, whether or not it's identifiable at the time. So do what you're most capable of that's most (likely to be) useful to you.

Perhaps more importantly, remember that being picked and forming part of an alliance that you want--particularly at upper-tier events--is in fact an active and difficult job. It takes energy and practice, and you need to decide how to allocate those resources. Having been both there and in the "Hey, you're an alliance captain!" spot, I still won't let the fear of Captaining prematurely interfere with giving the team its best possible shot at peak performance. 1640's system is getting better at Captaining, but I've also had great 'gut' scouts that help me play our best in quals while "selling" the team for different alliance strategies. That's not to say that qualitative is necessarily best when you're understaffed, or that quantitative doesn't help you with qual strategy. You need to have people who are truly good at either for it to work. We've evolved through approaches that fit our skill sets and situations at the time. (Notably Einstein Finalists 2014, Einstein Semifinalists 2013, MAR Champions 2013, MAR Champions 2012. We also won Philly 2011 as 2nd Captain.) Don't mimic, emulate.


On a separate note, I do coach and select from the "gut" (or just not purely quantitatively) in other situations. Picking needs quantitative data, but there's a lot more to it than that: how do we play together, how do we work together, how do you think their crew will handle the pressure, and so on. That discussion might make for another interesting thread.

[EricH]

I would generally regard "gut" data as being a tiebreaker at best. Something like "Data says these two are about the same, which do I think will work better?"

[Citrus Dad]

Originally Posted by josephus

We have a relatively small team but would like to have an effective scouting group. How might we scout matches effectively with fewer than six people? I'm not worried about pit scouting, as that can be done by one or two people.
--josephus

First, focus on WHY you are scouting? Are you expecting to be an alliance captain or a first pick? Do you want to use your scout data for alliance strategy or just for draft picks? Make an honest assessment of your team's likely success and position in the competitions. Being an alliance captain demands much more than being a draft pick, but you can still bring substantial value if you have a good means of identifying a 2nd alliance member that fits well with your robot. For alliance strategy focus on what your robot does and what are the other skillsets that other robots need to have to be successful.

Once you figured that out, then focus your scouting on the information you need. Try to keep it from being overwhelming for your scouts.

And as EricH said, pull in mentors and parents. We've done that in the past, even having junior mentors testing out specialized scouting skills, and I've sat in for students when they need a break. Really make it a team effort.

[New Lightning]

First to cover a few things on this thread. Using Google spreadsheets while not at an event could be a useful tool; however, since most events will not allow you to run your own WIFI, using Google spreadsheets is not a viable option while in the stands. Secondly for the person that asked, yes Scouting apps, many teams develop their own or use ones published by other teams. There is a whole thread on this topic Scouting the Scouting apps.

Secondly what has been mostly ignored is that in order to get the best data possible there needs to be a mix of both qualitative and quantitative data. Having one or the other is a very limiting factor and you may miss a key aspect of a certain robot that makes it either a great choice, or a horrible choice. That being said if your understaffed and you have to pick one over the other go with the quantitative. Quantitative data is unbiased and irrefutable. What did a team actually did, not what can the robot potentially do, not what the team said there robot can do, quantitative data tells you exactly how the robot performed. While qualitative data/ gut scouting is by no means bad there will always be some elements of human bias in this thinking and that can lead to problems when making a pit list and selecting people on the filed.

[BrendanB]

Great points in here mainly figure out what/why you want to scout. Aside from collecting numbers what I find more important is being able to look at a field of robots and determine what type of robots we want on our alliance and then rank the robots we see on the field that would best fit those roles.

Prioritize robots that have strengths you don't have. Pick robots who will work well with you. If all three of you need to go to the landfill to get totes you'll all be in each other's way. The same goes for the HP station.

The same can be said for autonomous: pick robots that have routines that work well with yours to achieve the highest score.

Keep an eye on the field for consistent performers who can perform tasks reliably from match to match. Keep an eye on teams progressing as the weekend goes on. Some teams increase their performance near the end of their rounds and can be huge dark horses in alliance selections. Find out why teams are under/over performing by talking in the pits.

Don't just look at the top robots on the field the more important partners are the ones who will be around for the last 8 picks and can often be the most important. Last year at early events it was hard to get a robot in the third round who could quickly and consistently gain possession for a three assist cycle or consistently score a ball in autonomous but a few were out there. Most of the time teams won't be in a position to pick the one of the top 2-4 robots at an event so don't dwell too much on them.

Having the numbers on who scores more totes/rcs in auto and teleop is very important but don't just focus on getting numbers because you need to examine each robot through a variety of lenses that numbers might not show. In previous years teams who had a low average score could have been slaughtered with defense in their matches while your higher average teams never had defenders on them. Defense isn't applicable this year since alliances are separate but whenever a scout tells me "This team is really good they scored xxx on their own" I want to know if they were defended.

Scouting with small groups isn't easy but its doable. You can throw 20 people at scouting but if they don't understand how to analyze a field of robots all the data in the world can't help you.

[jaykris284]

1261 is at Palmetto (a 1st week regional) but is also sending 3 or 4 people to Perry (also a 1st week regional) who will be scouting in collaboration with team 2974 Walton Robotics. We will share data from Palmetto and they will share data from Perry, it is a win-win for both of us.

[Caleb Sykes]

Originally Posted by BrendanB

Having the numbers on who scores more totes/rcs in auto and teleop is very important but don't just focus on getting numbers because you need to examine each robot through a variety of lenses that numbers might not show.

I agree that the perfect combination for scouting is a mixture of quantitative and qualitative data. However, few teams actually have a good enough scouting system to collect meaningful quantitative data, much less qualitative data. I've reviewed data from teams (including my own) after events, and at least half of the data sets I have seen are absolute garbage. I believe that almost all teams should focus first on improving their quantitative scouting before even thinking about qualitative scouting.

Basically, if your scouts can't reliably count the number of totes a team can stack in a match, why would you trust that their "qualitative" assessments of teams mean anything at all?

[TDav540]

With a small team, it may be smart to avoid collecting a large number of repeats. Have a list of the teams you would like to collect during which matches to maximize the number of teams covered. Yes, you will miss some matches, but it's better to collect some data on every team than skip some teams that could have a vitally important mechanism.

Additionally, take pictures during your pit scouting operation. Pictures will help your scouters/drivers with recalling the specific bot, so having those available will help tremendously.

Finally, as many others have mentioned, work with other teams in the area to scout effectively. Almost every team has some sort of scouting operation, and many rookie/small teams will be understaffed. Combining resources will be like coopertition, beneficial for all simultaneously.

[BrendanB]

Originally Posted by Caleb Sykes

I agree that the perfect combination for scouting is a mixture of quantitative and qualitative data. However, few teams actually have a good enough scouting system to collect meaningful quantitative data, much less qualitative data. I've reviewed data from teams (including my own) after events, and at least half of the data sets I have seen are absolute garbage. I believe that almost all teams should focus first on improving their quantitative scouting before even thinking about qualitative scouting.

Basically, if your scouts can't reliably count the number of totes a team can stack in a match, why would you trust that their "qualitative" assessments of teams mean anything at all?

We've had our issues with bad data at during events as we have worked our way through various scouting methods: pen/paper input into Excel and our tablet scouting from last year.

That has put us in positions where we have no accurate data during an event which has been caused by technology failing or the input/collection failing. Either way I can still rely on my scouts to give me and our pick list makers a good evaluation of robots. Last year this played a critical role at two of our events and many before then where we had to say, "Data aside how do we feel about this robot from what we've seen".

Like many have said in this thread you need to prioritize what you are scouting and why which is why we work hard to make sure our scouts know what to look for and why. If our data collection fails we have over half a dozen students who can offer their honest opinions regarding what they saw and work through their thoughts as a group to get a good feel for the field of robots.

I think too many teams overlook the aspect of having a serious discussion regarding scouting and why teams need to do it which is why they get poor results. If you want to be successful at an event you need to treat scouting like you do the drive team in terms of importance and not a separate group in the stands.

[Siri]

BrendanB's discussion of consistency in pick lists reminded me of this, so I'll just leave it here.

Originally Posted by Caleb Sykes

...Basically, if your scouts can't reliably count the number of totes a team can stack in a match, why would you trust that their "qualitative" assessments of teams mean anything at all?

I see where you're coming from, but there are also a lot of different reasons for "garbage in, garbage out" scouting. Quantitative data can be really, really bad in weak scouting systems--but is it because scouts are untrained? Unmotivated? Overtaxed each match? Exhausted?

Understaffed quantitative scouting has a tendency to be simultaneously boring and utterly exhausting, while not feeling particularly useful. It takes a lot of bandwidth to count game pieces and even more to track important quantitative movements. Qualitative scouting is like that, except it can be worse. Because (as you pointed out), if you're not well enough trained to count totes, it's unlikely that you're good enough with FRC to give meaningful qualitative input. That's the usual route. But--and I've been down this route as well--maybe you're understaffed but it's with trained, experienced scouts. It can be better to use those skills and keep them engaged with qualitative discussions than to bog down some of your best strategists in tote counting when you're not going to get good coverage anyway. It's all resource-dependent.

By the same token, if you're trying to make a team culture that wants to scout, selling it as "sit here and count totes" can be rather trickier than "sit here and talk to me about matches. What do you see? Was that a smart move? What will they do in match 34? What should we ask them in their pit?" I struggled for years to build up a quantitative scouting system in a very anti-scouting environment. Qualitative is sometimes an easier first step. Not always, and it's not necessarily more effective in the short-term--in fact it probably isn't, but very little is effective in weak scouting systems anyway. But it's a way to fix some of the "just make up numbers" plague in match scouting.

You're going to want both sides eventually. Scouting isn't just about how many totes someone scores. It's about predicting opponent's match strategies and individual play responses. Making a pick strategy isn't about ranking the highest scorers. It's about strategic decision trees and adaptability. In the end, blue banners aren't won by numbers on a page. They're won by allies that work well together, know their opponents and can manifest their work in their scores.

[Citrus Dad]

Here's one scouting app that might be useful. Look for others as well.
http://www.chiefdelphi.com/media/papers/3098

Also, "gut" comes into play for pick lists no matter the scouting system. The arguments at our draft sessions are epic and famous on our team. We're all passionate and have great observations (just mine are better... ). Our final draft list may look little like our initial list, but we had a good starting point that we trust.