Complete scouting overhaul - where to start?

[Talks Too Much]

The way my team scouts is pretty lackadaisical right now. The head scout hands out papers before the start of each match, one sheet per bot, to whomever is around. Scouts don't really get any training and don't necessarily know what they're looking for. Our system is kind of sloppy, as we end up with tons of forms, six per match, floating around, and those forms may or may not contain useful information. So, two main questions:

1) How do you choose scouts? Do your scouts get training? If you answered yes to either, can you describe a bit?

2) How do you scout? Netbooks may be an option for us, but I have no idea what software we would use. A googledoc at first seemed like it could work, but after some thought it seemed it would be too clunky.

[LeelandS]

1) All non-drives team and non-pit lead students will scout at some point. We have a schedule for each student, with rotations between scouting and pit work, so students don't spend all day in the stands, but still do their fair share of scouting. Scouts are trained during competitions. Training consists of a quick-run through of the application (it's pretty simple, so it doesn't require much time).

2)Traditionally, we have used pencil and paper scouting, but more recently we have given a run to using computers for scouting. We use a homemade application that feeds information from each of the scouting computers to a main computer that compiles the data into a spreadsheet. I'm not particularly familiar with the program (I've been on drives team since we started using it, so I haven't had the chance), so I couldn't provide many more accurate details other than that. If you don't want to create your own application, there are others out there. There are several scouting applications out there for iTechnology (iPods, etc), as well as internet-based scouting applications. Cheesy Scout is a great web-based scouting application to use, in my opinion. I would recommend looking into it if they sounds like your cup of tea

I hope this helps! Scouting is very important to becoming a successful team, so I wish you the best of luck!

[Phyrxes]

Before you jump feet first into a technology solution make sure you can get power to it for the entire event, at the events my team attends power in the stands is nonexistent and I haven't been willing to dump the money into power inverter(s) and batteries to go along with a technology solution.

Technology wise I've seen the following implemented in some form or another:
DS, Smartphone, Tablet, and Laptops.

If tablets get cheap enough before this build season we might consider it but I wonder what kind of battery life we are looking at under heavy usage.

Speaking of "cheap" tablets rumor has it you can boot a nook color from and SD card to run a ROM and if you take the SD card out its a normal nook color again.

All that said we still use paper forms, clipboards, notebooks, and the magic white board!

[ttldomination]

Phyrxes makes a good point about technology needs for an overhaul, and I will make a point about the determination needs.

Make sure that if you do this overhaul, everyone is on board. For the past two years, I've spent a lot of time and effort searching for scouting solutions, convincing the students to use this method, and then making sure the students have everything they need, only to see the blunder of one careless person cause the entire operation to go haywire.

This is a full time job, and the dedication required to do this at an all-star level is sometimes more than what a team can afford.

As for your questions.

(1) We've only recently started placing an effort in scouting. So we look for dependable veterans to pick up the scouting ticket. No one really steps forward to say that they're going to scout, and well, I think that's played a part in our short comings.

(2) There's a ton of useful software out there. My favorite has to be the Cheesy Poof system. It's a nice, comprehensive system which allows for the collection of quantitative and qualitative data, and it presents the data in a nice, manageable way.

Dr. Ed Law, goes by 'Ed Law' on the forums, also posts scouting spreadsheets of data for teams in the events. These spreadsheets are all numbers, but they give a person a good feeling about how things are moving.

- Sunny G.

[gblake]

Originally Posted by Talks Too Much

The way my team scouts is pretty lackadaisical right now. The head scout hands out papers before the start of each match, one sheet per bot, to whomever is around. Scouts don't really get any training and don't necessarily know what they're looking for. Our system is kind of sloppy, as we end up with tons of forms, six per match, floating around, and those forms may or may not contain useful information. So, two main questions:

1) How do you choose scouts? Do your scouts get training? If you answered yes to either, can you describe a bit?

2) How do you scout? Netbooks may be an option for us, but I have no idea what software we would use. A googledoc at first seemed like it could work, but after some thought it seemed it would be too clunky.

You start by asking this question:

0) Who is telling you that they need information, and what information do they want to consume?

Do not assume that you should be doing scouting.

If there is no demand for information within your team, implementing the tips others will give you will just be a frustrating waste of time.

If some part of your team is hungry for information that will allow them to perform better, then scouts collecting that information will feel useful. Scouts who feel useful are usually not lackadaisical.

Once the important info is identified you then can focus on how to gather that information. The info might not be the information most other teams need, or the information that you currently think your team might need. Until you know what is needed, it's hard to pick the right method and tools for collecting it.

So... First, do your market research to identify your consumers, and identify what they really want (and will take the time to use/consume), and then create the product that satisfies their needs.

Blake

[Basel A]

Originally Posted by ttldomination

Mr. Ed Law, goes by 'Ed Law' on the forums, also posts scouting spreadsheets of data for teams in the events. These spreadsheets are all numbers, but they give a person a good feeling about how things are moving.

- Sunny G.

Dr. Law, actually, and it's true, his often-updated spreadsheets are as good as you can get without actually watching any matches.

What Mr. Ross notes is very important. Scouting should not simply be gathering information, it should be gathering information for a defined purpose.

Purposes could be to decide match strategy, whether in qualifiers or eliminations, or to choose alliance partners. Once you know what you want from scouting, you can begin to discuss how you want to obtain that data.

Sometimes, categorising the robots as "defenders" and "scorers" is all you need. In other cases, you want accurate, precise scoring predictions for particular teams.

Most teams, to obtain data, use scouters in the stands filling in sheets of subjective (qualitative) or objective (quantitative) data. This data is sometimes compiled into spreadsheets and should be used for scouting meetings (used to decide team strategy, both for matches and selections). Other teams utilise match video or other resources. But, when it all comes down to it, what matters most is obtaining all the data you want to or can use and no more.


As for actually answering your questions, I would suggest that scouters be as prepared as they need to be. If they're analysing robots strictly by watching match video, they'd have to know strategy very well and be extraordinarily observant. However, if they're simply marking how much a robot scores, penalties, etc., they only have to be trained insofar as being able to fill out their sheet. The "training" for the former is usually just practice. Practice can be over years of watching FRC games or hours spent watching robots and strategies for a specific game. It's simply not an easy skill to teach or master, especially with how quickly strategies evolve over the course of a season.

We scout using a combination of methods. We have scouters filling in objective data about robots for all robots in every match. This is compiled into a spreadsheet and the resulting data is used for all match strategies. The data becomes more accurate as more matches are entered into the system (later in events). We also have a few scouters who watch matches and write subjective notes, but their role is limited due to the difficult nature of writing accurately from a subjective standpoint. In the past year and half, we've recorded matches and used these as references during scouting meetings, when necessary. Like in sports, game film can be used both to better your own play and anticipate that of your opponents.

[Talks Too Much]

That's an excellent point. Our main scouting goal is to gather info about how the other teams play, so we can tailor our strategy to who we're playing with and who we're playing against. I think one of our problems last season was that the information we were getting seemed directed towards choosing alliances, and we were very unlikely to be in a position to need that information.

[jwfoss]

We have been working in the offseason to document our scouting process and create a manual to train new members of the team. Our goal is to ensure that the students understand the importance of scouting. If you click on my name in my signature you will be directed to my blog, scrolling down a bit you will find our draft "Scouting Manual".

This year we also posted our excel based scouting system here on forums for use by anyone. I believe this worked out well for us and many teams. I am not 100% sure we will be doing the same thing again. Here's the link to that system: http://www.chiefdelphi.com/media/papers/2450?

[Phyrxes]

Off season events and non FRC robotics events you team is involved in also offer great opportunities to test scouting materials and/or train scouts.

[Talks Too Much]

Thanks for all of your help everyone. We have an off-season event in November, so hopefully we'll be able to experiment a bit there.

[Brandon Holley]

Theres a lot of information on this forums in relation to scouting so you are definitely in the right spot as far as that goes.

As for your specific situation, the key piece of advice I'll give is to make sure the system you are using fits your team. Some teams have the ability to use very elegant scouting systems, while other teams simply rely on pen and paper.

For us, we spend a great deal of time developing an easy to use and effective scouting sheet during the build season. It goes through many iterations and trials to determine if it will function as we hope. Usually the sheet ends up being rather small so we are able to fit 6 or so "sheets" on one piece of paper. This ends up cutting down the number of floating sheets quite considerably. You simply use one or two sheets for one team's entire event. We then take this information and compile it into spreadsheets and make various calculations from it using metrics that we develop for each game.

Examples of useful metrics would be, hurdles per match (2008), moonrocks scored per match (2009), goals scored per match (2010), tubes scored per match (2011), average minibot climb time (2011), % consistency of minibot deployments (2011) etc.

We combine this with our more subjective notes that we gather about a team throughout the match. These subjective notes are extremely important, and can only really come from experience. For example, this past year at the Boston regional we were fortunate enough to seed #2 and be able to choose our alliance. Our first pick was a no brainer, team 2648 had been performing fantastically all day, and their scoring metrics matched our own robots in many cases. For our 2nd pick, we had our eyes on a couple teams, but one in particular which was 222. They hadn't competed yet on the season (this was week 6) and were experiencing some issues throughout qualifications. Our scouts kept tally of how many tubes they scored and how their minibot performed, but we were really trying to look beyond that and see how their team was progressing. Fortunately, no one else seemed to pick up on their development and we scooped them up in the 2nd round. We managed to make it to the finals and lose by the skin of our teeth in the 3rd match. These subjective notes by our scout team are what made this decision possible, and what made our event so successful.

Thats just a few notes on how our team does things. Maybe it will relate to your team, maybe it won't, but it will hopefully give you an idea on how you can mold a system to fit your team.

Good luck!
-Brando

[Jon Stratis]

Originally Posted by Talks Too Much

1) How do you choose scouts? Do your scouts get training? If you answered yes to either, can you describe a bit?

Everyone who sits in the stands at any point scouts. That pretty much means everyone but the drive team and a few other "permanent" pit crew individuals.

We started training last year. A few days before the event, our lead scouter walked the entire team through the scouting sheet, explained what we were looking for, and then had them score some actual matches from video footage for practice. Each individual watches and scores 1 robot in each match. We do our best to keep the scoring sheets to factual information (number of tubes scored in auto, number of tubes scored in tele, minibot position, number of logo's completed, etc) in order to remove any concept of "liking" or "disliking" teams - we want nothing but the facts at this stage.

2) How do you scout? Netbooks may be an option for us, but I have no idea what software we would use. A googledoc at first seemed like it could work, but after some thought it seemed it would be too clunky.

We start with everything on paper - The lack of availability for power in the stands and the sheer cost of a more "high-tech" solution is currently preventing us from going paperless. The scouting sheets can usually hold 3 matches at a time. Once a sheet is done, we funnel it to another team member (or if no one if available at that time a mentor) in order to put that data into an excel spreadsheet.

The excel spreadsheet has a few different pages on it. The first page is set up as an input form, laid out exactly like the scoring sheet. It has a nice, large "Submit" button on it that launches a macro. The macro takes the input form and condense it into a single line on the second sheet, then clears the form for the next entry. The second sheet is basically a large table with 1 row per robot per match (so 6 rows per match, 1 for each robot), and is essentially the raw data for each team. From there we can fire off another macro to condense each team into a single line on the third page - essentially a summary view with information like average tubes scored, average minibot position, and calculations like the OPR that's often talked about here.

All of this lets us sort the data in various ways to see how good teams are, what position they played most often (defense or offense), and forms the basis for our strategy going into every match.

We can also use this for alliance selection. Once we determine what specific stats we want on the team (for example, a second offensive robot and a defensive robot), we have another macro that will simply list the teams in preferred order by that stat. As alliance selection continues, we input selected team numbers in another column and they are automatically removed from our list. That way, it's a simple matter of reading the number off the top of the list to pick an alliance member. We also have a separate list of "preferred" teams we work well with (based on driver input), and a "blacklist" of teams the drivers feel we don't work effectively with (lets face it... Even though this is FIRST, not every team is going to have the same concept of how an alliance should work together, and some of those concepts, regardless of how good they are or the results they end up with, clash).

The biggest issue we have is making sure we get the information to those who need it (like the drive team) in a timely manor.


Perhaps the best part of the spreadsheet is our ability to expand it very rapidly. If, at some point, we gain the ability to go paperless, we can have 6 laptops/netbooks loaded up with the spreadsheet and everyone entering data directly. Then we just have to take each spreadsheet and copy the second page into a common "master" sheet - the rest of the work can then be done by launching a macro.

[Phyrxes]

Having all sorts of empirical robot data is nice but make sure you drive coach and your head scout are on the same "wavelength." Our drive coach requested we make him a one page overview for a match that contains the following generic information and any important game specific details.

He prefers the head scout and/or myself walk up hand him the sheet, let him read it, ask any questions he has, and then we are out of there if we don't need to discuss anything else and he can then talk to the drive team in peace.

I've attached a version of this sheet from Lunacy for reference. While it doesn't provide the drive coach with any numbers as most of the sheet is subjective, it provided him with what he needed to start forming a strategy. We still use a variation of this form for qualification matches.