Poor team scouting methods?

[platypus]

As a former scouting lead and general strategy guru, I can speak to this.

In my experience, objective evidence is king. Subjective information, regardless of the source, is riddled with issues. My reasons for avoiding subjective information:

1. Scouts form biases early and often. The number of times I've looked over the objective data and realized that my thoughts or someone else's are inconsistent with the data is astounding, and it's often due to the fact that a scout will see a team do well early and then over-credit that team for later alliance successes. The same concept works in the reverse (over-crediting teams that strike scouts as bad with later alliance failures), but to a lesser extent.

2. Opinions differ scout to scout. Everyone is rubbed differently by what they see in a robot. Period. Anything that is debatable as to whether the robot in question is at fault or another robot will have multiple opinions. Whose opinion do you trust? Any of them? None of them? What makes one scout more trustworthy than another? How do you determine any that objectively? See my point?

3. Some scouts are smarter/more experienced than others. Hence, point #2. Good luck determining how much intelligence/experience helps an individual scout - it differs from person to person.


Ultimately, subjective scouting often gives you more issues than helpful feedback. Sometimes you'll get lucky with it, but most of the time, these issues will bias your data and taint your results. Don't risk it.

Thus, I reiterate what has been said in favor of objective scouting. Go by the numbers. Crunch them in excel Friday night. If you can, set up a software database to do it for you, but if your team lacks the resources to put all of its scouts on laptops (very few teams can), then take the data on paper and enter it into the database on one laptop.

[kborer22]

Originally Posted by platypus

As a former scouting lead and general strategy guru, I can speak to this.

In my experience, objective evidence is king. Subjective information, regardless of the source, is riddled with issues. My reasons for avoiding subjective information:

1. Scouts form biases early and often. The number of times I've looked over the objective data and realized that my thoughts or someone else's are inconsistent with the data is astounding, and it's often due to the fact that a scout will see a team do well early and then over-credit that team for later alliance successes. The same concept works in the reverse (over-crediting teams that strike scouts as bad with later alliance failures), but to a lesser extent.

2. Opinions differ scout to scout. Everyone is rubbed differently by what they see in a robot. Period. Anything that is debatable as to whether the robot in question is at fault or another robot will have multiple opinions. Whose opinion do you trust? Any of them? None of them? What makes one scout more trustworthy than another? How do you determine any that objectively? See my point?

3. Some scouts are smarter/more experienced than others. Hence, point #2. Good luck determining how much intelligence/experience helps an individual scout - it differs from person to person.


Ultimately, subjective scouting often gives you more issues than helpful feedback. Sometimes you'll get lucky with it, but most of the time, these issues will bias your data and taint your results. Don't risk it.

Thus, I reiterate what has been said in favor of objective scouting. Go by the numbers. Crunch them in excel Friday night. If you can, set up a software database to do it for you, but if your team lacks the resources to put all of its scouts on laptops (very few teams can), then take the data on paper and enter it into the database on one laptop.

i have been doing scouting at events for 6 years, and now that im in college we still use the paper, picture, general pit scouting method. Especially now its a matter of molding the students to what you want to get out of them. For example, i try to sit with them all day thursday (if time permits) and will do an active commentary about the match that's going on. So we were watching matches in AZ and right away saw that 1726 was an excellent hurdler, and then we went into a quick description of how they hurdled. Now even though they were throwing balls on thursday that did not mean they could/would on fri/sat, but you know they are a team to watch out for. And sure enough, they were one of the best teams in AZ. We had another college mentor that was injured and he was just inputting match data into an excel sheet, honestly that sheet was useless with out a description. When you encourage descriptions you will mostlikely get their honest opinion about a robot as well, since the comments usually have to be short and to the point, "amazing or Terrible" you should be able to get a feel for the team by watching matches.

To me it would be like going to buy a car and only asking if it runs and what features it has, with out actually driving it and seeing for your self what work it may need, and if it actually does what the owner says it can do (aka pit scouting) It's all about conveying to the scouts how you want the data

In 2004 as a member of MORT (founding member of their now stellar scouting team), sitting in the stands for every match and being in a position to pick we were able to find robots that were amazing for us. 281 and 122, 281 i don't remember exactly what they werwe ranked, but they did well at palmetto, but our sleeper team was 122 Nasa Knights. This machine was truly amazing and was able to do it all, we were able to see that despite their poor performance in the quals, we knew they were a strong 3rd partner. If we had shown our team rep just the "raw data" there is NO WAY we would have picked them, that data does not show potential of teams, and i think you miss out on what could be the missing peice to you championship winning alliance.

[heather]

To me it would be like going to buy a car and only asking if it runs and what features it has, with out actually driving it and seeing for your self what work it may need

After using a scouting method where we mostly used subjective information in Trenton, my team was thinking of switching over to almost entirely objective information. However this^ makes a lot of sense. Does anyone have any suggestions about the best categories to look at and best types of descriptions to use in terms of the more subjective information?

Thanks.

[kborer22]

ours consists of a hybrid mode section, and just a blank spot to write down what they did, (# of laps, knocked ball, etc.) we are going to add a column for # of penalty points.

Then we have a small section for Drive Train that we would use to pit scout, (type of drive system, #of wheels/motors,speed) and space for comments on how it actually performs( fast, decent speed, slow, trouble turning, easily pushed etc.)

And then for Herding, hurdling, knocking ball each have a y/n choice to circle and then we have "How?" and the "comments"

"how" is a description of what mechanism they used, and then how effective.

If you want pm me with your email and i will send you a copy of what we use

[rogerlsmith]

Remember, anyone can go to:
http://www.firstobjective.org

Take 10 seconds and sign up, then pick My Team.

Once you are on My Team, pick 2008 Team Tool.

You can enter any FRC team number and get a nice report of their performance for this year.

If you have a mobile phone that can access the Internet, point it to:
http://www.firstobjective.org/myteam/teamdata.wml

This will give you the same info right on your phone.

AND IT'S ALL FREE!!!

Roger

[Uberbots]

our scouters are amazing at what they do. They have a system set up of 5 scouts and 1 head scout, who are each assigned to watch one robot during a single match. They record various characteristics... like driver experience, how much 'fumbling' they do while picking up the ball, average hurdling/lap times, amount of defense played on them, etc. our goal with the scouting thing is to pick out the best robots from the crowd, so if a good team gets paired with an incompatible alliance against a dominating alliance, the one good team will still get recognition amongst our scouters.

Its a neat system, and all without computers- its all paper, pencil, and mind!

[SunKing]

Oh too kind...

All this conversation has gotten me thinking though, about the difference between subjective and objective scouting. We've tried objective during the beginning part of last year, and my problem with that is that a robot that scored slowly with 4 ringers with no defense would register the same on a sheet as a robot that scored 4 under heavy defense. One's clearly better, but not according to the sheet.

Then again, at the Regionals this year though, I did see evidence of a lot of scouter bias. I guess once people make up their minds about a robot, it takes a lot to make them change their minds.

I think in the end I'd prefer a subjective system operating under a skilled, trained scouting squad. We'll have to see in Atlanta.

Speaking of Atlanta too, if any teams want to stop by Team 1124 in the crowds (look for the blue superman shirts), our scouting team would love to share our data.