How 492 won the PAC NW regionals

[Ethulin]

Originally Posted by J Flex 188

While this thread and its data are rather interesting, I don't believe that the actual contents of the message (that is to say, that strategy and scouting are fundamental to winning a regional) are any different in past years. It only seems like a focus this year because of the fact that there is only one field object to manipulate, as opposed to a variety last year. Every robot has the same basic function.

My point was that this year a certain type of scouting was important. The type that required individual scouting rather than going by ranking and scoring.

Originally Posted by Rick TYler

Impressive scouting method. Nice work, and you certainly picked good partners for the finals.

I wanted to take this opportunity to point out to the utes here that purely quantitative analyses have to be subjected to some critical thinking before they are accepted. Since we were at the PNW regional, I'm going to go ahead and use that as an example.

1. Your quant analysis draws some misleading conclusions. Using my own team as an example, your quant analysis shows that we scored between 0 and 4 times per round, and you averaged that out to 1.6 per round. This analysis using the mean (average) number neglects that fact that our scoring did not have a normal distribution. We had an arm fail in three rounds (twice due to a PWM cable being knocked loose before the match, and once when the arm operator jammed the arm by moving it into the stop when our software limit detector had been accidentally overridden by some code changes). This means that we scored 0 tetras in three early rounds, but scored between 2 and 4 times in the other rounds. A more useful measure might have been the median (center value when ranked) or even the mode (the most common result) rather than the mean for this particular measure. Using 1294 as an example, our average score was 1.6 capped tetras per round. Our median was probably 3, and our mode was probably 3 or 4. My records aren't complete, but this is pretty close. This means that when our robot hadn't been sabotaged by our own team, we reliably capped 3-4 tetras per round. Your quantitative analysis missed this. (Now you could make the argument that being unreliable should count against a team, and I would agree with you. That's not my point. My point is that, by its nature, a simple formula cannot take everything into account.)

In our newest version of the scouting app we take into account mean, median , and mode. Also, the reason we have all those columns is precisely that, no one formula can take into account all the aspects of the game.

Originally Posted by Rick TYler

2. You have no sense of time series in your data. Bots (like 997) that started off really strong ended up fairing comparatively worse as others learned to defend against them. Likewise, other teams became stronger as their driving teams got better. By treating all data the same, you probably over-emphasize early results. Try a weighting factor over time next year and see if it changes your analysis.

Weighing all data equally requires a consistent bot, I will grant you that. But do we really want a bot that only worked well for the last 4 matches?

Originally Posted by Rick TYler

4. Some of your data are wrong, probably because of under-sampling. What process do you use to collect and quality-control it? You probably want to make sure to have different scouts evaluate each robot. As an example, in most of our matches we started off with a held tetra, yet you say "no" to this in your spreadsheet.

Collect: we had 4 scouts covering every match. 1 for each row, as well as 1 for entering data.
For quality control: well even though each person was assigned to a row, they were always keeping an eye on the others, just to make sure know one had fallen asleep at the wheel.
If we say "no" for your team it was only because that was what were told by your team. That n/y was recorded PRE games on thursday, so I guess one of your team mates was confused.

Originally Posted by Rick TYler

5. Unlike a baseball statistical analysis, your universe of measurements is too small to be statistically significant. This means that you should always apply human analysis before accepting the results. (You may, in fact, do this. I just wanted to encourage all teams not to blindly accept nice-looking quantitative results that may actually mean nothing.)

Well we went on the data we had to go by. Of course if we had more data it would be more accurate, but that is just wishful thinking.

Originally Posted by Rick TYler

Your methods are proven by your results. You went through the finals like Patton through France. As I said up there, I am just encouraging all scouting teams to not just trust their numbers without fully understanding what those numbers mean and where they came from.
I think you did a nice job.

Thanks.

[Josh Fritsch]

Scouting is very important part of the game every year. Over the past 8 years I believe our team has come up with a very effective scouting method. On Thursdays/Fridays of the competition teams of students are have a set group of teams they are to be scouting. They are required to get pictures and also watch all of their practice/qualifying rounds. While they are watching they fill out papers with different questions based on their performance as well as overhead sheets showing the path of their auton. This way we can overlap the overheads to see if they made any changes to their auton during the day. Then they go down to the pits and ask the same questions that they filled out while watching the teams to the people on the team to see if the information provided matches what they saw on the field and also to get any information they missed. All of this information is then entered into a database that we have.

Before each of our rounds we have our head scouter bring us our "cheat sheets" on each of our partners as well as our opponents as well as a proposed strategy based on the teams in the match. The drive team then takes the sheets and goes to see our opponents robots to see some of the technical aspects of the robots that the scouts up in the stands my not see(i.e. 6 wheel drive-can we push them/push us etc...). Then based on all the information provided by our scouts and the drive team, the drive team then goes to talk with our partners with a proposed strategy, if everyone is in agreement then thats what we go with, if not we make changes as necessary to come up with the best plan possible.

[Emil]

Originally Posted by Ethulin

What made you choose to you access rather than a php app to do data entry?

It is really easy to build a powerful GUI for data entry with MS Access. Basically, we saved ourselves the time and effort to create a PHP data entry interface. MS Access also provides a much nicer and faster GUI than an HTML page. This helps in navigating quickly through records.

[J Flex 188]

Heh, I agree with you there, but what I was saying was that this type of "individual" scouting has always been used by any successful team to get where they need to be. Ranking and scoring is never enough. Reliability, consistency, even the geniality of the coaches and the field crew all play an important role in getting together a successful alliance.

Originally Posted by Ethulin

My point was that this year a certain type of scouting was important. The type that required individual scouting rather than going by ranking and scoring.

[Ethulin]

Originally Posted by J Flex 188

Heh, I agree with you there, but what I was saying was that this type of "individual" scouting has always been used by any successful team to get where they need to be. Ranking and scoring is never enough. Reliability, consistency, even the geniality of the coaches and the field crew all play an important role in getting together a successful alliance.

Yep, just more important this year. This year your match results are influenced by 5 other teams per match rather than 3.