How 492 won the PAC NW regionals

[Ethulin]

Something is very different in this years game than the my knowledge has never happened before (or at least no recently).

3v3.

Though this seems like a pretty obvious thing, very few teams took it into account any way other than robot design, which even then only a few did.

Don't get me wrong, team 492 's bot Tyr was a very well designed bot, and I believe with that alone we could have placed very highly, it is not what made us win the Pacific NW regional almost unchallenged.

No, I would attribute our sweep of the PAC-NW to two other, though often overlooked, reasons, namely scouting and strategy.

Scouting:
To get truly accurate data people had to change their scouting this year. With a 3v3 game match ranking was just a horrible way to judge teams. So we made our own.

492's scouting this year I have to say was the best that I myself personally have ever seen. As soon as our robot was shipped we started working on our scouting application. We made a php based scouting application that allowed our scouts to input data from every match. Here is a screen shot:
http://trc.freedomdown.net/scout1.gif
We then coded a php info script too allow us to make sense of all our raw data. Screen shot:
http://trc.freedomdown.net/scout2.gif
In the app you can sort be what ever you want, call up data on what ever team you want etc.

At regionals we set up a wifi network with a laptop server running in the stands.

We assigned 4 team members full time scout duty. These members dutifly recorded data for every match in the entire regional.

This allowed us to find the truly two best teams in the regional to partner up with, team 1595 and team 604. Due to 604s ranking they were not picked as first pick by any other team, even though they were one of the best. We ended up getting our first two picks, and getting an owning alliance due too our scouting.

Strategy:
Sadly not until after nationals can I really go into what strategies we used that made our alliance so effective, but all I really can say is that with this 3v3 game, it changes allot. You have to know how to exploit being able to use all 3 robots at once to do different things, not have everyone trying to do the same.
One of the reasons that 604 had not done to hot before finals even though they had a really nice bot and very good driving is that they had no method to their madness, they could cap very well but just didn't know where. When we hooked up with them for the finals we made this all very clear.

Shout out too 1595 and 604, you guys were some rocking alliance partners, able to get a 40-40 tie going 2v3 without is in finals!

[n0t4g4iN]

That is a very impressive piece you got there. Great work on your scouting. Scouting can do a lot in a robotics competition and help make the best decision about strategy.

[Nate Edwards]

It was a great scouting system I think also, even though I am not sure where some of the information comes from because more then 3 or four times we had more then 3 tetras capped in a match. We consintently had tetras on top of the center row and the we would be able to cap some of the back row also. I know our max was not 3. Everything else seems very good. I was trying to figure out the quality and schedule rankings but I'm really confused where those numbers came from. We could see why you picked 1595 (consistant auto mode, 4 points off the bat, and then 4 or 5 caps a match it was awesome.) I was told at regionals you guys weren't going to make it to nationals, is this true. You will be missed if it is. What are the stats on 847 (#2 alliance captain at regionals)? They seem to missing from the top of the list and I wondered what thier stats were like)

[Ethulin]

Originally Posted by Nate Edwards

It was a great scouting system I think also, even though I am not sure where some of the information comes from because more then 3 or four times we had more then 3 tetras capped in a match. We consintently had tetras on top of the center row and the we would be able to cap some of the back row also. I know our max was not 3.

Hehe, when we were going over this data back at the hotel we were saying the exact same thing. How TF could our max tetras be sooo low? But then we went back and did the math, we even reviewed pictures we had taken with a really high quality digital camera to count them up. Something you should know though, those are just In-Game tetras, so any capped/dropped in autonomous are not in there. But after much review we found that actually that data was correct. It appeared that everyone thought that they were capping more than anyone really was. Right after the match ended all 4 of our scouts double checked to make sure that the number of tetras on the field accumulated to the number we had gotten for both alliances.

Originally Posted by Nate Edwards

I was trying to figure out the quality and schedule rankings but I'm really confused where those numbers came from.

Schedule is how hard their schedule was, based on how good the teams were that they were going up against.

Originally Posted by Nate Edwards

I was told at regionals you guys weren't going to make it to nationals, is this true. You will be missed if it is.

We are. We weren't until we won regionals, and decided when we got back too the pits. Exactly what happened last year too...

Originally Posted by Nate Edwards

What are the stats on 847 (#2 alliance captain at regionals)? They seem to missing from the top of the list and I wondered what thier stats were like)

Here is the same image with a little more high, so you can see 847, the bottom team.
http://trc.freedomdown.net/scout3.gif

[Crash852]

Originally Posted by Ethulin

We are. We weren't until we won regionals, and decided when we got back too the pits. Exactly what happened last year too...

Well so far, we have register with FIRST and it looks like we are going to nationals, but that really depends on if we can fundraise several thousand dollars by the time the money is due.

Hopefully we'll make it.

[Nate Edwards]

Originally Posted by Crash852

Well so far, we have register with FIRST and it looks like we are going to nationals, but that really depends on if we can fundraise several thousand dollars by the time the money is due.

Hopefully we'll make it.

Good luck, we will see you there

[Ethulin]

Originally Posted by Crash852

Well so far, we have register with FIRST and it looks like we are going to nationals, but that really depends on if we can fundraise several thousand dollars by the time the money is due.

Hopefully we'll make it.

Correct. During the regional a man from NASA came up to us. He talked with some of our mentors, and offered us a grant up to $5000 pending a budget. Well we only found out weeks later he meant a matching grant, so we are going to go into the fund raising furry.

[Eugenia Gabrielov]

That kind of upset occured at BMR as well. Team 93 had a 1-9-0 record, and nobody picked them except for Team 135. This alliance went on to become finalists.

Congratulations on your scouting. 461 used a similar system. We however assigned 6 students to be field scouts, one for each station, and 1 student to be a runner between the data input area and the stands. This way, things were a bit less hectic in the stands for entry and people could focus on one bot at a time.

[Ethulin]

Originally Posted by Eugenia Gabrielov

Congratulations on your scouting. 461 used a similar system. We however assigned 6 students to be field scouts, one for each station, and 1 student to be a runner between the data input area and the stands. This way, things were a bit less hectic in the stands for entry and people could focus on one bot at a time.

No this intruiges me. Did you use a web based (php) app such as ours, or did you use exel (like we did last year)?

[Emil]

We (461) did extensive scouting at the Boilermaker Regional.

We used the following for our scouting:
* Microsoft SQL Server for data storage
* Microsoft Access for data entry
* PHP for making our data available cross-platform as a website

I am in the process of adding a web page to our site that explains this in more detail. I will post the page as soon as I finish it (most likely some time today).

If you are going to the Midwest regional or you are in our division for Nationals we are looking for teams that will join with us so we don't each have to do our own scouting. If not, you can contact me and you may be able to use our software for scouting. I will be posting a new thread explaining this further here on Chief as soon as I get that web page written up.

[Ethulin]

Originally Posted by Emil

We used the following for our scouting:
* Microsoft SQL Server for data storage
* Microsoft Access for data entry
* PHP for making our data available cross-platform as a website

What made you choose to you access rather than a php app to do data entry?

[Rick TYler]

Impressive scouting method. Nice work, and you certainly picked good partners for the finals.

I wanted to take this opportunity to point out to the utes here that purely quantitative analyses have to be subjected to some critical thinking before they are accepted. Since we were at the PNW regional, I'm going to go ahead and use that as an example.

1. Your quant analysis draws some misleading conclusions. Using my own team as an example, your quant analysis shows that we scored between 0 and 4 times per round, and you averaged that out to 1.6 per round. This analysis using the mean (average) number neglects that fact that our scoring did not have a normal distribution. We had an arm fail in three rounds (twice due to a PWM cable being knocked loose before the match, and once when the arm operator jammed the arm by moving it into the stop when our software limit detector had been accidentally overridden by some code changes). This means that we scored 0 tetras in three early rounds, but scored between 2 and 4 times in the other rounds. A more useful measure might have been the median (center value when ranked) or even the mode (the most common result) rather than the mean for this particular measure. Using 1294 as an example, our average score was 1.6 capped tetras per round. Our median was probably 3, and our mode was probably 3 or 4. My records aren't complete, but this is pretty close. This means that when our robot hadn't been sabotaged by our own team, we reliably capped 3-4 tetras per round. Your quantitative analysis missed this. (Now you could make the argument that being unreliable should count against a team, and I would agree with you. That's not my point. My point is that, by its nature, a simple formula cannot take everything into account.)

Another way your Scout might have looked at an event like this would have been to qualitatively note that a robot which performed well at the Bellevue trials and on practice day failed to perform early Friday, and then worked fine after that. This is not an argument that we should have been chosen for the finals, but rather that I wanted to show you that a simple quantitative analyses won't always tell you what you think they will tell you.

2. You have no sense of time series in your data. Bots (like 997) that started off really strong ended up fairing comparatively worse as others learned to defend against them. Likewise, other teams became stronger as their driving teams got better. By treating all data the same, you probably over-emphasize early results. Try a weighting factor over time next year and see if it changes your analysis.

3. Have you done sensitivity analysis on your metrics? This means that you should play with your data to see if fairly minor input changes result in large outcome changes. I'm not asking you to tackle serious multiple-goal multi variable equations here, but you should know in advance if your model is robust enough to only reflect small ranking changes with small changes in the input data. Some models in the world fluctuate wildly with small input changes.

4. Some of your data are wrong, probably because of under-sampling. What process do you use to collect and quality-control it? You probably want to make sure to have different scouts evaluate each robot. As an example, in most of our matches we started off with a held tetra, yet you say "no" to this in your spreadsheet.

5. Unlike a baseball statistical analysis, your universe of measurements is too small to be statistically significant. This means that you should always apply human analysis before accepting the results. (You may, in fact, do this. I just wanted to encourage all teams not to blindly accept nice-looking quantitative results that may actually mean nothing.)

Your methods are proven by your results. You went through the finals like Patton through France. As I said up there, I am just encouraging all scouting teams to not just trust their numbers without fully understanding what those numbers mean and where they came from.

I think you did a nice job.

[J Flex 188]

While this thread and its data are rather interesting, I don't believe that the actual contents of the message (that is to say, that strategy and scouting are fundamental to winning a regional) are any different in past years. It only seems like a focus this year because of the fact that there is only one field object to manipulate, as opposed to a variety last year. Every robot has the same basic function. Scouting is always going to be important and I am certain that every successful team has its own individual methodology when it comes to this fact, that isn't to say that 492's system isn't impressive; rather it is another piece of the puzzle. At the same time, strategy should also always be a given when it comes to success in the qualifications and the eliminations. It isn't un-GP not to share strategies especially because of their heightened importance this year with only two ways to score. It takes effort to think about and determine what the best course of action is. I would argue that flexibility is the key point this year. Stacks and rows can change hands quickly, and it takes a fast, analytical person (luckily, our field coach happens to be one ) to make a decision on the fly. Strategies are important as well, but they are not the be all and end all of this year's game.

This also brings up a good point. Do not discount the scouting and strategy members on your team! Some of them may not have done hands-on work with the robot, but if they are able to tell you go to "X" goal and cap it and win the match, then they are as valuable as someone who spent the time building the arm that did the capping. Far too often there is a perceived notion that strategy and scouting represent the "soft" parts of the team, but let us all remember that with an efficient and effective system, a seemingly small time alliance can work its way up to become giant slayers.

[Goldeye]

Great job on winning, 492! Such a scouting database is quite an impressive feat. Having scouts focused enough to keep such a complete record is no doubt a challenge, and very nice work on the backend too. Our plan was to do something similar with manual input to a spreadsheet, but our scouts in the stands didn't come through with the numbers. (Although to be fair they weren't entirely informed that we wanted to compare them.) To see how big a difference such good scouting can make is really inspirational, though.

[sforsancho]

That is definitely a very impressive strategic aid, and if it helped you like that I'm very interested in better definitions of the different columns, after nationals of course.