Strategy Sub-Team

[evanperryg]

Originally Posted by Michael Corsetto

However, I want to emphasize that, in practice, filtering pick lists based off of pit data can be very helpful. I call it "strategic generalization".

For 2010-2014 era games, 1678 found that making an initial filter based on drivetrain type improved the quality of our 2nd pick. I think this is what Gregor was talking about.

A better-said version of what I was trying to say. Although it isn't usually helpful for much else, pit scouting data can make a good filter.

Originally Posted by gblake

Do you suppose that it's possible that an even more effective and useful filtering or ranking could have been done using accurate observations of how the drivetrains (and drivers) performed on the field, instead of using the results of your pit scouting?
I'll wager a nice dinner that it is more than just possible; and I think that is what EricH was talking about.

I'm sure it's possible, but it would be highly subjective. I'm not saying that subjective information is bad, but trying to use subjective information to filter quantitative data is unreliable, at best. Say you ranked each team's drivetrain/driving abilities on a scale of 1-5, and only viewed the 4s and 5s. This, of course, raises a few inevitable questions:
-what's the difference between a 4 and a 5? You'll need a list of differences between a 4 and a 5, which will include even more subjective criteria.
-how important is the difference between a 4 and a 5, versus the more objective quantitative data? What if there's a team with a 5 that can't do any scoring, but there's a 4 that could score a few points in auto, and a few in teleop if needed?
It's probably possible, but it's the kind of thing that would get very messy very easily. It's better to do your first-order sort by easily-quantified information, then take into account more subjective information to do more detailed sorting. It makes the entire picklisting process more efficient.

[Kevin Leonard]

Originally Posted by evanperryg

/snip/
Say you ranked each team's drivetrain/driving abilities on a scale of 1-5, and only viewed the 4s and 5s. This, of course, raises a few inevitable questions:
-what's the difference between a 4 and a 5? You'll need a list of differences between a 4 and a 5, which will include even more subjective criteria.
-how important is the difference between a 4 and a 5, versus the more objective quantitative data? What if there's a team with a 5 that can't do any scoring, but there's a 4 that could score a few points in auto, and a few in teleop if needed?

Oh god subjective rating systems make me cringe.
For some reason, in 2013, we had a metric on some of our match scouting sheets called "Speed". It was a rating of 1-5 based on what they saw during the match. We never used it for strategy (so I don't know why we had it), but it was funny to see what different students rated different robots. Sometimes teams with 2-speed, aggressively geared drivetrains were given 1's and 2's, while some robots with single-speed, relatively slow drivetrains were given 4's and 5's. Most notably was the fact that somehow our single speed 12 fps tank drive from that year had the highest "average speed rating" at the event, due to obvious bias in the scouts.

We've tried since then to weed out poor, subjective rating systems like that.

[gblake]

Originally Posted by evanperryg

I'm sure it's possible, but it would be highly subjective.

If I knew a way to write this without having come across snarky, I would do it, but I not that good of a wordsmith today: Thanks for making me smile.

I understand what you are saying. I suppose I just reach a different conclusion when I think about the minimal set of observable variables I would use at the end of quals.

STEM robotics has plenty of room for both approaches.

Blake

[Citrus Dad]

Originally Posted by Kevin Leonard

Oh god subjective rating systems make me cringe.
For some reason, in 2013, we had a metric on some of our match scouting sheets called "Speed". It was a rating of 1-5 based on what they saw during the match. We never used it for strategy (so I don't know why we had it), but it was funny to see what different students rated different robots. Sometimes teams with 2-speed, aggressively geared drivetrains were given 1's and 2's, while some robots with single-speed, relatively slow drivetrains were given 4's and 5's. Most notably was the fact that somehow our single speed 12 fps tank drive from that year had the highest "average speed rating" at the event, due to obvious bias in the scouts.

We've tried since then to weed out poor, subjective rating systems like that.

Our solution is to rank robots within their alliance in an ordinal fashion (although there can be some cardinal ranking, e.g., no one is worth a 3 as the best), and then to pick the best of the match in each dimension, giving them a 4. We worked through the math and assuming the transitive property, the distribution of rankings fall quite close to a cardinal ranking system that relies on our "superscouts" keeping a constant metric across the entire tournament. The system worked extremely well in 2014. (As Mike has said, this system wasn't very important in this year's game.) We're looking at adjusting that ranking system to use the variance of the scoring distribution to standardize the metric.

I think this is where I take over from Mike. The next twist is that we use our quantitative scouting system results and the match scores from the previous competitoin to run predicted scores. We then add in our qualitative scores as defensive effects and minimize the squared error using Solver by varying the weights of those qualitative scores. We're then able estimate the defensive contribution expected for a given qualitative score and the relative weights for each dimension. For example, I think we found the 4814 contributed about 20 points a match (maybe higher?) in defense in the 2013 Curie Division which was multiples of the next robot.

Originally Posted by gblake

However, it sounds like you are trying to tell me that in your method, at the end of quals, if you ranked the teams according to the on-field performance your scouts see, you would then also adjust those ranks non-trivially based on pit-scouting data. That sounds a bit odd. I can certainly see making a case for it because the number of qual matches played usually isn't enough to supply an excellent assessment of each teams abilities. But ... with that in mind, I think we might at least agree that as the number of qual matches increases (and for the sake of argument, lets assume everything else is constant), the value of pit-scouting data steadily declines.

What I was saying in support of what I think EricH was saying, is that by the end of a typical tournament, I would side with him and be unlikely to let early pit-scouting data significantly alter any ranking I had created using on-the-field scouting.

If you guys do let pit-scouting data significantly affect your end-of-quals rankings I'm surprised. And, if you do, maybe that has helped you win, or maybe you have won regardless of any possible harm done by those changes. Get out a ouija to answer that one.

Regardless, congrats on the wins.
Blake
PS: In all of this I am setting aside aspects of team performance that depend on how well any two teams get along when they need to communicate/cooperate. For the sake this discussion, let's assume everyone is equal in that regard, and in other similar characteristics.

We don't use early pit scouting data other than to get pictures, and probably the drive train configuration. In fact as the season goes on we use previous competition results to pre-seed our scouting data, and we progressively replace that pre-competition data with actual matches. We run a regression of our scouting data on the OPR metrics to estimate the relationship to our quantitative scouting parameters.

We get pit scouting and drive team information as the competition goes on. We've had specific task questions the last two years about robot configuration that we can't really see from the stands, and that our scouts probably can't discern. Our drive team and match tactician gives input about working with particular teams.

We do the quantitative ranking and then we use the pit scout and drive team info to move teams up and down. The fact is that 10-12 matches is not enough observations, and those observations are not independent of each other. Teams change performance over the tournament. The initial ranking is a starting point. Then we introduce the non quantifiable factors such as drive train configuration (no mecanum until this year), robot configuration and team cooperation. And we include our past experiences. We moved both 1671 and 5012 up our list because of positive experiences with their organizations.

So in the end, it may not be pit scouting that trumps our initial rankings, but it is qualitative assessments that are not feasible by our field scouts.

[evanperryg]

Originally Posted by Citrus Dad

We get pit scouting and drive team information as the competition goes on. We've had specific task questions the last two years about robot configuration that we can't really see from the stands, and that our scouts probably can't discern. Our drive team and match tactician gives input about working with particular teams.

We do the quantitative ranking and then we use the pit scout and drive team info to move teams up and down. The fact is that 10-12 matches is not enough observations, and those observations are not independent of each other. Teams change performance over the tournament. The initial ranking is a starting point. Then we introduce the non quantifiable factors such as drive train configuration (no mecanum until this year), robot configuration and team cooperation. And we include our past experiences. We moved both 1671 and 5012 up our list because of positive experiences with their organizations.

This is almost exactly what we do, except our pit scouting question list is generally very long and uses language that approaches legalese. This will be changing in the coming season as we've found that a lot of the criteria go unused. Our pre-scouting data is largely qualitative but we have a spreadsheet that runs through a team's season info and shows what events a team was at, component OPRs, etc. The most valuable pe-event data we get is notes from scouters watching match videos. Early in the event, we rely on those notes heavily in doing match strategy, and we slowly progress over to using event data. Our picklists are primarily based on quantitative performance in relation to what type of robot we want. (i.e. the first picklist this year started with the highest-scoring feeder bot with a stack auto) Second picks are much more based in utility, not in scoring ability. Teams move up, down, or off the list based on qualitative data, our personal experiences with them, and drive team comments. Some teams are immediately put on the DNP list because of repeated bad experiences with them, although this is uncommon.

[Edxu]

Our strategy sub-team is pretty small (4 members, 1 leaving this year), but we play a great role at competition in terms of communicating between teams and planning matches.
We've recently converted to a tablet scouting system, which has the benefit of giving us real-time data on the performance of a team at a competition, allowing us to plan matches better, using hard data as the base.

To answer OP's questions:

-Strategy Brainstorming
At the start of every build season, 610 watches the game release and then we break for approximately 2 hours while each team member reads the game rules. In that time, a collaboration between the Strategy and Administration heads create a rules quiz that each member must pass in order to participate in the build process.

People who pass are allowed to participate in the Textbook Strategy discussion, which is basically a round-table discussion between the entire team, led by the Strategy division, where ideas are thrown around and considered. By doing this as a team, we can get on the spot estimates of whether or not something is feasible, as well as getting the attitude of the team on a certain idea. For example, this build season, there were a few people who were initially opposed to our idea of being a container specialist without the ability to handle totes, but by calculating the maximum possible score with containers won on our side, the Strategy division showed that the middle containers were incredibly important to any Einstein-winning alliance.

The Strategy should drive the Mechanism, not the other way around. By designing a component that specifically fulfills a task, it's more likely to be successful, and it also gives your Mechanical guys a goal to work towards (eg this year: We need a 4-bin mechanism).

In terms of Strategy's purpose at competitions:
-Plan matches with Alliance Partners using scouting data off tablets
-Organize expert scouts and create a list of special traits to watch out for (This year again: opposing canburglars, strong stacking robots that synergize and robots that may have the potential to add canburglars for Playoffs).
-Explain match strategy to the Drive Team, making sure that they know where our Alliance Partners will be throughout the match and what they'll be doing. This makes it much easier for our Drive Team to focus on what they need to do without worrying about the rest of our Alliance.
-Walk around the pits and act as "superscouts" who look for tiny traits that may be useful in an alliance partner.

If you have any other questions, feel free to shoot me a message.

[Citrus Dad]

Originally Posted by Edxu

Our strategy sub-team is pretty small (4 members, 1 leaving this year), but we play a great role at competition in terms of communicating between teams and planning matches.
We've recently converted to a tablet scouting system, which has the benefit of giving us real-time data on the performance of a team at a competition, allowing us to plan matches better, using hard data as the base.

To answer OP's questions:

-Strategy Brainstorming
At the start of every build season, 610 watches the game release and then we break for approximately 2 hours while each team member reads the game rules. In that time, a collaboration between the Strategy and Administration heads create a rules quiz that each member must pass in order to participate in the build process.

People who pass are allowed to participate in the Textbook Strategy discussion, which is basically a round-table discussion between the entire team, led by the Strategy division, where ideas are thrown around and considered. By doing this as a team, we can get on the spot estimates of whether or not something is feasible, as well as getting the attitude of the team on a certain idea. For example, this build season, there were a few people who were initially opposed to our idea of being a container specialist without the ability to handle totes, but by calculating the maximum possible score with containers won on our side, the Strategy division showed that the middle containers were incredibly important to any Einstein-winning alliance.

The Strategy should drive the Mechanism, not the other way around. By designing a component that specifically fulfills a task, it's more likely to be successful, and it also gives your Mechanical guys a goal to work towards (eg this year: We need a 4-bin mechanism).

In terms of Strategy's purpose at competitions:
-Plan matches with Alliance Partners using scouting data off tablets
-Organize expert scouts and create a list of special traits to watch out for (This year again: opposing canburglars, strong stacking robots that synergize and robots that may have the potential to add canburglars for Playoffs).
-Explain match strategy to the Drive Team, making sure that they know where our Alliance Partners will be throughout the match and what they'll be doing. This makes it much easier for our Drive Team to focus on what they need to do without worrying about the rest of our Alliance.
-Walk around the pits and act as "superscouts" who look for tiny traits that may be useful in an alliance partner.

If you have any other questions, feel free to shoot me a message.

Ditto! How did you explain what we do so much more elegantly than me?

[Citrus Dad]

Originally Posted by Citrus Dad

We have two presentations (slides + video) on our season strategy and scouting strategy. We are trying to get them up on our website.

As promised.