As the title states, I am very curious on how teams discuss pick lists for their team. Particularly if you have a system where you categorize teams into classifications (For example this season [Cargo Bot vs Hatch Bot, etc]). I am also curious about who is involved with your decision making process (Whole team vs small group). Lastly, if a team finds themselves in a position where being an Alliance Captain is unlikely, do they still scout and create a full pick list? Or a smaller list aiming at 3rd picks specifically.
We determine what we need to win, in both first and second picks. This year we needed a climb as ours was both slow, and would take a lot if work to do in matches as close as elims. We also needed a fast cycler to cycle with us. We also said that defense was a must. From there, we go through all the teams and sort them as first or second pick. We sort them relative to eachother. We start with one, put them in first or second, and say is this team better than this one, in what areas, and why. We invite the whole team, as everyone’s perspective is appreciated. However, it usually ends up with just the scouters talking, with a few comments from the other people. Now, we are by no means a top team, but we take our scouting very seriously.
When you open up the discussion to the team, is that just preliminary discussions or do you make decisions right then and there?
We do the enture process in one meeting. If you show up, you stay for all of it. I have found it very useful as sometimes the scouting team forms our own biases towards teams. Plus it makes it a true team decision.
So for this year, differentiating teams on hatch vs cargo was not very useful since combo cycles were so difficult. It is still important to differentiate picks, because often there are 3 distinct roles that must be played. This is not always the case and sometimes 2 roles are similar enough to have 1 list. For the group size, what we have found was that a small group including drive team, lead scouters and a few mentors, facilitates a process that is both comprehensive and as quick as possible. It is always important to scout, no matter if you are ranked dead last, And making a pick list is just as important. If you become a first pick to an alliance who is unprepared it is your job to help the captain in the next selection and even if they are prepared it can never hurt to make sure everyone is on the same page and has not overlooked a sleeper bot.
My favorite pick list discussions always end in more of a choose your own adventure game than a simple list of teams.
It usually involves all mathematical possibilities going into the day at elims and lists must ALWAYS include more teams than are available left. We like to include every team at the event in order, but sometimes that isn’t possible.
The strategy meetings are always open to everyone, but many (often my drivers and strictly technical folk) choose not to come to prioritize sleep or other competition activities.
This method was inspired by 4488 who runs an awesome strategy operation and has prepared us well for every alliance selection in recent years!
Our secret method has been unveiled
1293 finally needed a pick list meeting this year, and while it didn’t yield banners (out in quarters at Palmetto and Smoky Mountains, both times as the captain of the lower seed) it was fundamentally sound.
First thing we did was scouting. Our spreadsheet calculated each individual robot’s contributions in a match, which in hindsight could’ve also stood to have an unweighted cycles-per-match column. (We could see the cargo and hatch breakdown and calculate it, that just meant math.)
If you weren’t putting up 6.00 points per match (that is, crossing the auto line and getting back on Level 1), you were at the very bottom of our list. No discussion, no thought, just the bottom.
If you were on our drive team’s Does Not Play Well With Others list, you were probably just ahead of that first group. I think we sussed out one team to be a little higher at one event, but still lower than they would be on raw performance.
Then we started to discuss our main needs and where teams fit in. We were good on the cargo ship, reasonably ambidextrous between hatches and cargo, and could cargo the rocket. So our shopping list included higher-level Hab climbs and rocket scoring. Once we got the obvious top-tier out of the way, we’d sort out that middle tier we were also occupying. That often meant shifting down robots that were in the category of “quality scorer, but playing the same game as us”. With how we shifted to defense at Smoky, we probably shouldn’t have put as much of a penalty on that attribute.
From here, it’s a lot of sorting out based on the spreadsheet, and a lot of debating whether one team is better than the other. And sometimes, it really comes down to making it an A-or-B conversation, then repeating. It’s tedious, but necessary to ensure someone isn’t missed. Eventually, we would get worn out from staring at it and paste the remaining lower-mid-tier in points contribution order. (Not ideal, but hey, honesty.)
Every single team wound up on our list, even if at the bottom, to ensure we didn’t miss somebody.
This season I was the primary person creating pick lists. I’m not a huge stats person so I mainly make judgements based off of performance. However, that only really works for the first 8-10 teams. So as I went farther down the list I used scouting data to determine what teams in what order go on the list. If I get stuck, or have my preliminary list complete, I talk with the major scouts on our team and our scouting mentor about the list and create a final list. Of course this isn’t the only way to do it, but this is how we do it.
With varying levels of success, here is my pick-list strategy:
Before the Event:
The night before picking
- Don’t talk about individual robots/teams
- Identify what would be a solid elims strategy, if the two best robots teamed up, how could we beat them? Does that strategy apply to less-powerful foe?
- Identify what we can do in that strategy and what traits we need to fulfill our needs
- Pick out the traits that would make a perfect first pick and rank them from most important to least important. Repeat with the 2nd pick.
- Write down every team and relevant stats on an index card
- Start sorting teams based on ranked traits
- Remove teams that are clearly not a good fit
- Some years (2017, 2018) the first and second pick are really just one list. Some years (2019), there are more specialized roles. Sort teams as appropriate.
- Write down the top 24 and any teams we want more information on.
The day of
- Scout the teams we want more information on.
- At least an hour before picking, re-convene to determine if the additional information has changed your rankings
- Write down our finalized list(s).
- Trust the list.
I go into more detail and give rational in this post.
We’ve got all our scouts inputting data into our scouting app, which feeds into a nicely sortable database.
This year, strategy in regionals was pretty easy. We knew that we needed a partner to put up big points, and we needed a partner to hold off the other alliance. Due to robot limitations on our end, we were looking for partners with level 2 starts and level 2 climbs.
We started by filtering out any teams that were sitting dead on the field, or scoring 1-2 on our “robot stability” and “driver skill” metrics. We split the remaining teams into teams that have level 2 starts + climbs, and teams that don’t. From there, we sorted by average game pieces per match, and demonstrated defense skill.
We make sure that we have enough “first picks” chosen so that even if other teams are picking directly from the same list as us, we still have a good partner ready. Second picks are similar, but a little harder to scout for because we need a longer list. We also make a list of teams to take a second look at, typically if we think they have an un-demonstrated capability or if their performance went up or down sharply over the first day.
Edit: Watching match videos is critical here. Nobody makes it onto a pick list without a room full of people watching a video of one of their matches.
Our team’s process is very similar to what @Katie_UPS mentioned in her post. Do that and you’re already ahead of so many other teams that “scout”.
It was mentioned if it’s worth scouting even if you aren’t going to be in a picking position. I would say that it is. Two reason. Number one, scouting data is important throughout the event, not just for picking. Number two, even if you aren’t going to be in a picking position at your current event. You may very well be in a picking position at a future event and practice makes perfect. Going through the motions is important, especially if you want to improve your process.
To build on reason number two: If our team is pretty sure we won’t be in a picking position, we will still create a pick list and theory craft some of the alliance selection based on our data. That way we can corroborate our list with the selection results to see how well we did based on our data. That affords us the opportunity to self correct and tweak our scouting/picking approach and process for future events/seasons.
This is actually something I would suggest all teams do regardless of their picking situation. Especially if there was some oddity that stood out from your list during the selection process. You can learn a lot by reviewing things after the fact. (One word of caution, as with most data of this kind, it’s not always 100% and it won’t tell you the whole story. Especially when you get into the qualitative stuff. For example: Friendships & Feuds impacting a teams decision to pick one another.)
Additionally, If you’ve had students scouting all weekend, you’ve got to do something with that data. If you don’t use the data, scouting just becomes busy work. Which could negatively impact the culture around scouting on your team.
We begin with an alliance strategy as part of kick-off discussions. It helps us focus on game play and our selected roll through out build season and practice. It also helps you create scouting data points important to your alliance plan. The alliance strategy may need adjustment after we see actual game play and competition depth.
We get opinions and input from anyone on the team that wants to contribute to the discussion. But the final list is usually generated by key scout team members with input from the drive team. Too many people involved in sorting the list usually slows down the process.
We always create a full pick list. Never assume that teams you believe will perform well will be picked.
On 1678, our scouting system provides a massive amount of data on each robot at every competition we go to, which allows for an effective picklist meeting. Our scouting whitepapers can be found here (the 2019 whitepaper is a WIP and will be released soon). Below is an in-depth explanation of our picklist process.
Here is a list of things that happen before our picklist meeting:
- Game-specific robot evaluation in the pits (this season, we tested how good robots were at driving onto forks).
- Throughout the first day of quals, the strategists and a couple of mentors compile a short list of likely-DNPs based on observations from qual matches and robot data collected in the pits. DNP criteria obviously varies from year to year as the game changes; this season, a big factor was the type of drivetrain robots had, which affected both their ability to play defense and drive onto our ramp for our triple climb.
- Scouting data is imported into a spreadsheet optimized for viewing and sorting through data.
- Robots are pre-ordered based on a first and second pick ability metric, calculated by plugging scouting data into a pre-determined equation.
Our super scouts, strategists, pit strategist, and mentors attend the picklist meeting on the night before elim matches. With this setup, we then begin the meeting and execute the following steps:
We start by reviewing each robot on the likely-DNP list. A justification for DNP’ing them is given, and if no one objects, they are officially DNP’ed. However, if at least one person objects with a valid argument, the robot is not DNP’ed.
Next is a round of “Hot or Not” (also known as “Robot Tinder”), where we go down the list and compare each robot with the robot above it. If we think the robot is performing worse, we leave it where it is. If we think it’s performing better, we move it up the list accordingly.
By this point, the first pick list is pretty clear and is usually only 2-3 robots long for regionals, and may stretch longer for Houston champs. After “Hot or Not”, the first pick list is finalized.
To sort through second pick robots, we first identify which robots stood out to strategists and mentors during matches in their notes. We watch match videos of each robot and adjust their position on the list accordingly. This is tedious and usually takes a couple hours to do, but we believe it’s fully necessary to make sure we don’t miss anything.
With our picklist drafted, we’re ready to enter the second day of quals (for regionals). We also identify robots to pay close attention to, especially when they haven’t performed a certain task that we want to see (such as starting off HAB level 2).
On the second day of quals and before alliance selection, the strategists watch the robots on the second pick list and reorder them throughout matches based on their performance the second day. First pick robots may also be reordered here, but it’s rare. More robots may also be DNP’ed if they are significantly under-performing or are having technical problems. Towards the last few quals matches, we also approach our planned alliance partner to share pick lists and make any necessary changes. After the last quals match, the finalized pick list is sent to the pit strategist. The list is usually 24 robots long for regionals (including ourselves) and 32 robots for champs, to ensure that we are prepared for every scenario.
I apologize for the lengthy post, but feel free to ask me any questions you might have!
- Bucket teams into High/ Medium/ Low/ No/ Game-Specific Categories as a team
- Small groups stack rank each category
- Lead scout(s) make adjustments up until picking time
- Whiteboard or text choices to field rep
Can you shed any light on what that equation looks like, or how it’s determined? Learning how best to process the data we have is always a challenge.
I also like Robot Tinder. It sounds like a more organized version of what we already do.
Edit: note to teams: Processes like this are why it’s not helpful to go up to a high performing team and say “hey, I think you should pick us.” Team testimony isn’t a category they sort by. If you want to sell yourselves as a 2nd or 3rd to a top team, give them something more like “check out our ability to fill the cargo ship under defense in Q95” or “look how partner-compatible our climb is in match 57”. Give the scouts something concrete to chew on.
Look at the Scouting System Development videos and slides on this page for our team’s approach. It is constantly evolving and too long to discuss here. https://www.citruscircuits.org/fall-workshops.html
For sure! Our equations were determined (and modified throughout the season) based on what we want in a first or second pick. For example, our first pick ability equation incorporated scoring and climbing data heavily, as we wanted a robot who could score efficiently with a reliable climb. Our second pick equation incorporated defense and driving data heavily, because our playoff match strategy relied on the third robot on our alliance as a defender. These numbers are multiplied by weight constants that can be changed at anytime.
Hanson (hli) above has done the dirty work of laying our process in detail.
I believe the ranking equations for each year are contained in the white papers. They are always modified as the season goes on and we learn more about the game. In addition, because the field depth is so much greater at Champs, we change our criteria between Regionals and Champs. This year, we had even has separate lists for our 2nd and 3rd picks, but just used our 2nd pick list for our 3rd pick because 1939 was so high on our overall list–we didn’t think they would be available at that point.