Gettin' Picky: A Guide to Alliance Selections (Part 1: The Picklist)

This is a valid point, and definitely one that could spun off into a whole thread.

The way I see it, I want to pick the team that gives us the best chances of achieving success in the playoffs. I think past success can be an indicator of the ability to achieve success, but doesn’t really say much about their ability to execute our chosen playoff strategy, and that’s what really determines whether the alliance will play well or not.

Team history comes up most often for us when talking about second picks, some of which are really hard to differentiate on data alone. It sometimes comes up from a “pedigree” angle and sometimes from a “friend picking” angle:

  • 771, our third pick on Darwin, gained some points with us because their two event wins showed that their robot could take a beating
  • 7651, our second pick at Tech Valley this year, was one of two teams we had as basically tied on our picklist. The tie was broken by our past experiences with them, having assisted them through their rookie season and playing a quals match together at the event. We knew they could follow directions, take feedback well, and be an effective team player, which made them feel like a safer option than the other team, who we had never worked with.
  • 3624, our second pick at Tech Valley 2018, was one of two teams we were considering, and part of why we chose them was our experience playing with them in 2016. We knew they’d be in it to win it and work well with us

On the first pick side, there was Central New York this year. 195’s sudden dramatic improvement the morning before alliance selection was scary and caused us to bump them from 5th to 1st on our list. Their history of consistency in playoffs made it easier for us to overlook their inconsistency during the first day, combined with talking to them about their process of ironing out the kinks in their robot. We did consider them the second best robot there (behind us) going into alliance selection, but our fear that their upward trend would continue in playoffs if we didn’t pick them really sealed the deal.

While facing 1678 is scary because of their history and their scouting/strategy cred, I think the real terrifying part is that their robots are really freaking good.

9 Likes

2791 played 14 qualification matches at the Central New York regional. Each team got 13 matches and we played a surrogate match. It was the first time in my life I was unhappy about playing too many qualification matches. Having 3 one-match turnarounds isn’t fun.

Then you’ll enjoy part 2, there’s already a whole section written up about this :slight_smile:

Great post, looking forward to the next articles.

One thing I’ll add that I like to do is to compare my list with an alternative list to see the big differences. Elo and OPR are good for this, but If you can get some other team’s list that can work too. When comparing the lists, don’t worry about the teams that are 1 or 2 positions different. The teams that are 5 or more positions apart though might merit some additional investigation. Really think through why your system might be overrating or underrating them, and likewise for the alternative system. It really helps to have a solid understanding of how Elo/opr work for this, because although they certainly have their shortcomings, they actually do have power though to “see” things that scouters overlook, and if you can take advantage of that you can improve your picklist.

4 Likes

For any Ontario teams: Many events here are held at universities/colleges, and thus you can reserve classrooms for pick list meetings. Perhaps other regions can do something similar, it’s super nice to just walk to a different part of campus, eat, and jump into some pick list discussions, and it probably helps a lot with:

This issue… Long picklist meetings aren’t good. Teams get distracted too easily, aren’t efficient enough, and sometimes go into too much depth than what should be done. 4+ hours is completely bonkers for a non-championship event. 2ish hours seems like the real sweet spot for us at least, and you definitely shouldn’t need more than that for an event of 30-40. Long picklist meetings from what I have experienced personally are very overglorified, and most definitely don’t usually create a better picklist.

Another big thing you can do is only have your scouts in the meeting. Don’t keep all of the driveteam, pit crew, and cheering people in for the meeting. If someone is getting tired, send them home/to their hotel room.

Evaluate what your priorities are and your event win conditions are before you attend that event. Review them at the start of the meeting, write them down somewhere visible.

Another big thing is rapidly tiering potential picks into groups (we use 5, Elite, Good, Average, Below average, Bad.) Spend literally 30 seconds going through each team. Don’t get side-tracked. 30 seconds ish, tier the team, move onto the next team, don’t add anyone to DNP. After you go through all the teams, it gives you two very important things:

  1. Context of what the whole field looks like performance wise
  2. Direction for each stage of your pick list of the range of robots you are looking at selecting.

Create your list by taking tier 1, sorting them till you get to a few left, merge tier 2, resort… etc

I think it’s very important to sort in order of importance what your needs and wants are. There are always major factors with making a picklist that cause points of contention where it’s unclear if x is more valuable than y, even though both teams may have all the “needs” checked off. Go deeper than just talking about what and why is important, talk about what is the MOST important first, and know what that is going into your meeting.

  • at Houston/Detriot championships.

(i’m not going to tell you that the team that has been broken and hasn’t scored anything all event is going to make a picklist)

Yous should go deeper than this. It sucks working with an alliance captain that hasn’t thought about a 3rd robot that you liked, and isn’t on their list, and they have no formed opinion of them. You don’t need to be perfect on the post 23 pick range, but having some sort of rough idea on other potential teams is very valuable when a team says:

“I like team 1234!”
and all you have to say is “Oh uhh, they aren’t on our picklist”

(or vice versa)

Highly recommend going +4-6 over the limit of teams that play on your pick list. It has saved us more times than once with picking incorrectly for our final pick. The one time we didn’t do this we got burned pretty hard for not having a good enough understanding of the team we selected.

Something else that should be highlighted more is:

Don’t look for perfection in your picklist, look for a comprehensive discussion and ranking of teams. Plans change, matches get played (at most events) after the picklist meeting. You collaborate with your alliance partners during the selection. Don’t get too hung up on teams. Make an executive decision and move on.

Another thing to think about is predicting where teams will fall when it comes down to it (both ranking and who they pick) considering what alliances you have to beat to reach your goal is a big deal.

Also a plug for the talk section I gave about picklist creation -> elims process

Very good foundation post. Keep up the good work! Lots of good info in here.

5 Likes

Our scouting meetings usually take about 1.5 hours these days. Worlds is usually more like 2.5 hours, IRI is like 30 minutes, and other offseasons are like 10 minutes. We’ve been more successful at events that are not Worlds, so maybe we should be making more snap decisions there like we do at the other events. Lol.

We’ve dropped or added teams over 1 match before. There are also usually other red flags though that ‘seal the deal’, typically that we feel we may not work well with them behind the glass.

People like to think they shouldn’t care about teams with pedigree until they lose to those teams. A lot of times these teams have more experience in high pressure playoff rounds, which increases overall playoff performance versus the performance you see in qualification rounds. It’s not something you want to ignore on your picklist, and we’ve won more competitions since we started factoring it in. Obviously data is #1, but this is more than just a tiebreaker at least for us.

Lots of good info in this thread. We’re probably a bit unusual in that we operate through ‘gut informed by data’ as opposed to ‘data informed by gut’.

8 Likes

I can vouch for just about everything the OP said. If anyone’s curious, I go into detail about 1678’s picklist process here.

5 Likes

This is an amazing guide and I love how in depth this goes putting things in writing and explaining the reasoning behind things that my time does but we always took for granted. I would just like to emphasize a few points that were brought up in the resulting discussion.

First off, I agree with what a lot of other people have said about trends being key but even past that there’s a special piece to that being a leap of faith. Is there a team that you want to gamble on as a second or third pick. Have they been chasing an issue all comp and are about to solve is propelling them into the realm of being a sleeper pick. Can there robot be cheese caked into being better than other picks. Could this leap of faith provide the x-factor that you need to beat an alliance. However, this can back fire as maybe issue they’ve been chasing is worse than previously thought. This strategy is highly dependent on the data that is collected during match scouting but also knowing teams previous competition performance as well as good pit scouting.

I also agree with a lot of what has been said in this thread about the length of scouting meetings. An hour or two is enough in my experience past that productivity takes a sharp decline. Keeping a strict schedule is a lot easier when scouting meetings are not a whole team affair in my experience. Having a carefully curated group of drive team, team leadership, and committed team members that know the teams, know the data, have played with and against these teams, talked to them in the pit, and have watched matches does a lot to limit the time wasted by the inefficiency of having a large group. Additionally having members that have spent the whole competition fixing the robot in the pit or cheering in the stands maybe don’t know the teams at the event as well as those who have been more devoted to data collection and might even pollute the discussion with opinions based off of pedigree or previous competition performance.

Another thing to be said about creating a pick list is that it opens a team wide discussion into the larger topic of why certain robots are good and what event winning strategies look like. There is something to be said about the mentality that if you do the same things that a top tier team does your team will probably see improvements. That is also why I advocate for every team to create a pick list. even if a team doesn’t meet to create a pick list doing something like making it into a game and having students submit a list of what alliances are going to look like.

This is such a useful thread. This is extremely helpful, even if you think you’ve got scouting fairly well nailed down.

I can attest to the whole data part. In 2018 at an event, 1506 was seeded #1, and we wound up choosing the #10 seeded team as first pick due to the data, and the fact that as we watched that team compete, they were incredibly consistent. The data was in their favor, even though the ranking wasn’t indicative of their performance. Quite a few teams were surprised/somewhat upset at our first pick. As an added bonus, the alliance played extremely well together, and we all complimented each other’s abilities.

Data is key, and our scouting team and mentors spend a lot of time poring over the data as the competition proceeds and adjusts as necessary. It’s absolutely critical no matter where you happen to be ranked, as you need the data to collaborate with potential alliance partners for strategy.

“Do we even need to make a picklist this event? We’re below rank 15 and there’s no way we’re getting picked in the first round.”

“Well if someone does pick us in the first round they’re really going to need our picklist.” - Sean Lavery, 2015

22 Likes

We also operated this year through ‘gut informed by data’. I feel too often that qualitative analysis of the robots takes a backseat to quantitative measures (data points). The ‘eye ball test’ is hard to deny and we’ve picked based on that alone at times. Experience with teams is also a big factor for us (see Carver selection, our first pick is 1746 ranked 44th). Knowing how our teams interact and work together can overcome a cycle or two. We paper scout teams and try to get information for three matches per team (early, mid rounds and late) although miss getting tree data points for every team sometimes. A lot of data is already out there, so this is verification/qualitative…the ‘eye ball test’ for the data available.

Quantitatively, who compliments our abilities (this year, anyone who could score) is the first measure. Seriously, we needed cargo points more than anything and that was out biggest quant factor. This data was provided by TBA.

Pit scouting gave us info about double/triple climbs. Also gave us info on teams expected ability which we could validate by watching a match.

What quant didn’t do was tell us offensive drop off when defended. This was critical for pick list and match strategy. Also, who we played with and against…who was easy and effective to work with…who do we avoid because they ave a large productivity drop when defended (we can’t really pick up the slack). First and experience did as much to craft our selections as any quant factor.

Team experience is also considered. Some teams know how to win in elims (dang 1678) and we would love to work with them. Some teams know how to win qual matches by the fistful and struggle with the game in elims. We definitely consider teams more favorably that have demonstrated they know how to win in elims.

May not be ideal for everyone, but in our brief history, we never lost in quarters as an alliance captain (5 captain position, 1 win, 2 finals, 2 semi). Finals this year, we lost to a very good 2910 team and 1678 team.

This is just our process. Everyone should take these suggestions on this tread and figure out what works and doesn’t work for them. No system fits all.

1 Like

There’s a couple more things I make sure my team discusses at the pick list meeting:

  • Are there any teams we would decline, if they invited us to join their alliance? There’s a strategic component if you’re likely to be in the top 8 and think you could form a stronger alliance on your own. But there’s also sometimes discussions around “we (usually the drivers) really seriously don’t want to be on an alliance with Team X because they’ve been deeply disrespectful/condescending/their adult drive coach screams and swears at us/etc”, and that’s something you want to get out in the open and make a decision about as a team
  • Are there any teams you want the folks in the pit to connect with our drive coach/strat lead/etc if they come asking tomorrow? If you’re one of the top teams, you may be inundated with teams wanting to talk to your decision-makers on Sunday morning, and it can be helpful to know who to tell “sure, I’ll go get them right now” vs “sorry, they’re not here, I’ll tell them you came by”
  • Conversely, are there any teams your drive coach and/or strat lead should go talk to tomorrow?
  • How much scouting are you going to do tomorrow, if any? Are there any teams you want to keep an eye on to see if they’ve fixed their issue from yesterday or are still broken/struggling?
  • Who’s going to be on the field for alliance selection? Hopefully this has been decided well in advance, but if it’s not announced at this meeting I get asked who it’s going to be about a dozen times the next morning

Not directly related to making the pick list, but they’re questions I want settled before everyone disperses on Saturday night.

3 Likes

That last point is the most important. If you are a team wanting to be successful, do not follow this guide. Learn from it and from the experiences of others, but in the end you should do what you are confident in doing and what works best for your team.

1 Like

This is a good point and I’d add to try not to make it public. If the top seeds makes it aware who they are going to pick, they just gave that advantage to the second seed.

1 Like

To start, I don’t actually know how long my meetings have lasted. They have a start time and they stop when we’re done. I would imagine 1-2 hrs is the time frame. Sometimes we have to re-watch match video to correct erroneous scouting, sometimes its a quick and dirty affair.

Here are the rules that I enforce that keep the meetings moving:

  • Curfew is waived if you’re in the meeting
  • As soon as you are not being productive, you will be asked to leave (and curfew is now enforced)
  • Scouting Captain/Strategist have final say on disagreements
  • When we hit a lull (debating semantics) or feel “done,” we’re done

If a pit person is contributing then that’s great. If they’re distracting, they get to go to bed. Same with cheering/drive team/etc. Teammates want to add value to the team and I encourage that.

I’ve done lists with 2 students in an hour, it was stressful for everyone involved and we made mistakes. I like throwing human-power at the problem, so long as its productive. Also, the meeting isn’t mandatory for anyone - its 100% opt-in.

100% agree. I don’t think anyone should take we what we wrote and regard it as law - but if you’ve never done a pick-list before, here’s a starting place. If you don’t like your process, here’s a starting place.

14 Likes

We usually try to be very upfront about who we’re picking if we’re locked in at #1, if we know for sure and they ask directly. It tends to build goodwill and trust for future competitions and provides very little competitive advantage. It’s obvious anyway at some point, since you usually see lead scouts and drive coaches from the 2 teams cluster together near the field with a whiteboard.

5 Likes

All good stuff in your post, just want to provide the counter argument for this point.

There are a few key reasons I really like advocating for limited access scouting groups for most people/teams:

  • Most teams have issues with valuing scouting. Making the scouting part of the scouting meeting invite only is a way to make it more valuable. (We still eat together before the meeting)
  • Most non-scouters don’t have much to add… simply because they weren’t watching matches. There are very few times that a non-scout added useful info in a scouting meeting. (Drive coach we have always given an exception to as an ambassador of “how was this team to work with”)
  • More people = more chances to get good info = more chances to get distracted. Having a strong person leading the meeting is not a given for most teams, let alone one that is willing or able to send individuals home/to their room if they get distracted.
  • Members of other parts of your team (most notably driveteam and pit crew) don’t get destroyed by these 4+ hour long picklist meetings that a lot of teams like to have.

If someone wants to contribute to the scouting meeting, they have to meet the threshold of being a scouter, and actually watching matches. That I think is a better way for most teams to cause good change, and get new people involved.

Ultimately a ton of teams seem pretty weak at picking, despite having strong robots, and in some cases strong scouting processes. This kind of discussion is great for the community.

2 Likes

I scouted 3 matches over 3 in season comps. I don’t think I qualify as a scouter, however I attended every meeting from start to end with decent contribution I like to think. Where’s the line? I’m a strategy person that doesn’t scout, am I welcome to the meeting in your example?

The key ingredient is watching matches. Scouters are just one of the ways to show that you have “watched matches”

2 Likes

I think there is value to having non-scouts at the meeting IF they are not distracting. Our lead technical mentor routinely brings up really helpful information of teams (“they did some pretty sketchy things with their robot” or “they don’t have sensor feedback on their shooter so they may be inconsistent”) based on his experience and time in the pits.

I generally agree that watching matches is pretty important being able to contribute for most individuals but there are exceptions.

A few smaller things I didn’t see mentioned (maybe coming in a later installment):

  1. Make sure your strategy aligns with what is at the event. For example - If you want a tank drive robot for defense with a second pick, make sure there are enough at the event for there to be someone available in the second round (or have a fall back plan).
  2. Low alliance captains (8th in particular) at regionals/district event can control which alliance station their partners work from. For example - in 2017 the two side driver stations were fairly different as far as sight lines went.
  3. As an add on to Korny’s comment - Have an idea of where your likely picks are sitting in case you need to converse between more than the reps about a second or third pick if you aren’t high enough to meet beforehand.
  4. Decide how much and what type of communication you want between your rep and the rest of the team.
2 Likes