Last year was our first time being an alliance captain, and this year is the first year we’ve been prepared for it.
I’m curious how other teams go about making their pick lists. I don’t mean who to pick so much as the process by which who to pick is determined on an event-wide scale. This year we’ve experimented and found a low-tech system that works with our 100% paper scouting system.
The night after Day 1, we make a big spreadsheet listing every team at the event, go through the paper scouting sheets we have on each team at the event and classify them by tote-feeding location (HP/LF/either) and role (stacker, capper, etc). We also put together a couple sentences describing the robot’s performance.
Based on the robot’s performance, its compatibility with our strategy, and what the drive team said after a match with them (if we had one), all the teams are rated on a 0-5 scale:
5: ideal/best
4: desirable
3: moderately desirable
2: neither desirable nor undesirable
1: undesirable
0: do not pick under any circumstance (reserved for severe reliability issues and major strategy or drive team incompatibility)
From there, within the 5s, we go past the decimal point and identify who the most desirable team is (5.9), the next most desirable team (5.8), etc.
We do this for the 4s, the 3s, etc until we have ~26 teams rated past one digit. Using the fractional ratings, we rank all the teams, and we have a preliminary pick list. We tweak it during the day as we learn more about teams and hear more from the drive team.
If you don’t want to read skip to here
While our system works, my one qualm is that it’s time consuming to prepare. I’m curious how other teams do it, especially the ones who mathematically calculate pick list rankings.
How does your team identify its best partners among 30-75 teams?
Good picklisting is always going to be time-consuming.
At Championships this year, we had the honor of helping out our friends on 5254 make a picklist for Saturday. They were involved in our scouting Alliance, but didn’t really know how to use our data to picklist.
How much you rely on quantitative data depends often on the game you’re playing. In 2014, important data to know was High goal accuracy or truss accuracy, whereas in 2015 important data to know was average number of stacks/match and standard deviation. Important qualitative data to understand is how the team scored those points: Landfill or feeder? Fender shooting or long range shooting?
Understand scarcity: in Carson this year, there were something like 12 landfill robots, and only a few that could make more than one stack in a match from there. As such, a robot that makes 2 stacks/match from the landfill was more valuable than a robot that makes 2 stacks/match from the feeder station.
Understand who the safe picks and the risk picks are: In 2013, 67 had the highest standard deviation of any team at IRI. In a good match, they could score every disc in the feeder station and hit a 50 point climb. In a bad match, they might only hit auto, a few discs, and mess up their climb. 1114 was consistently making 4 cycles in a match and hitting at least a 30 point climb. As the #1 seed, which would you rather pick? 1114, because they consistently scored well. As the #8 seed, which would you rather pick? 67, because they’re the risk pick that might perform well enough to win the event for you.
(That being said, 1114 was the first seed and 67 was the 2nd overall pick, and rightfully so. Just an example to consider.)
Considering all of these factors, make a picklist containing at least 24 robots (or 32 at the championship). This list includes every robot that is good, including your own and robots you wouldn’t be able to pick in order of how much you’d like to pair with them. This also helps you decide who you would reject. Sometimes it can be helpful to make multiple lists for different robot categories.
Whatever your list looks like- **don’t follow it blindly! ** Understand who on your list works better with the other robots you’ve selected and take input from your Alliance partners. Sometimes it can be better to take a slightly inferior pick for you because it prevents another Alliance from being able to pick that robot. This is a niche case, but it can be useful.
In 2013 at the River Rage off-season event, we had the option of selecting our second pick right before 3467 picked. The two smart picks were 190 and 2648. 190 consistently hit a 50-point climb and dump, whereas 2648 was less consistently hitting 2-3 cycles, auto, and a 10-pt hang. 190 scored more points on average, and we selected them, but if we picked 2648, we would have won. 190 and 3467 both couldn’t climb the pyramid because they both climbed inside the pyramid, so 2648 was the better pick to prevent 3467 from being able to pick them.
Lastly, at most events where Saturday morning qualification matches happen, be on the lookout for teams who change things up, or show flashes of either poor play or brilliance. Your picklist needs to change with these results.
Now obviously a lot of this data is necessary for this type of process. 20 uses paper scouting which we input into an excel database, which computes these averages for us and the teams working with us scouting and makes this process much easier.
We also like to watch match video to really understand what teams are doing, so we film all the matches.
This might seem like a large ordeal for a team that requires a lot of manpower, so 20 allies with other teams using our scouting system to provide data to everyone involved.
If you have any questions about our scouting system, feel free to PM me. I’ve always felt scouting is one of 20’s biggest strengths.
We take our quantitative and qualitative teams and organize the data into all these fancy graphs and stuff (I help make sure the kids get the data, not on the team that analyzes the numbers). Once we have the data sorted, pictures set up we meet and organize the teams into categories:
9: Nope, not gonna pick them. Lotsa reasons to go on the 9 list. At World’s all landfills went there because that was our niche. Also teams that didn’t mesh with our drive team might end up here if the conflict was deemed too much.
5: Good 3rd pick or 2nd pick if 1’s and 3’s are not available.
Good solid 2nd pick.
We’d love to have them but pretty sure they will not be available, but we rank them anyway because you never know.
We always no matter our rank go through the motions. It has come up more often than not that our data is a lot better than the Alliance captain and they look to us for help making the next pick. At World’s we skipped one and went to our next pick because our first pick worked better with them.
We usually spend an hour or two at dinner crunching the numbers and then another 2-3 going through each team. Looking at their pictures and wrapping our mind around strategies. If Team ## picks us who would be good 3rds, etc.
Step 0: SCOUT. Know who is at your event and what they can do. Know who is getting lucky, know who is a victim of murphy’s law. As the event goes on Mean Reversion will happen.
Step 1: Know what it takes to move between the different rounds. Know your capabilities, and know what you need to put up to move on.
At Greater Pittsburgh (its not remotely close to Pittsburgh) I knew it would take at least 60 points to move to Semis. About 100 to move to Finals. (Its week 2). We couldn’t carry the alliance ourselves, so we needed 1) somebody who could stack high quickly, and 2) somebody to move cans on top and noodle if needed. We could be fairly flexible in our own approach to the game.
Step 2: Since you are picking, know WHERE you are going to be picking.
This year at Pittsburgh we were sitting in 11th after Quals. Knowing the capabilities of teams ranked above us and who was ranked below us, I knew we were going to be picking in the 8th slot. That cut off everyone ranked above us. And one or two obvious picks.
Step 3: Be realistic and know what you can do. Approach the teams, talk to their strategy guys ahead of time.
Step 4: Pick the teams that best fit your strategy and will maximize your alliance’s points. If you aren’t in the top 3, this will likely be a motley crew of 3 robots that have had a tough time but will come together and put on their best matches of the event.
Step 5: Execute on the strategy. Make sure the robots work. Pray Murphy’s law doesn’t kill you.
We don’t really have a formal method of making pick lists, we’re often not in the top 8. But before you even start making your pick list, you have to assess the abilities of your own robot. What is your robot really good at? Where might you want help from an alliance partner? From there, you have to evaluate the other robots that have been performing consistently well and pick the one that you think would be the BEST fit for your robot. It’s never a good idea to just pick the best robot available at the event if you’re just going to get in each other’s way, unless you are willing to drastically change your game plan, which I wouldn’t recommend.
Example: We ended up as the 4th seed alliance captain at Chezy Champs back in August. We hadn’t been an alliance captain at all that season. We knew that our finishing capability was extremely unreliable, but we were good at inbounding and playing defense, and we were OK at trussing. Our autonomous was also having tons of issues so we needed a first pick that could consistently do a 2-ball auton and a second pick that could do a 1-ball auton. And because of the heavy amount of defense in that game, we wanted a finisher that could lob the ball into the high goal by sitting in front of the low goal. The pick was easy for us: 399. They could easily be our finisher, they had a 2-ball auton, and they were veterans. The second pick was tricky, especially considering what we wanted in our second pick and where we were in the alliance selection. We ended up picking 2928 (whom we’ve never ever played with before the event, as they were from a different state) and they were very consistent at making their auton shot and proved to be an excellent partner. And we made these picks with pretty much zero scouting data. We spent the night before in the hotel room watching the recorded live stream from qualification matches to evaluate potential partners.
Team 2338 takes scouting very seriously and in order to describe how we get to making a pick list I have to describe our scouting team. This year we divided scouting into 2 sections , quantitative and qualitative. For quantitative scouting our programming team had developed an android app called “FRC Gear Scout”. We had pre programmed every match into it and every person on the team with an android device would therefore scout a single robot per match in order to cover 100% of the data. This was to get individual stats for robots. We also have a qualitative team which works on how well a robot would compliment ours. We end up getting on average about a paragraph on the positives and negatives of each robot. This is all then compiled onto a spread sheet designed and maintained by our electrical team captain. We also have a member of the team go around and do pit scouting with general questions about hidden features of their robot and basic features of their robots as well as weight. We then get drive team and scouting team together and discuss what we want in a robot then we look through the list for first picks. Then we try to make a list of second pick about … 20 picks for the second tier list. After this we work on scenarios for other alliances we might be going against. We discuss a solution to every problem that could come up and we try our best to have a thought of every possible situation to eliminate any surprises. Also a big part of our scouting is the alumni that come back and help out. Their experience in first is very beneficial to the team.
we make multiple second-pick lists. The list we used is based on our first pick. For example, at Midwest we had lists for “landfill short stackers” and “can uprighters.” We went with a can uprighter because 1756, our first pick, tipped over the cans in auto, and we needed upright ones. In the event we had been with a bot that didn’t tip the auto cans (i.e. 16), we would have gotten a short stacker.
we plan on adding a system into our scouting which sorts teams by average points contribution. This will allow us to make our meetings more organized.
qualitative scouting can range from what a robot does in a match to how they would work with us. often times, these notes allow us to quickly assemble the DNP before the meeting even starts.
we use a combination of Excel, Access, and Tableau. Each has its own advantages.
First, we have white papers on the development of our tablet/smart phone scouting system. We use it to both develop pick lists and to push data to our drive team.
Second, we develop two pick lists, one for our first pick which is usually offensively oriented, and a second pick list for the special role needed that year which usually is defensive or counterdefensive. Our first pick usually relies on quantitative data much like the OPR. (The problem with OPR is that it is heavily influenced by the vagaries of alliance partners, but its useful as a first approximation in a pinch.)
The second pick is usually much more qualitative such as driving skill and blocking ability. We’ve found that we can use an in-match ordinal ranking to rank robots in those abilities. This holds due to the transitive property (classroom assignment: look it up.) We use regression analysis and linear programming to estimate the relative importance of each factor on the match score so that we can weight these versus offensive scoring. To the second pick list we’ll add other game specific features depending on the year’s game.
Finally, we consider the team’s organization. We almost always end up picking a team higher if we think they bring an X factor to the alliance.
Step 1: Scout. We’ve developed a shift system with four teams of six students each. Every shift has a captain, and they are responsible for their team making it to their shift on time. This way, every robot is scouted by on person every match, and we rotate the kids out to allow for rest and socializing. Data is preferably entered after every match.
Step 2: Analyze. We have a few student excel wizards that have written incredible code to crunch scouting data. It gives us information about every quantifiable aspect of the game, and allows us to cross reference with the rest of the field quickly and accurately. Note: Teaching several of your students how to use VBA and/or write macros can be VERY useful to the team. It is also a helpful life skill to have in your toolbox.
Step 3: Know what you need. Prior to looking at data for the purpose of creating a pick list, we outline various scenarios that could shake down during alliance selections. We first outline the best/ideal alliance and identify what parameters we need to look for within the data. Then, we flesh out with other scenarios and what we would need to do from there. For example, we mainly operated from the chute, but COULD do landfill effectively. Ideally, we would have 1 other very strong chute bot that could do 3-tote auton, and an ok landfill bot with a fast 2-container stealer.
Step 4: Create lists for your scenarios. Creating only one list is a little short sighted, because it limits your ability to quickly respond to changing conditions. As a result, it’s wise to make several (if possible) in excel, then employ code to black out those options as you enter teams that have been picked. This way, you can see across all your lists who is still available while still having time to deliberate with your partner for a third round pick.
Step 5: Realize it’s all about the numbers, except when it isn’t. We ask our students to write comments for every match for a reason. We interview our drive team regarding other drive team’s behavior/conduct for a reason. If a team has qualitative problems that aren’t reflected in the numbers, it needs to be addressed when ranking for a pick list.
This year was our first year with our scouting program that our programmers created, that plugins in all of our stuff during a match, for each robot by pressing on an X-Box Controller!
Please note: This data is for each robot, not the alliance. We are ranking the best robots, soley by their robot, and not the chances of having a strong schedule or weak schedule. We can easily find the robots that stand out the best. For now, we call our measure JPR.
Step 1: Scouting in the Pits
We have a pre scouting program, go around in two groups of 2 people that know how to talk to other teams, and they fill in fields, and snap a picture of their robot. This sends it into a local MYSQL database, that is imported onto our scouting computer in our pit.
Step 2: Scouting the Matches
We have developed a match scouting program, primarily my responsibility and main contribution to the project. We have a rotation schedule of about 28 people, and break it up into groups of eight members. Each rotation is about five matches, and goes fairly quickly. Six of them press buttons on X-Box controllers, connected to a Winbook, with a master operator. At the end of each match, a SQL file is generated, and is exported to a flash drive. At this point, our ‘runner’ walks it down to the pit, and plugs it in to our scouting computer.
Step 3: Importing the Data
Next, our data is imported onto the MYSQL Databse on the computer, and our Leaderboard System, a fancy program to rank teams based off of our data refreshes with our newest data, into human readable forms.
Step 4: Alliance Selections
We can print off a human friendly charts and graphs and tables, visualizations with our data for our captain. We can find the best tote bot, the best ranked team that hasn’t been picked. We are working on getting Tableau to import and visualize our data, immediatly. Tableau was included in our KoP this year.
Step 5: After Regionals
After regionals, we do some more analyzing of our data. We use the OPR’s, CCWM’s, Average Scores, DPR’s, JPR’s (our measure) and other measures to rank our teams. I hope to be releasing a cool visualization on my website for Greater Kansas City regional soon.
I will also be releasing a whitepaper + thread with my teams permission for more information about our scouting operations.
One other point: we start working on our scouting system at the beginning of build season (and even before). It’s been a very useful task for our programmers–we have more on the scouting system than on the robot. We test our system several times on Week 0 and early season competition webcasts (and even on “human” robots.) We train our scouts to use a common set of metrics (and that’s more difficult than it sounds.)
There are many other things discussed here, but also other great ideas that we’ll look at using next season. Which brings me to the best advice in the end: pay attention when meeting and talking with other successful teams. In particular we’ve benefited tremendously from working with 971, 1114 and 118, all of whom have very impressive systems.
This is the reason my team was picked 2nd pick by a high ranked alliance at worlds: my team could put up 1 6 stack from the landfill, but we were able to easily upright sideways containers, and had a canburglar. Even though there were better robots left, we fit the alliance better than they did.
When I make a pick list, I always do it with my co-lead and the scouting mentor, also getting input from our strategist. I take the data from our match scouting and find out who gets the most points. then, I look at their other qualities- auto, location used, other special abilities (good noodler, etc.), then add in a personal bias: who do I feel has been the most consistent, who’s kinda sketchy, etc… after making that list, usually with my co-lead, we show it to our mentor, who will usually point out one or two robots that we missed.
once picked or after my first pick, I consult with the other team in case we still managed to miss something, or if they know something that I don’t.
Do the scouts also track the human players? Many years, this would not have been a big issue, and this year the chute loading and placing litter into RCs could easily have been done while watching a robot. However, it seems that tracking litter tossing would have required a fourth observer for each alliance.
We monitored human players on a sliding scale: Human Player Competence, or HPC. This is a 1-5 scale that rates whether or not a human player is a benefit to the alliance, or is doing things to hurt the alliance. If they’re average, then the default rating is a 3. Ratings of 1 or 2 include wasting game pieces, causing penalties, having inconsistent aim when throwing litter, not paying attention, etc. A rating of 4 or 5 indicates great accuracy, speed, and decision making skills.
We found this was more useful than trying to track litter individually while trying to watch a robot at the same time. When giving a score other than 3, students were asked to give justification in the comments. This would tell us what was going on to warrant an abnormal rating, like stellar aim or reckless behavior.
Yep. You had exactly what we needed. And you did a great job, by the way. With your help, we were able to pass over alliances that had more offensive potential but who complemented one another less and were less reliable. One “X” factor you had: you were experienced and poised under pressure. I think of when you ran over and put a spare can on the single available tote in the remaining three seconds of the match. You squeezed every point possible out of the resources you had. That sort of mindset can make the difference between winning and losing and can get overlooked in a purely quantitative approach.
When trying to make a pick list our team has a key philosophy. If there is a team below you and you don’t want ot go up agaisnt them. Pick them. In other words, If you can’t beat them, pick them. That has helped our team speed up our making of our pick list, but only by about 20-30 minutes.
But most importantly pick teams that work well with your strategy and your drive team. There will be some teams your drive team cant work well with, so having your drive team work with your scouting team to make your pick list is a great benefit.
We try to keep feelings out of pick lists. We don’t pick teams because we’re scared of them (or because we like them), we pick teams based off of real numbers, specific comments, and match video review.
There is something to be said for picking a team because you don’t want to go against them, which is what I think halowaffle meant.
Ex: I seeded first at a 2014 event. I have the 2nd best finisher at the event. I could either pick Team A, who’s a phenomenal trusser, or Team B, who’s the best finisher at the event.
Do you pick Team A because they’re a very good trusser who would work well with you? Or do you pick Team B, so you don’t have to face the best finisher at the event?
It obviously depends on other factors (are there other good trussers/finishers, how good are you and the other finisher at trussing), but it very much makes sense to pick Team B in many scenarios.
Lastly, drive team chemistry can be a factor IMO, but usually as more of a tiebreaker than anything else. Even if your teams hate eachother, if pairing up means you’re more likely to win the event, its the pick you have to make.
With the power of RADAR GRAPHS!
So during build season one of the jobs I take up is tracking game information. From chief delphi, youtube, whatever gives me info about teams I look at all of them. I look for what teams say they want to emphasize when designing a robot as well as design trends that crop up (sure is el toro this year). This happens all the way up to competition. Just constant sifting through information about who thinks what. As soon as I feel I have a solid grip of the game I then create a Radar graph. The axes of the radar graph are the skills/traits I find are crucial for winning. The radar graphs are great because of how easy it is for make team comps with them and see how those team compositions match up against other teams. You can quickly throw together two team comps that are roughly different and see how they stack up and you aren’t limited to looking at your team, it can be used to predict what another alliance captain might aim for in a team.
The aspect I hate is the judgement calls I have when doing the scale on the axes, for example this year if I remember correctly one axes was stacking ability and the other was object manipulation ability. Figuring out how to measure those apples to oranges was an interesting experience.
