Predicting Alliance Selections

Competition Scouting
1

Hello everyone!
I’ve decided to do an investigation of alliance selections as I am planning to predict them in my event simulator.

Alliance selections are a tough system to model because there are lots of dimensions to them. Teams choose which teams to select based on (among other things):
The team’s overall robot ability
How well the team complements their own
Goal for the event (win the event, maximize district points, obtain a wildcard), these all are similar but not exactly the same
Team history/relationships
Experience/preparedness of the alliance representative on the field

The selected team then has the option to decline the invitation, which can involve (among other things):
The overall strength of the inviting team
How well the inviting team complements their own
The inviting team’s seed
The invited team’s seed
The overall strength of alternative teams the inviting team could partner with instead of the inviting team
Goal for the event (win the event, maximize district points, obtain a wildcard), these all are similar but not exactly the same
Experience/preparedness of the alliance representative on the field (sometimes even if a decline is clearly the best choice, the alliance representative does not have the nerve to go for it).

If the inviting team expects the invited team to decline, there are even higher level strategies, including:
burning the field
ordering of picks to encourage an acceptance (for example, if the 1 seed would like to partner with either the 2 or the 3 seed, but expect both will decline, it is better to invite the 3 seed first so that when they decline the 2 seed knows they cannot select the 3 seed, and the 2 seed is now more likely to accept the 1 seed’s invitation)

Basically, these are incredibly complicated systems. My goal is to create as good of a system as I can that can be applied to my event simulator every season with minimal tweaks. So, for example, in 2022 alliances may prioritize 2 traversal climbers on an ideal alliance (because 3 traversal climbs is generally unachievable and 1 traversal climb is generally leaving points on the table). But this kind of strategy will not be applicable to 2023 and beyond because traversal climbs will (probably) not exist in future years. As such, I have restricted myself to the following metrics when predicting alliance selections:
Each team’s end of qualification Elo rating
Each team’s end of qualification rank at the event

Elo ratings are nowhere near perfect, but in general they are more predictive than OPR and they can be easily used every season, so as a general “team ability” metric they are the best tool I know of. I took these two numbers and combined them to get each team’s overall “selection desirability” according to the following formula:
Selection_Desirability = End_Of_Quals_Elo - 1.0*Rank (I will discuss later how I arrived at this formula)

Now that we have a desirability value for each team, we then have to simulate the alliance selection process. Predicting the 1st Alliance captain is easy, the 1st alliance captain will always be the team with rank 1 at the end of quals. Next though, we must predict who the 1st captain will invite onto their alliance, and that’s where things become tricky. On one extreme, we could say that they will pick the most “desirable” team according to our metric 100% of the time. Obviously this is not true since our “desirability” metric is a crude estimation of each team’s alliance selection strategy, and will not match reality anywhere close to 100% of the time. On the other extreme, we could say that every team at the event is equally likely to be selected by the captain. This is also clearly absurd, as teams who rank higher and have higher Elo ratings will obviously be selected more often than teams who rank lower and have lower Elo ratings. To hit that nice middle ground where more desirable teams are selected more often but every team still has some chance of being selected, we will use a logistic function.

A properly designed logistic function is the perfect tool to predict selections. Essentially, we will take all team desirabilities and plug them into this logistic function to get the probability that they will be invited. For example, say that a team making a selection only has two options: Team A, who has a desirability of 1600, and Team B, who has a desirability of 1500. If we use the logistic function P(selection) = 1/(1+e^(-(desirability-1550)/s) (where s is a scale factor that determines how confident the predictions are) we will get a total probability of 100% that one of the teams is selected. For example, if we choose a scale factor of 50:
P(team A selected) = 1/(1+e^(-(1600-1550)/50)) = 1/(1+e^-1) = 73.1%
P(team B selected) = 1/(1+e^(-(1500-1550)/50)) = 1/(1+e^1) = 26.9%
Alternatively, we could choose a scale factor of 10:
P(team A selected) = 1/(1+e^(-(1600-1550)/10)) = 1/(1+e^-5) = 99.3%
P(team B selected) = 1/(1+e^(-(1500-1550)/10)) = 1/(1+e^5) = 00.7%

So, as you can see, the lower the scale factor, the more confident the predictions are. The same formula can be applied to multiple teams as well, the only difference is that the pivot “desirability” of 1550 will need to be adjusted depending on the teams available for selection.

I analyzed all alliance selections at regular season events from 2016 through 2019 (excluding 2016 Michigan Champs) and tweaked the desirability formula and the scale value until I found the values that provided the best predictive power according to the Brier score. The optimal value for the desirability formula was:
Selection_Desirability = End_Of_Quals_Elo - 1.0*Rank
And the optimal value of the scale factor s was 35 (which is a middle ground between the two scale values used as examples above.

I did try a few tweaks to this general formula, but none provided substantial predictive power increase, so I left them out. I tried:
Having different scale factors for the first and second round of selections
Having different scale factors for captains 1-4 vs captains 5-8
Giving a “bonus” desirability to the next highest ranked available team (this provided similar predictive power to just subtracting 1.0rank from desirability, but I opted for that instead of this bonus)
I tried a lot of different values for how much to weight rank vs Elo, but 1.0rank worked the best. I’m keeping the 1.0 there in my formula to make it obvious that this was a parameter that happened to have a value of 1, and is not just a constant in a mathematical formula.

Next, I tried to add in simulation of declines
Declines are extremely difficult to predict for a variety of reasons. One of the biggest difficulties is that declines are not recorded in the FRC API. The next best alternative is to look at @Brian_Maher’s big list of declines, for which I am immensely grateful. Unfortunately, as helpful as this document is, it is still incomplete. There are almost certainly declines out there that are unrecorded, and there is no way to say for certain that events were clear of declines even if they do not appear in the list.

Another huge difficulty in predicting declines is that they are relatively rare, with less than 300 recorded in the list between 2016 and 2019. There are generally at most 7 opportunities to decline at an event, and most events contain no declines at all. Compare that to alliance selections, of which there are 16 guaranteed per event, and you see that there are about 10-100 times more data points for selections than there are for declines.

After a long battle trying to predict declines, adding more and more parameters to the model, I eventually had to admit that I was chasing ghosts and gave up trying to predict declines. Maybe I will try again someday when they are added to the official FRC API and there are more data points available, but I have given up predicting declines for now. Fortunately, declines are still indirectly accounted for in the selection model. The scale factor of 35 would likely be a bit lower (maybe 30) if declines were directly accounted for, meaning that the predictions are a little bit less confident than they might otherwise be, and that is likely partially due to declines.

With all of that together, I then made an alliance selection simulator that went through each event, simulating alliance selections 200 times and aggregating the predictions. Here is the full workbook for the curious.

9 Likes
2

Here is how well my model did at predicting alliance selections at each event, events with lower Brier scores were predicted more accurately than events with higher Brier scores:

2016-2019 Alliance Selection Prediction Accuracies
Year Name Brier
2016 Silicon Valley Regional presented by Google.org 14.7
2018 Archimedes Division 14.8
2018 FIM District East Kentwood Event 14.9
2018 MAR District Bridgewater-Raritan Event 15.2
2019 Seven Rivers Regional 15.3
2018 NE District Greater Boston Event 15.4
2018 PNW District West Valley Event 15.5
2019 Northern Lights Regional 15.5
2018 Roebling Division 15.5
2018 FIM District Kettering University Event #2 15.5
2018 MAR District Seneca Event 15.6
2018 FIRST Israel District Championship 15.6
2017 ONT District - Ryerson University Event 15.7
2016 Newton Division 15.7
2019 FIM District Southfield Event 15.8
2018 MAR District Hatboro-Horsham Event 15.8
2016 FIM District - Ann Arbor Skyline Event 16.0
2016 Rocket City Regional 16.1
2018 Southern Cross Regional 16.1
2019 FIM District Alpena Event #2 16.1
2017 New England District Championship 16.2
2018 ONT District North Bay Event 16.3
2018 Curie Division 16.3
2019 Central Valley Regional 16.3
2018 Central New York Regional 16.3
2016 NE District - UNH Event 16.4
2017 ONT District - Western University, Engineering Event 16.4
2018 FIRST North Carolina State Championship 16.4
2016 Los Angeles Regional 16.4
2016 Hub City Regional 16.4
2018 Seven Rivers Regional 16.4
2019 FIM District Traverse City Event 16.5
2016 Galileo Division 16.5
2017 ONT District - Windsor Essex Great Lakes Event 16.5
2016 Windsor Essex Great Lakes Regional 16.5
2017 Arkansas Rock City Regional 16.5
2016 Central Valley Regional 16.5
2019 Hawaii Regional 16.5
2019 ONT District University of Waterloo Event 16.5
2016 MAR District - Montgomery Event 16.6
2016 Israel Regional 16.6
2018 FIM District Lansing Event 16.6
2017 Alamo Regional 16.6
2018 Colorado Regional 16.6
2018 Minnesota North Star Regional 16.7
2018 Central Valley Regional 16.7
2018 Wisconsin Regional 16.7
2018 El Paso Regional 16.7
2016 Tesla Division 16.7
2016 Arizona North Regional 16.7
2016 Midwest Regional 16.7
2016 Lake Superior Regional 16.7
2016 Greater Toronto East Regional 16.7
2016 NE District - Rhode Island Event 16.7
2019 FIM District Kettering University Event #1 16.7
2018 Aerospace Valley Regional 16.8
2019 ONT District McMaster University Event 16.8
2018 PNW District Clackamas Academy Event 16.8
2017 Lake Superior Regional 16.8
2016 MAR District - Bridgewater-Raritan Event 16.8
2019 Smoky Mountains Regional 16.8
2017 Finger Lakes Regional 16.9
2019 NE District Pine Tree Event 16.9
2016 IN District - Perry Meridian Event 16.9
2019 Pacific Northwest FIRST District Championship 16.9
2019 Newton Division 16.9
2019 FIRST Ontario Provincial Championship - Science Division 16.9
2018 San Francisco Regional 16.9
2017 South Pacific Regional 17.0
2017 NE District - SE Mass Event 17.0
2016 Indiana State Championship 17.0
2017 Newton Division 17.0
2016 Oklahoma Regional 17.0
2019 FIM District West Michigan Event 17.0
2017 NC District - Greensboro Event 17.0
2017 Orlando Regional 17.0
2019 Los Angeles Regional 17.0
2019 Regional Monterrey 17.1
2017 ONT District - University of Waterloo Event 17.1
2018 FIM District Center Line Event 17.1
2018 NE District UNH Event 17.1
2018 ONT District Durham College Event 17.1
2018 Carson Division 17.1
2017 Daly Division 17.1
2019 IN District Tippecanoe Event 17.1
2019 FIRST Ontario Provincial Championship - Technology Division 17.1
2016 PCH District - Kennesaw Event 17.1
2018 Darwin Division 17.2
2018 FIM District Lakeview Event 17.2
2018 Festival de Robotique - Montreal Regional 17.2
2017 Lone Star North Regional 17.2
2018 ONT District University of Waterloo Event 17.2
2019 New York City Regional 17.2
2019 Bayou Regional 17.2
2019 FIM District Center Line Event 17.2
2016 Archimedes Division 17.2
2019 FMA District Hatboro-Horsham Event 17.2
2018 Canadian Rockies Regional 17.2
2016 Northern Lights Regional 17.2
2016 PNW District - Mount Vernon Event 17.3
2017 Western Canada Regional 17.3
2018 ONT District McMaster University Event 17.3
2018 Turing Division 17.3
2018 CHS District Hampton Roads Event sponsored by Newport News Shipbuilding 17.3
2019 Michigan State Championship - Consumers Energy Division 17.3
2018 PNW District Lake Oswego Event 17.3
2018 FIM District Gaylord Event 17.4
2019 ONT District Durham College Event 17.4
2018 CHS District Southwest Virginia Event sponsored by Anton Paar 17.4
2016 IN District - Walker Warren Event 17.4
2018 Peachtree District State Championship 17.4
2016 Waterloo Regional 17.4
2017 ONT District - McMaster University Event 17.4
2018 Central Illinois Regional 17.4
2019 Orange County Regional 17.4
2017 Northern Lights Regional 17.4
2017 FIM District - Howell Event 17.4
2017 Peachtree State Championship 17.4
2017 Festival de Robotique - Montreal Regional 17.4
2019 NE District UNH Event 17.4
2016 Orange County Regional 17.4
2016 FIM District - Kettering University Event #1 17.5
2018 New England District Championship 17.5
2017 MAR District - Springside Chestnut Hill Academy Event 17.5
2019 Aerospace Valley Regional 17.5
2019 PNW District West Valley Event 17.5
2017 Hudson Valley Regional 17.5
2018 Michigan State Championship - Consumers Energy Division 17.5
2017 PNW District - Auburn Event 17.5
2018 Iowa Regional 17.5
2017 PNW District - Glacier Peak Event 17.5
2016 Carson Division 17.6
2018 Lone Star Central Regional 17.6
2019 Great Northern Regional 17.6
2016 FIM District - Howell Event 17.6
2019 Iowa Regional 17.6
2016 NE District - Granite State Event 17.6
2018 PCH District Duluth Event 17.6
2019 San Francisco Regional 17.7
2018 MAR District Montgomery Event 17.7
2016 MAR District - Westtown Event 17.7
2019 Carson Division 17.7
2019 FIT District Channelview Event 17.7
2016 Hawaii Regional 17.7
2019 FIM District Detroit Event 17.7
2016 NC District - UNC Asheville Event 17.7
2019 NE District Southern NH Event 17.7
2017 PNW District - Central Washington University Event 17.7
2018 Greater Kansas City Regional 17.7
2018 FIM District Troy Event 17.8
2016 Western Canada Regional 17.8
2019 Arizona North Regional 17.8
2016 FIM District - East Kentwood Event 17.8
2017 ONT District - North Bay Event 17.8
2019 FMA District Seneca Event 17.8
2016 CHS District - Central Virginia Event 17.8
2017 Laguna Regional 17.8
2018 NE District Southern CT Event 17.8
2018 ONT District Windsor Essex Great Lakes Event 17.8
2018 NE District Granite State Event 17.8
2017 South Florida Regional 17.8
2019 FMA District Bensalem Event 17.8
2018 NE District Rhode Island Event 17.9
2017 NE District - Granite State Event 17.9
2019 FMA District Mount Olive Event 17.9
2018 Michigan State Championship - Dow Division 17.9
2018 NC District Forsyth County Event 17.9
2018 FIRST Chesapeake District Championship 17.9
2018 Pacific Northwest District Championship 17.9
2019 FMA District Springside Chestnut Hill Academy Event 17.9
2016 FIM District - Waterford Event 17.9
2017 FIM District - Lakeview Event 17.9
2017 FIM District - West Michigan Event 17.9
2018 Smoky Mountains Regional 17.9
2016 Minnesota North Star Regional 17.9
2018 Sacramento Regional 18.0
2017 FIM District - Waterford Event 18.0
2018 ONT District Georgian College Event 18.0
2017 Midwest Regional 18.0
2017 ONT District - Durham College Event 18.0
2018 FIM District Southfield Event 18.0
2019 Idaho Regional 18.0
2017 Orange County Regional 18.0
2019 Curie Division 18.0
2017 MAR District - Montgomery Event 18.0
2019 South Florida Regional 18.0
2018 Los Angeles Regional 18.0
2016 Bayou Regional 18.0
2019 FNC District Guilford County Event 18.0
2019 Bosphorus Regional 18.0
2016 Smoky Mountains Regional 18.0
2019 Michigan State Championship - Ford Division 18.0
2017 Sacramento Regional 18.0
2018 FIRST Mid-Atlantic District Championship 18.0
2019 FIM District Lansing Event 18.0
2017 FIM District - Woodhaven Event 18.1
2019 FIM District East Kentwood Event 18.1
2017 San Francisco Regional 18.1
2019 FNC District UNC Asheville Event 18.1
2018 Utah Regional 18.1
2018 Arkansas Rock City Regional 18.1
2017 ONT District - Victoria Park Collegiate Event 18.1
2019 ONT District York University Event 18.1
2019 Las Vegas Regional 18.1
2017 PCH District - Albany Event 18.1
2017 MAR District - Mount Olive Event 18.1
2019 Regional Laguna 18.1
2019 Michigan State Championship - Dow Division 18.1
2019 NE District Western NE Event 18.1
2017 PNW District - Clackamas Academy of Industrial Science Event 18.1
2018 CHS District Northern Virginia Event 18.1
2019 FIRST Israel District Championship 18.1
2017 CHS District - Hampton Roads Event sponsored by Newport News Shipbuilding 18.1
2017 Buckeye Regional 18.1
2016 FIM District - Marysville Event 18.1
2019 PCH District Dalton Event 18.2
2019 Heartland Regional 18.2
2016 Sacramento Regional 18.2
2017 PNW District - Lake Oswego Event 18.2
2018 IN District St. Joseph Event 18.2
2016 Las Vegas Regional 18.2
2018 Shanghai Regional 18.2
2016 Carver Division 18.2
2018 FIRST Ontario Provincial Championship - Science Division 18.2
2019 PNW District Mount Vernon Event 18.2
2016 FIM District - Troy Event 18.3
2017 Iowa Regional 18.3
2016 San Diego Regional 18.3
2019 ISR District Event #1 18.3
2016 NE District - UMass-Dartmouth Event 18.3
2018 FIM District Livonia Event 18.3
2016 CHS District - Southwest Virginia Event 18.3
2018 SBPLI Long Island Regional #2 18.3
2017 CHS District - Central Virginia Event 18.3
2019 PNW District Auburn Event 18.3
2016 MAR District - Hatboro-Horsham Event 18.3
2019 FIM District Marysville Event 18.3
2018 PNW District SunDome Event 18.3
2017 NE District - Southern NH Event 18.3
2018 Galileo Division 18.3
2016 SBPLI Long Island Regional 18.3
2018 NE District Worcester Polytechnic Institute Event 18.3
2018 FIM District West Michigan Event 18.4
2017 Galileo Division 18.4
2016 FIRST Chesapeake District Championship sponsored by Booz Allen Hamilton 18.4
2018 Midwest Regional 18.4
2017 Brazos Valley Regional 18.4
2019 FIRST Chesapeake District Championship 18.4
2019 New York Tech Valley Regional 18.4
2019 Festival de Robotique a Quebec City Regional 18.4
2019 Peachtree District State Championship 18.4
2018 FIRST Ontario Provincial Championship - Technology Division 18.4
2019 NE District North Shore Event 18.4
2018 ONT District Ryerson University Event 18.4
2017 CHS District - Greater DC Event sponsored by Accenture 18.4
2018 FIM District Forest Hills Event 18.4
2019 Galileo Division 18.4
2019 PNW District SunDome Event 18.4
2016 MAR District - Springside Chestnut Hill Event 18.5
2018 NC District UNC Asheville Event 18.5
2016 PNW District - Glacier Peak Event 18.5
2017 Silicon Valley Regional 18.5
2016 NE District - WPI Event 18.5
2019 FIM District Lake Superior State University Event 18.5
2019 FIRST Mid-Atlantic District Championship 18.5
2018 CHS District Greater DC Event co-sponsored by Micron 18.5
2017 Dallas Regional 18.6
2016 NC District - Campbell University/Johnston Community College Event 18.6
2016 New York City Regional 18.6
2019 Festival de Robotique a Montreal Regional 18.6
2018 Finger Lakes Regional 18.6
2017 Arizona North Regional 18.6
2019 ISR District Event #3 18.6
2017 CHS District - Northern Virginia Event sponsored by Bechtel 18.6
2018 ISR District Event #4 18.6
2018 FIM District Traverse City Event 18.6
2018 San Diego Regional presented by Qualcomm 18.6
2018 NC District Pitt County Event 18.6
2016 Greater Pittsburgh Regional 18.6
2016 FIM District - Traverse City Event 18.6
2017 FIM District - Traverse City Event 18.6
2019 FMA District Westtown Event 18.6
2018 NC District UNC Pembroke Event 18.6
2018 Dallas Regional 18.6
2017 FIM District - Escanaba Event 18.7
2018 PCH District Columbus Event 18.7
2018 Arizona North Regional 18.7
2018 PCH District Dalton Event 18.7
2019 Arkansas Rock City Regional 18.7
2016 Dallas Regional 18.7
2018 PNW District Auburn Mountainview Event 18.7
2016 PCH District - Dalton Event 18.7
2016 PNW District - Clackamas Academy of Industrial Science Event 18.7
2018 Great Northern Regional 18.7
2016 FIM District - Lake Superior State University Event 18.7
2019 NE District Hartford Event 18.7
2019 FIT District Dallas Event 18.7
2018 SBPLI Long Island Regional #1 18.7
2019 NE District SE Mass Event 18.7
2018 FIM District Belleville Event 18.7
2016 Colorado Regional 18.7
2017 Ventura Regional 18.8
2018 Hopper Division 18.8
2019 Hudson Valley Regional 18.8
2017 NE District - North Shore Event 18.8
2017 MAR District - Westtown Event 18.8
2019 Hopper Division 18.8
2017 Tesla Division 18.8
2019 Carver Division 18.8
2019 Buckeye Regional 18.8
2018 Shenzhen Regional 18.8
2018 Greater Pittsburgh Regional 18.8
2016 PNW District - Wilsonville Event 18.8
2017 Las Vegas Regional 18.8
2017 IN District - Perry Meridian Event 18.8
2016 Mid-Atlantic Robotics District Championship 18.8
2019 Daly Division 18.9
2017 Arizona West Regional 18.9
2019 ISR District Event #2 18.9
2018 PCH District Gainesville Event 18.9
2019 FIRST In Texas District Championship 18.9
2019 PNW District Auburn Mountainview Event 18.9
2016 PCH District - Albany Event 18.9
2017 FIRST Chesapeake District Championship sponsored by Booz Allen Hamilton 18.9
2016 Curie Division 18.9
2018 Hawaii Regional 18.9
2017 Hawaii Regional 18.9
2016 Buckeye Regional 19.0
2018 ISR District Event #1 19.0
2017 Southern Cross Regional 19.0
2016 FIM District - West Michigan Event 19.0
2016 Lone Star Regional 19.0
2016 NC District - Guilford County Event 19.0
2019 CHS District Owings Mills MD Event sponsored by Leidos 19.0
2019 South Pacific Regional 19.0
2019 FIM District Livonia Event 19.0
2016 FIM District - Center Line Event 19.0
2016 FIM District - Livonia Event 19.0
2017 Bayou Regional 19.0
2016 FIM District - Woodhaven Event 19.0
2019 Finger Lakes Regional 19.0
2018 NE District North Shore Event 19.0
2019 Turing Division 19.0
2016 Central Illinois Regional 19.1
2017 NE District - Waterbury Event 19.1
2016 NE District - Waterbury Event 19.1
2017 MAR District - Bridgewater-Raritan Event 19.1
2019 Sacramento Regional 19.1
2019 FMA District Montgomery Event 19.1
2016 NC FIRST Robotics State Championship 19.1
2019 FIM District Shepherd Event 19.1
2019 SBPLI Long Island Regional #1 19.1
2018 Buckeye Regional 19.1
2018 CHS District Central Virginia Event sponsored by Dominion Energy 19.1
2017 NC District - Pitt County Event 19.1
2016 Orlando Regional 19.1
2019 Arizona West Regional 19.1
2018 CHS District Southern Maryland Event 19.1
2016 FIM District - Lakeview Event 19.1
2018 Ventura Regional 19.2
2016 North Bay Regional 19.2
2019 Utah Regional 19.2
2017 ISR District Event #2 19.2
2019 NE District Greater Boston Event 19.2
2017 Darwin Division 19.2
2017 Wisconsin Regional 19.2
2017 Colorado Regional 19.2
2018 FIM District Escanaba Event 19.2
2019 FIM District Belleville Event 19.2
2016 Queen City Regional 19.2
2016 New York Tech Valley Regional 19.2
2016 NE District - Hartford Event 19.2
2019 FIM District Gibraltar Event 19.2
2017 Miami Valley Regional 19.2
2017 IN District - St. Joseph Event 19.2
2017 FIRST Mid-Atlantic District Championship sponsored by Johnson & Johnson 19.2
2019 CHS District Haymarket VA Event sponsored by Booz Allen Hamilton 19.2
2018 Las Vegas Regional 19.2
2019 FIM District Lincoln Event 19.2
2017 CHS District - Southwest Virginia Event 19.3
2018 ONT District York University Event 19.3
2019 FIT District Austin Event 19.3
2017 Michigan State Championship - Ford Division 19.3
2016 CHS District - Northern Virginia Event 19.3
2016 Idaho Regional 19.3
2017 Michigan State Championship - DTE Energy Foundation Division 19.3
2017 ISR District Event #3 19.3
2019 FIT District Del Rio Event 19.3
2019 Oklahoma Regional 19.3
2017 FIM District - Center Line Event 19.3
2019 Central Illinois Regional 19.3
2018 FIM District Midland Event 19.3
2016 New England District Championship 19.3
2019 Wisconsin Regional 19.3
2016 Utah Regional 19.3
2016 NC District - Wake County Event 19.3
2019 Minnesota 10,000 Lakes Regional presented by the Medtronic Foundation 19.3
2019 Los Angeles North Regional 19.3
2019 FNC District UNC Pembroke Event 19.3
2018 FIM District Waterford Event 19.4
2019 Southern Cross Regional 19.4
2017 Hopper Division 19.4
2018 PNW District Glacier Peak Event 19.4
2019 FIM District Midland Event 19.4
2018 Lake Superior Regional 19.4
2019 SBPL2 Long Island Regional #2 19.4
2018 Idaho Regional 19.4
2018 ISR District Event #2 19.4
2018 PNW District Mount Vernon Event 19.4
2017 FIM District - Southfield Event 19.5
2017 New York City Regional 19.5
2018 New York City Regional 19.5
2016 South Florida Regional 19.5
2018 Monterrey Regional 19.5
2018 Heartland Regional 19.5
2017 FIM District - St. Joseph Event 19.5
2017 Lone Star Central Regional 19.5
2019 ONT District Western University, Western Engineering Event 19.5
2019 PNW District Clackamas Academy Event 19.5
2018 Indiana State Championship 19.5
2017 Archimedes Division 19.5
2019 CHS District Richmond VA Event sponsored by Dominion Energy 19.5
2019 St. Louis Regional 19.5
2019 Lake Superior Regional 19.5
2017 Oklahoma Regional 19.5
2018 South Pacific Regional 19.5
2017 NE District - Pine Tree Event 19.5
2019 FIM District Gull Lake Event 19.5
2019 ONT District North Bay Event 19.5
2019 ONT District Humber College Event 19.6
2018 ONT District Western University, Western Engineering Event 19.6
2017 ISR District Event #4 19.6
2019 Palmetto Regional 19.6
2018 MAR District Westtown Event 19.6
2017 PCH District - Gainesville Event 19.6
2017 Seven Rivers Regional 19.6
2019 New England District Championship 19.6
2019 PCH District Albany Event presented by Procter & Gamble 19.6
2017 Shenzhen Regional 19.6
2017 Greater Pittsburgh Regional 19.6
2017 Turing Division 19.6
2017 ONT District - Georgian College Event 19.6
2017 PNW District - Auburn Mountainview Event 19.7
2019 PNW District Lake Oswego Event 19.7
2017 CHS District - Central Maryland Event sponsored by Leidos 19.7
2017 IN District - Tippecanoe Event 19.7
2016 CHS District - Hampton Roads Event 19.7
2016 PCH District - Columbus Event 19.7
2019 Greater Pittsburgh Regional 19.7
2019 Rocket City Regional 19.7
2019 Canadian Rockies Regional 19.7
2018 Carver Division 19.7
2016 Iowa Regional 19.7
2018 Michigan State Championship - DTE Energy Foundation Division 19.7
2016 NE District - Boston Event 19.7
2018 NE District Pine Tree Event 19.7
2016 Arizona West Regional 19.7
2017 CHS District - Northern Maryland Event 19.8
2017 Greater Kansas City Regional 19.8
2017 Carson Division 19.8
2018 South Florida Regional 19.8
2019 FIT District Pasadena Event 19.8
2019 CHS District Portsmouth VA Event sponsored by Newport News Shipbuilding 19.8
2019 FIM District Kingsford Event 19.8
2018 Alamo Regional 19.8
2018 FIM District Gibraltar Event 19.8
2017 New York Tech Valley Regional 19.9
2019 FIM District Forest Hills Event 19.9
2016 NE District - Pine Tree Event 19.9
2016 CHS District - Greater DC Event 19.9
2017 PCH District - Columbus Event 19.9
2019 IN District Center Grove Event sponsored by Toyota 19.9
2018 FIM District St. Joseph Event 19.9
2017 FIRST Ontario Provincial Championship 19.9
2017 FIM District - Livonia Event 19.9
2019 CHS District Oxon Hill MD Event 19.9
2018 Laguna Regional 19.9
2018 FIM District Lake Superior State University Event 19.9
2018 FIM District Marysville Event 19.9
2018 Northern Lights Regional 20.0
2019 ONT District Georgian College Event 20.0
2017 Carver Division 20.0
2016 FIM District - Midland Event 20.0
2016 Palmetto Regional 20.0
2016 PNW District - Auburn Event 20.0
2017 FIM District - Lake Superior State University Event 20.0
2019 PCH District Columbus Event 20.0
2017 Roebling Division 20.0
2017 NE District - Hartford Event 20.0
2017 Michigan State Championship - Consumers Energy Division 20.0
2019 Indiana State Championship 20.0
2019 Colorado Regional 20.0
2018 St. Louis Regional 20.1
2017 FIM District - Troy Event 20.1
2016 Finger Lakes Regional 20.1
2019 NE District Rhode Island Event 20.1
2017 ISR District Event #1 20.1
2016 Minnesota 10000 Lakes Regional 20.1
2016 St. Louis Regional 20.1
2019 NE District Waterbury Event 20.1
2019 Del Mar Regional presented by Qualcomm 20.1
2018 FIM District Alpena Event 20.1
2016 Wisconsin Regional 20.2
2017 Smoky Mountains Regional 20.2
2019 FIT District Greenville Event 20.2
2016 MAR District - Seneca Event 20.2
2019 Central New York Regional 20.2
2019 IN District St. Joseph Event 20.2
2017 FIM District - East Kentwood Event 20.2
2019 FIM District Lakeview Event 20.3
2018 Hub City Regional 20.3
2019 Tesla Division 20.3
2018 Michigan State Championship - Ford Division 20.3
2019 San Diego Regional presented by Qualcomm 20.3
2017 Central Illinois Regional 20.3
2019 Regional de la Ciudad de Mexico 20.3
2016 Hopper Division 20.3
2017 FIM District - Marysville Event 20.3
2019 Midwest Regional 20.3
2017 NE District - Rhode Island Event 20.3
2018 PNW District Wilsonville Event 20.3
2019 PCH District Gainesville Event 20.3
2019 FIM District St. Joseph Event 20.3
2016 Peachtree District State Championship 20.4
2019 FIT District Amarillo Event 20.4
2017 Los Angeles Regional 20.4
2016 Greater Toronto Central Regional 20.4
2018 IN District Plainfield Event sponsored by Toyota 20.4
2018 Hudson Valley Regional 20.4
2017 FIM District - Midland Event 20.4
2016 PNW District - Auburn Mountainview Event 20.4
2017 FIM District - Kettering University Event #1 20.4
2019 NE District Granite State Event 20.5
2019 FNC District Wake County Event 20.5
2019 Minnesota North Star Regional 20.5
2018 FIM District Milford Event 20.5
2019 FIM District Kettering University Event #2 20.5
2016 Pacific Northwest District Championship sponsored by Autodesk 20.6
2017 Michigan State Championship - Dow Division 20.6
2016 FIM District - Lansing Event 20.6
2017 Toluca Regional 20.6
2017 FIM District - Gull Lake Event 20.7
2019 FMA District Bridgewater-Raritan Event 20.7
2019 PNW District Wilsonville Event 20.7
2016 Ventura Regional 20.7
2019 ISR District Event #4 20.7
2018 Istanbul Regional 20.7
2018 Rocket City Regional 20.7
2019 Orlando Regional 20.7
2016 CHS District - Northern Maryland Event 20.8
2017 San Diego Regional presented by Qualcomm 20.8
2016 PNW District - Philomath Event 20.8
2019 ONT District Ryerson University Event 20.8
2016 Mexico City Regional 20.8
2017 FIM District - Lansing Event 20.8
2016 CHS District - Central Maryland Event 20.8
2019 FIRST North Carolina State Championship 20.8
2018 NE District Waterbury Event 20.9
2017 FIM District - Gaylord Event 20.9
2018 Miami Valley Regional 20.9
2017 PNW District - Wilsonville Event 20.9
2018 FIM District Shepherd Event 20.9
2018 PCH District Albany Event 20.9
2016 PNW District - West Valley Event 20.9
2019 FIM District Jackson Event 20.9
2018 Palmetto Regional 20.9
2017 MAR District - Seneca Event 21.0
2019 FIT District El Paso Event 21.0
2017 Curie Division 21.0
2019 Monterey Bay Regional 21.0
2018 Orlando Regional 21.0
2017 Pacific Northwest District Championship 21.0
2019 FIT District San Antonio Event 21.0
2019 FIM District Alpena Event #1 21.1
2018 CHS District Central Maryland Event sponsored by Leidos 21.1
2017 FIM District - Shepherd Event 21.1
2018 FIM District Lincoln Event 21.1
2018 MAR District Mount Olive Event 21.1
2019 PNW District Glacier Peak Event 21.2
2019 Archimedes Division 21.2
2018 Lone Star South Regional 21.2
2018 NE District SE Mass Event 21.2
2019 FIM District Troy Event 21.3
2017 FIM District - Ann Arbor Pioneer Event 21.3
2017 Minnesota North Star Regional 21.3
2019 FIT District Plano Event 21.3
2017 PNW District - Mount Vernon Event 21.3
2019 Michigan State Championship - DTE Energy Foundation Division 21.3
2018 MAR District Springside Chestnut Hill Academy Event 21.3
2017 NC District - UNC Asheville Event 21.4
2018 FIM District Gull Lake Event 21.4
2019 Istanbul Regional 21.4
2018 Oklahoma Regional 21.4
2016 Alamo Regional sponsored by Rackspace Hosting 21.4
2018 Arizona West Regional 21.4
2017 Rocket City Regional 21.5
2016 FIM District - Standish-Sterling Event 21.5
2019 FIM District Muskegon Event 21.5
2016 FRC Festival de Robotique - Montreal Regional 21.5
2018 PNW District Auburn Event 21.6
2019 PCH District Forsyth District Event 21.7
2017 Idaho Regional 21.7
2019 Central Missouri Regional 21.8
2016 MAR District - Mt. Olive Event 21.8
2019 Canadian Pacific Regional 21.8
2018 NE District Hartford Event 21.9
2016 NE District - North Shore Event 21.9
2017 MAR District - Hatboro-Horsham Event 21.9
2018 Bayou Regional 21.9
2019 CHS District Bethesda MD Event sponsored by Bechtel 22.0
2018 New York Tech Valley Regional 22.0
2019 NE District Central Mass Event 22.0
2019 FIM District Milford Event 22.0
2018 FIM District Kettering University Event #1 22.1
2017 PNW District - West Valley Event 22.1
2016 Australia Regional 22.1
2017 Palmetto Regional 22.1
2016 PNW District - Central Washington University Event 22.2
2019 ONT District Windsor Essex Great Lakes Event 22.3
2018 IN District Tippecanoe Event 22.3
2019 Greater Kansas City Regional 22.3
2017 Minnesota 10000 Lakes Regional 22.3
2016 FIM District - Escanaba Event 22.4
2017 FIM District - Kettering University Event #2 22.4
2017 NC District - Raleigh Event 22.6
2018 Canadian Pacific Regional 22.7
2016 Greater Kansas City Regional 23.0
2017 Indiana State Championship 23.0
2017 SBPLI Long Island Regional 23.0
2019 Darwin Division 23.0
2019 Ventura Regional 23.0
2016 FIM District - St. Joseph Event 23.1
2018 ISR District Event #3 23.2
2016 FIM District - Southfield Event 23.2
2016 FIM District - Kettering University Event #2 23.3
2016 IN District - Tippecanoe Event 23.4
2018 Tesla Division 23.4
2017 NE District - Greater Boston Event 23.4
2018 Newton Division 23.5
2017 PCH District - Dalton Event 23.5
2017 FIRST Israel District Championship 23.6
2017 Central Valley Regional 23.7
2017 Hub City Regional 23.8
2016 Arkansas Rock City Regional 24.0
2019 Roebling Division 24.0
2019 CHS District Blacksburg VA Event 24.1
2018 Daly Division 24.1
2017 FIRST North Carolina State Championship 24.2
2017 St. Louis Regional 24.3
2019 Miami Valley Regional 24.3
2017 NE District - Worcester Polytechnic Institute Event 24.4
2018 Orange County Regional 24.5
2018 Medtronic Foundation Regional 24.8
2018 Silicon Valley Regional 25.4
2017 Utah Regional 25.8
2019 Silicon Valley Regional 26.4
1 Like
3

Let’s take a look at the predictions for the 2016 Silicon Valley regional, the event for which this model was the most accurate according to the Brier score:

2016 Silicon Valley Alliance Selection Predictions
Team Rank Final Alliance Alliance Position (1=captain) desirability Alliance 1 Captain Alliance 1 First Selection Alliance 2 Captain Alliance 2 First Selection Alliance 3 Captain Alliance 3 First Selection Alliance 4 Captain Alliance 4 First Selection Alliance 5 Captain Alliance 5 First Selection Alliance 6 Captain Alliance 6 First Selection Alliance 7 Captain Alliance 7 First Selection Alliance 8 Captain Alliance 8 First Selection Alliance 8 Second Selection Alliance 7 Second Selection Alliance 6 Second Selection Alliance 5 Second Selection Alliance 4 Second Selection Alliance 3 Second Selection Alliance 2 Second Selection Alliance 1 Second Selection Team Brier
1678 1 Alliance 1 1 1903 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
254 2 Alliance 1 2 1991 0% 73% 27% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.15
971 3 Alliance 2 1 1916 0% 27% 73% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.15
368 4 Alliance 2 2 1710 0% 0% 0% 73% 27% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.15
1280 5 Alliance 3 1 1545 0% 0% 0% 2% 73% 2% 24% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.13
2489 6 Alliance 4 1 1524 0% 0% 0% 1% 0% 4% 74% 0% 22% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.12
5924 7 Alliance 5 1 1521 0% 0% 0% 0% 0% 2% 3% 3% 73% 1% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.11
5700 8 Alliance 6 1 1451 0% 0% 0% 0% 0% 1% 0% 2% 6% 2% 71% 0% 20% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.13
6039 9 Alliance 7 1 1489 0% 0% 0% 1% 0% 1% 0% 2% 0% 2% 10% 2% 66% 1% 17% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.15
852 10 Alliance 8 1 1485 0% 0% 0% 1% 0% 2% 0% 1% 0% 1% 0% 1% 15% 1% 64% 1% 0% 1% 1% 0% 0% 1% 0% 1% 0.15
256 11 1490 0% 0% 0% 0% 0% 2% 0% 2% 0% 3% 0% 4% 0% 1% 18% 3% 4% 2% 2% 1% 1% 4% 3% 3% 0.04
649 12 Alliance 5 2 1565 0% 0% 0% 2% 0% 11% 0% 12% 0% 11% 0% 9% 0% 10% 1% 5% 8% 7% 5% 2% 4% 2% 3% 2% 0.85
4904 13 Alliance 6 3 1469 0% 0% 0% 1% 0% 1% 0% 1% 0% 0% 0% 1% 0% 1% 0% 1% 3% 2% 1% 3% 1% 3% 4% 4% 0.99
4255 14 1447 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 1% 0% 1% 0% 0% 1% 2% 1% 2% 3% 1% 2% 2% 0.00
5940 15 Alliance 4 3 1453 0% 0% 0% 0% 0% 0% 0% 1% 0% 2% 0% 0% 0% 0% 0% 1% 2% 3% 1% 1% 1% 1% 1% 2% 0.98
114 16 Alliance 3 2 1522 0% 0% 0% 1% 0% 4% 0% 5% 0% 3% 0% 7% 0% 5% 0% 9% 2% 3% 4% 4% 5% 5% 6% 5% 0.96
8 17 Alliance 7 2 1487 0% 0% 0% 1% 0% 1% 0% 2% 0% 3% 0% 3% 0% 3% 0% 2% 4% 2% 2% 4% 4% 4% 4% 4% 0.96
2643 18 1437 0% 0% 0% 1% 0% 1% 0% 2% 0% 2% 0% 0% 0% 0% 0% 2% 1% 1% 0% 1% 1% 2% 1% 0% 0.00
192 19 Alliance 4 2 1576 0% 0% 0% 4% 0% 13% 0% 12% 0% 14% 0% 11% 0% 8% 0% 7% 8% 7% 5% 4% 3% 3% 2% 2% 0.85
1868 20 Alliance 8 2 1560 0% 0% 0% 2% 0% 9% 0% 4% 0% 10% 0% 6% 0% 11% 0% 12% 6% 5% 7% 7% 4% 5% 5% 3% 0.83
3303 21 Alliance 2 3 1521 0% 0% 0% 1% 0% 3% 0% 5% 0% 6% 0% 2% 0% 6% 0% 6% 7% 5% 5% 6% 5% 5% 4% 4% 0.96
5677 22 1501 0% 0% 0% 1% 0% 2% 0% 3% 0% 3% 0% 3% 0% 3% 0% 1% 4% 3% 4% 3% 2% 5% 4% 6% 0.02
1351 23 1491 0% 0% 0% 1% 0% 3% 0% 4% 0% 1% 0% 2% 0% 1% 0% 2% 4% 2% 2% 5% 3% 1% 4% 3% 0.01
2367 24 1489 0% 0% 0% 1% 0% 1% 0% 1% 0% 2% 0% 3% 0% 2% 0% 4% 3% 4% 3% 4% 1% 6% 3% 4% 0.01
972 25 1468 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 2% 0% 1% 0% 1% 1% 1% 2% 1% 2% 2% 3% 2% 0.00
5089 26 1512 0% 0% 0% 1% 0% 4% 0% 3% 0% 4% 0% 4% 0% 4% 0% 4% 4% 3% 5% 5% 5% 4% 6% 7% 0.02
751 27 1482 0% 0% 0% 0% 0% 2% 0% 2% 0% 1% 0% 3% 0% 1% 0% 2% 2% 3% 4% 3% 4% 4% 2% 2% 0.01
4186 28 1435 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 0% 0% 1% 0% 1% 1% 0% 1% 2% 1% 0% 1% 2% 0.00
1662 29 Alliance 1 3 1540 0% 0% 0% 1% 0% 8% 0% 7% 0% 6% 0% 6% 0% 9% 0% 7% 5% 9% 4% 7% 5% 3% 3% 3% 0.99
1700 30 1480 0% 0% 0% 0% 0% 2% 0% 1% 0% 1% 0% 1% 0% 2% 0% 2% 4% 2% 2% 3% 2% 2% 3% 5% 0.01
1967 31 1485 0% 0% 0% 0% 0% 2% 0% 2% 0% 2% 0% 0% 0% 2% 0% 4% 1% 1% 5% 4% 3% 5% 4% 4% 0.01
2035 32 1502 0% 0% 0% 1% 0% 3% 0% 0% 0% 3% 0% 3% 0% 3% 0% 2% 3% 5% 5% 4% 6% 2% 7% 3% 0.02
100 33 1521 0% 0% 0% 1% 0% 5% 0% 3% 0% 3% 0% 6% 0% 4% 0% 3% 5% 8% 4% 7% 4% 5% 7% 4% 0.03
5171 34 1431 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 2% 0% 0% 0% 0% 0% 1% 0% 1% 1% 0% 2% 1% 0.00
766 35 1455 0% 0% 0% 1% 0% 0% 0% 2% 0% 1% 0% 2% 0% 0% 0% 1% 4% 1% 1% 2% 3% 2% 2% 1% 0.00
115 36 1397 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 1% 1% 0.00
3256 37 Alliance 6 2 1519 0% 0% 0% 1% 0% 2% 0% 3% 0% 5% 0% 6% 0% 3% 0% 6% 4% 6% 8% 3% 7% 6% 4% 4% 0.92
5026 38 1413 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 2% 1% 0% 0% 0% 0% 0% 0% 1% 0.00
2473 39 1448 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 1% 0% 2% 1% 0% 1% 2% 2% 0% 1% 2% 0.00
5027 40 1414 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 1% 0% 1% 1% 1% 1% 1% 1% 1% 0% 0% 0.00
4171 41 1403 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 1% 0% 0.00
5104 42 1472 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 1% 0% 2% 0% 2% 1% 0% 2% 4% 1% 4% 2% 3% 0.01
2813 43 1363 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0.00
3482 44 Alliance 5 3 1451 0% 0% 0% 1% 0% 1% 0% 3% 0% 1% 0% 1% 0% 3% 0% 1% 2% 1% 1% 1% 1% 2% 2% 2% 0.98
846 45 Alliance 7 3 1551 0% 0% 0% 4% 0% 6% 0% 11% 0% 8% 0% 5% 0% 10% 0% 6% 4% 6% 8% 6% 7% 4% 3% 3% 0.94
4990 46 1402 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 1% 1% 0% 0.00
670 47 1457 0% 0% 0% 1% 0% 1% 0% 1% 0% 0% 0% 1% 0% 1% 0% 0% 0% 1% 2% 1% 2% 2% 0% 2% 0.00
6059 48 1473 0% 0% 0% 0% 0% 2% 0% 1% 0% 2% 0% 2% 0% 1% 0% 1% 2% 0% 3% 4% 3% 4% 1% 1% 0.01
604 49 1489 0% 0% 0% 1% 0% 1% 0% 1% 0% 2% 0% 2% 0% 1% 0% 2% 2% 2% 2% 2% 6% 2% 2% 5% 0.01
199 50 1355 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0.00
2854 51 1416 0% 0% 0% 0% 0% 1% 0% 2% 0% 0% 0% 1% 0% 0% 0% 1% 1% 1% 0% 1% 1% 1% 0% 0% 0.00
4765 52 1405 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
6036 53 Alliance 8 3 1417 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 1% 1% 0% 0% 1% 1% 0% 1% 0.99
581 54 1425 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 0% 1% 1% 0% 0% 1% 1% 1% 1% 0.00
841 55 1425 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 1% 1% 0% 0% 1% 1% 1% 1% 0.00
2135 56 1417 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 0% 0% 1% 0.00
5905 57 Alliance 3 3 1381 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 1% 0% 0% 2% 1.00
4643 58 1394 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 1% 1% 0% 0% 0% 1% 0.00
4159 59 1427 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 2% 0% 1% 1% 1% 1% 1% 1% 1% 2% 1% 0.00
5728 60 1346 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0.00
668 61 1437 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 2% 0% 1% 0% 1% 0% 1% 0% 0% 2% 1% 1% 1% 0.00
5655 62 1345 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
253 63 1370 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 0.00
5737 64 1388 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 1% 1% 0.00

This event went about as perfectly as it could for predicting alliance selection. My model was 73% sure 254 would be selected first by 1678 and it was correct on that. This also meant it thought 971 was 73% likely to be the 2nd alliance captain, which was also true. It then thought there was a further 73% chance that 368 would be the second overall pick (note this could mean they were picked either by 254 or 971 depending on who was captain. That was also a correct prediction. That also directly led to 1280 becoming the 3rd alliance captain, which was also a 73% likelihood. Basically, up to this point, both picks were pretty obvious, 254 first and 368 second. After this, things become trickier. The most likely pick for 1208 was 192 at 13% likelihood, but they ended up selecting 114 at 4% likelihood. Not a spectacular prediction like the first two, but still not bad considering there were 59 available teams, meaning each team has a naive 1/59=1.6% chance to be selected. All of the remaining captains had a pretty low likelihood of being selected by each other, so all of the other captain positions were at >60% accuracy. The remaining first round and second round selections are all pretty in line with the predictions, with the biggest surprises being 6306 and 5905 being selected in the second round.

In a fascinating outcome, the least accurate alliance predictions also came from the Silicon Valley, but in 2019 instead of 2016. Let’s take a closer look at that event:

Silicon Valley 2019 Alliance Selection Predictions
Team Rank Final Alliance Alliance Position (1=captain) desirability Alliance 1 Captain Alliance 1 First Selection Alliance 2 Captain Alliance 2 First Selection Alliance 3 Captain Alliance 3 First Selection Alliance 4 Captain Alliance 4 First Selection Alliance 5 Captain Alliance 5 First Selection Alliance 6 Captain Alliance 6 First Selection Alliance 7 Captain Alliance 7 First Selection Alliance 8 Captain Alliance 8 First Selection Alliance 8 Second Selection Alliance 7 Second Selection Alliance 6 Second Selection Alliance 5 Second Selection Alliance 4 Second Selection Alliance 3 Second Selection Alliance 2 Second Selection Alliance 1 Second Selection Team Brier
846 1 Alliance 1 1 1674 100% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
254 2 Alliance 2 1 1893 0% 86% 14% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.48
649 3 Alliance 1 2 1745 0% 9% 86% 4% 2% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.58
1868 4 Alliance 3 1 1636 0% 0% 0% 9% 90% 1% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.02
604 5 Alliance 4 1 1741 0% 5% 0% 61% 9% 19% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.28
199 6 Alliance 5 1 1495 0% 0% 0% 1% 0% 0% 92% 0% 8% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.68
4669 7 Alliance 6 1 1493 0% 0% 0% 0% 0% 0% 1% 1% 91% 1% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.68
7419 8 Alliance 7 1 1523 0% 0% 0% 0% 0% 2% 0% 3% 2% 3% 85% 0% 7% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.58
2367 9 Alliance 8 1 1481 0% 0% 0% 0% 0% 0% 0% 2% 0% 1% 9% 3% 80% 0% 6% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1.53
5737 10 1430 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 13% 1% 80% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.65
7667 11 Alliance 7 2 1487 0% 0% 0% 1% 0% 0% 0% 1% 0% 1% 0% 2% 0% 2% 14% 3% 2% 3% 2% 3% 5% 4% 4% 4% 0.99
2220 12 Alliance 6 2 1632 0% 0% 0% 10% 0% 21% 0% 27% 0% 18% 0% 9% 0% 8% 0% 3% 1% 3% 1% 1% 0% 0% 0% 0% 0.99
6039 13 1462 0% 0% 0% 0% 0% 2% 0% 1% 0% 0% 0% 2% 0% 1% 1% 1% 1% 2% 1% 3% 2% 4% 3% 4% 0.01
6241 14 Alliance 8 3 1424 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 1% 0% 1% 1% 1% 2% 2% 3% 0.98
5499 15 Alliance 2 2 1582 0% 1% 0% 1% 0% 6% 0% 8% 0% 16% 0% 12% 0% 17% 0% 7% 9% 8% 5% 5% 5% 1% 0% 1% 1.08
4159 16 1481 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 2% 0% 1% 0% 2% 2% 2% 3% 3% 2% 3% 2% 8% 0.01
1458 17 1501 0% 0% 0% 1% 0% 0% 0% 3% 0% 2% 0% 2% 0% 4% 0% 3% 6% 3% 8% 5% 5% 6% 5% 2% 0.02
4171 18 1455 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 1% 0% 2% 0% 1% 1% 1% 0% 3% 2% 2% 4% 2% 0.00
256 19 1425 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 2% 1% 1% 2% 2% 2% 0.00
7736 20 1422 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 1% 1% 0% 0% 1% 1% 0% 1% 0% 0.00
4990 21 Alliance 3 3 1556 0% 0% 0% 1% 0% 5% 0% 6% 0% 10% 0% 13% 0% 10% 0% 9% 9% 5% 4% 4% 9% 5% 4% 2% 0.97
581 22 1470 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 2% 0% 2% 0% 1% 1% 2% 4% 4% 3% 4% 4% 3% 0.01
8 23 Alliance 3 2 1546 0% 0% 0% 1% 0% 3% 0% 4% 0% 6% 0% 8% 0% 8% 0% 8% 9% 10% 11% 6% 8% 8% 2% 4% 1.01
2135 24 1475 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 4% 0% 3% 3% 6% 3% 2% 4% 2% 5% 3% 0.01
5026 25 Alliance 5 2 1637 0% 1% 0% 8% 0% 30% 0% 26% 0% 15% 0% 10% 0% 6% 0% 3% 1% 1% 1% 1% 0% 0% 0% 0% 0.90
6036 26 1516 0% 0% 0% 0% 0% 1% 0% 1% 0% 3% 0% 4% 0% 3% 0% 6% 3% 8% 6% 6% 7% 7% 6% 5% 0.03
5171 27 1468 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 1% 0% 4% 2% 5% 5% 3% 2% 1% 1% 4% 0.01
1967 28 1468 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 2% 0% 1% 0% 2% 1% 2% 3% 2% 2% 2% 3% 4% 0.00
5773 29 1418 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 1% 0% 1% 1% 2% 2% 0.00
192 30 Alliance 5 3 1517 0% 0% 0% 0% 0% 1% 0% 2% 0% 2% 0% 4% 0% 6% 0% 5% 7% 4% 6% 6% 5% 7% 8% 3% 0.91
2489 31 1471 0% 0% 0% 0% 0% 1% 0% 2% 0% 0% 0% 1% 0% 1% 0% 2% 2% 3% 2% 4% 3% 3% 4% 4% 0.01
2035 32 1476 0% 0% 0% 1% 0% 2% 0% 1% 0% 0% 0% 1% 0% 2% 0% 4% 3% 3% 4% 5% 4% 5% 4% 4% 0.01
841 33 1468 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 1% 0% 4% 2% 2% 2% 1% 3% 4% 4% 3% 0.01
4973 34 1469 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 2% 0% 1% 0% 1% 4% 1% 2% 3% 2% 3% 4% 5% 0.01
972 35 Alliance 4 2 1484 0% 0% 0% 0% 0% 3% 0% 0% 0% 3% 0% 1% 0% 0% 0% 6% 3% 1% 6% 4% 6% 5% 3% 6% 1.02
6418 36 Alliance 2 3 1538 0% 0% 0% 2% 0% 2% 0% 4% 0% 6% 0% 6% 0% 6% 0% 5% 13% 10% 8% 6% 7% 4% 8% 3% 0.91
670 37 1520 0% 0% 0% 2% 0% 1% 0% 4% 0% 2% 0% 7% 0% 6% 0% 7% 5% 4% 4% 6% 4% 5% 5% 8% 0.04
3482 38 1496 0% 0% 0% 0% 0% 0% 0% 1% 0% 3% 0% 4% 0% 4% 0% 5% 5% 4% 2% 5% 3% 4% 4% 6% 0.02
4186 39 1447 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 1% 1% 1% 1% 3% 2% 2% 2% 2% 0.00
751 40 Alliance 7 3 1428 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 1% 1% 1% 1% 1% 1% 1% 1% 1% 0.99
766 41 Alliance 6 3 1426 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 1% 2% 1% 1% 1% 0% 2% 3% 0% 0.98
5027 42 1290 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
2473 43 Alliance 8 2 1450 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 2% 0% 2% 0% 1% 1% 2% 2% 1% 3% 1% 3% 1% 0.99
6238 44 1402 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 1% 0.00
6665 45 1383 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 1% 1% 0% 0.00
7245 46 1306 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
7308 47 Alliance 4 3 1376 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 0% 0% 0% 0% 1% 0% 1% 0% 1% 0% 1% 1.00
6619 48 1349 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0.00
6970 49 1425 0% 0% 0% 0% 0% 0% 0% 0% 0% 2% 0% 1% 0% 1% 0% 0% 1% 1% 1% 1% 1% 0% 1% 2% 0.00
4255 50 1425 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 0% 1% 0% 1% 1% 1% 1% 0% 2% 2% 0.00
840 51 Alliance 1 3 1366 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 1% 0% 0% 0% 0% 0% 1.00
668 52 1391 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 1% 1% 0% 1% 0.00
2854 53 1399 0% 0% 0% 0% 0% 0% 0% 1% 0% 1% 0% 0% 0% 1% 0% 0% 1% 1% 0% 1% 0% 1% 0% 1% 0.00
5831 54 1397 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 1% 1% 1% 1% 0% 0.00
7445 55 1442 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 2% 0% 2% 1% 2% 1% 1% 0% 1% 1% 3% 0.00
1351 56 1427 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 0% 0% 0% 1% 1% 1% 1% 2% 1% 1% 1% 1% 0.00
5849 57 1361 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 1% 0% 1% 0% 0% 0% 0% 0% 0.00
299 58 1346 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0.00
2813 59 1401 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 0% 1% 0% 0% 0% 0% 1% 1% 1% 1% 1% 1% 3% 0% 0.00

These predictions went horribly wrong right out of the gate. The model expected with an 86% chance that 846 would select 254 first and that did not happen. Why they didn’t is beyond me, perhaps 254 declined and this decline went unrecorded in Brian’s sheet, or perhaps 254 privately told 846 before selections that they were going to decline and then 846 didn’t even bother to try to select 254. Or perhaps 846 truly just thought 649 was a better partner for them than 254. Who knows. Whatever the reason, that one choice of selecting 649 first completely throws off all remaining captains predictions (except 1868 correctly being predicted as the 3rd alliance captain), and there are no other top 8 interpicks to help straighten things out. Beyond that, the teams 972, 8, 7667, 5499, and 6241 were picked earlier than expected, and additionally 4990 and 2220 were picked later than expected. Furthermore, a few of the second round picks were surprises that the teams were even picked at all. Overall, just a really bad set of predictions for this event.

In summary, this was a fun challenge. Not sure how useful the analysis is, I think the biggest takeaway is just how complicated of a system alliance selections can be. Beyond the first ~4 picks, it is a pretty darn tricky system to model. Still very happy with how it all came together, even though I am disappointed I was unable to predict declines with any meaningful accuracy, because those would have been some of the coolest to have in real time.

Hoping to get these alliance selection predictions incorporated into my simulator in the coming weeks, but no promises.

Sorry for the triple post, I was above the character limit.

4 Likes
4

This alone is going to make these predictions so difficult at high levels.

6 Likes
5

Incredible work - thank you for sharing it! I’m always impressed with the models that you publish.

Forgive me if you wrote about this elsewhere - but what are your goals for the event simulator you are making?

3 Likes
6

Thank you! It’s a fun thing for me to work on.

It has evolved and expanded over time. It started as a tool to automatically generate calculated contributions that I could feed into my scouting database. I eventually added seed values and live calculations so that it could be used within an event to calculate team metrics. Later I added in ranking projections. Bob Croucher helped to incorporate images and schedule strengths. This year, I added a sheet to calculate hub return distributions. Sometime, I’m going to also add these alliance selection predictions as well, and probably playoff predictions after that eventually.

Overall, it is a useful tool that I can work off of to analyze FRC events, and I publish it so that others can do the same.

2 Likes
7

Hey Caleb,

This is something I wanted to get working for the predictions contest but I am running too close on time for that. I think we should be able to use alliance selection predictions like these to update the Elos of teams and improve match predictions.

Teams make picks using actual scouting data, which is much more individual and informative than 3v3 match scores. I have found that being a first pick is more predictive of future success than being an alliance captain. pic: 2016 Championship Alliance Selection Rates by Qualification Type

This is a dataset that is not used for OPR - and could leave to even better predictions than OPR! (Especially at events later in the season like DCMPs and Houston).

I think the alliance selection predictions would need to change to use strictly Elo, instead of Elo + rank. Then after each pick in the draft you could update the Elo for every team that was eligible to be picked like this:

selectProbability = 1/(1+e^(-(elo-avg_available_elo)/s)
selectDifference = selected - selectProbability
change = k * selectDifference

You would need some experimentation to find the right k value. I also think that this might work better if the k for playoff matches was reduced. I have long thought that most of the benefit that Elo got from playoff matches was the fact that the first pick is typically the best/second best team and they typically win the event, but that information can also be captured during the alliance selection phase.

Let me know what you think!

1 Like
8

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.