Attached are projections for qualification for each of the 2019 district championships:
2019chs.html (23.7 KB)
2019fim.html (96.3 KB)
2019fma.html (24.0 KB)
2019fnc.html (13.9 KB)
2019in.html (11.9 KB)
2019isr.html (12.1 KB)
2019ne.html (37.6 KB)
2019ont.html (32.3 KB)
2019pch.html (16.6 KB)
2019pnw.html (28.2 KB)
2019tx.html (34.6 KB)
I previously created a post for just the PNW: PNW District Championship Projections 2019 That post has more information about the methodology.
Hey, thanks a lot for doing this! How do you come up with the prediction numbers for the cutoff? Is there anyways you can apply that same method to project a cutoff for the Detroit world champs?
The algorithm to come up with the cutoff is basically:
There’s no reason that something fairly similar couldn’t be done to try to predict the cutoffs to go to either of the championship events. I’ve thought about doing this and if anyone wanted to code this up I’d accept a patch.
Without some other changes I’m not sure that it would be super helpful though. I think it might tell you something like the #1 team coming in has a 60% chance to advance and the last-place team has a 35% chance to advance. And neither of those numbers is something that I would want to plan a trip based on. Given how heavily the district championship events are weighted (3x) I think I’d have to incorporate an actual model of team strength before it could generate interesting results.
I love what you have done with this!
What happens when you click on them? And what kind of a computer are you using?
Can you include the DCMP event size information you were using to make these estimates in the HTML? Your FMA estimate seems to hit 50% at rank 50, despite 60 teams being invited. I realize this is a possibility of the fact there are a handful of teams that haven’t played yet (although fewer than 10 and far fewer than those who have played twice) and how points end up being distributed, but it seems odd to me that the break even point would be 10 spots higher than the invite quantity.
Thank you for posting this, fascinating!
You might be right and I have no idea what his method is, but he could be accounting for chairman’s teams reserving 8 spots, so only using 52 points spots. Still doesn’t bring the break even point to 50 but its closer
I’ll think about including the number of slots when I run this again next week. It definately does take this into account. As I was developing this I noticed that if you don’t do that you get FiM with a 90+ pt cutoff.
There is no particular reason to that the place for number of 50% odds would be anywhere near the cutoff. For example, if I ran the program before any of the events had been played in a district with 50% of teams advancing you’d get the rookies above 50% and all the year 2+ veterans below 50%, and that might be 80% of teams.
Slots are reserved for Chairmans only for the awards that have already been won.
Attached are predictions updated post week 4:
2019chs.html (23.5 KB)
2019fim.html (96.1 KB)
2019fma.html (24.0 KB)
2019fnc.html (13.7 KB)
2019in.html (11.6 KB)
2019isr.html (12.1 KB)
2019ne.html (37.5 KB)
2019ont.html (32.3 KB)
2019pch.html (16.5 KB)
2019pnw.html (28.0 KB)
2019tx.html (34.4 KB)
There has been one minor improvement to the methodology: teams that are not scheduled for a first or second event are not assumed to play them. This mostly means that teams that dropped out before the season began aren’t counted as potential competitors.
And there is one more caveat that I didn’t previously mention that teams should be aware of: it doesn’t understand tiebreakers. And it doesn’t get that there might be multiple teams with the same final point total, some of which get in and some of which don’t.
Nice work, but the CHS district champs will have 58 slots this year, not 60. Source: https://www.firstchesapeake.org/first-programs/frc/first-chesapeake-district-events/first-chesapeake-district-championship
Good catch. I was following an outdated version of the manual. Here’s a new version:
2019chs.html (23.5 KB)
Attached are updated projections. Note that this is not official in any way.
2019tx.html (35.2 KB)
2019pnw.html (28.0 KB)
2019pch.html (16.7 KB)
2019ont.html (35.9 KB)
2019ne.html (41.8 KB)
2019isr.html (12.5 KB)
2019in.html (11.3 KB)
2019fnc.html (13.5 KB)
2019fma.html (24.2 KB)
2019fim.html (102.2 KB)
2019chs.html (23.6 KB)
The methodology has been changed to improve the handling of ties in the number of rankings points earned. If you look at the bottom of the page, it will give a much longer explanation of what the program thinks the cutoff might be and the portion of teams at that cutoff that will make it in. Note that it still doesn’t understand tiebreakers, just that some of the teams at a given score will get in, and some won’t.
I like the predictions. Helpful to have an idea of where you stand. Thank you.
However, I think it would be helpful if your pages included the number of points already earned by a team and the number of events contributing. It would make it easier to understand the probabilities.
FYI, it appears that you are using TBA to pull your data from. I would recommend against this practice as their data is not always consistent/reliable.
An example of this is FRC68 Truck Town Thunder. They are listed on TBA as winning the Judges Award at their 2nd event (Troy), however received 0pts for this award win. On TBA they are listed as only having 63pts (which it appears is what your projections have them as), while the FIRST official leader board correctly has them at 68, which should put them closer to 95% in compared to the 0.2% they are sitting at in your model.
I’m sure if you dug deep enough, you’d find more discrepancies in the TBA data. Pull from FIRST’s site; it’s more reliable.
For anyone interested, I’ve generated my own FiM State Championship projections that can be seen here: https://docs.google.com/spreadsheets/d/12baxdwjKqgOsiG_4fOd2FaXuO7fNFF20L1lPwZob6WM/
It takes a more categorical approach rather than assigning probabilities to teams.
I see the error on The Blue Alliance that you’re describing. That’s strange because not only does it not match the official source, but it’s also not internally consistent – it’s showing the award but no points for it.
I have previously found anomalies in both the data from The Blue Alliance and from the official source. Just this weekend the official pages were not showing the “In progress” marker when they should have been. I’m not sure that either of them is really a better option. I may have to consider pulling from both and reconciling the differences.
By the way, I have at least one other error in the predictions for this week: I pulled the data after all of the matches had finished but before all the awards had been recorded, which meant the 3786, who won a chairman’s award didn’t get included in the list for the PNW championship.
Those categories in your predictions look familiar :). In previous years I’ve done things that way.
Here’s a summary of how well my predictions worked out this year. There’s a section by week and then overall. Within each section there is a table that gives an idea of how well-calibrated the results are.
The entry under “MAD” is the average difference between the prediction and the final. If one were to just flip a coin for each team, this would be .5, or if you were to just guess that this team had a 50% change to make it in it would also be around .5. The entry under “rms” is also known as the Brier score.
mar10
Predicted Observed Samples
0<x<10 0.0 5
10<=x<20 0.09 34
20<=x<30 0.28 18
30<=x<40 0.34 53
40<=x<50 0.71 7
50<=x<60 0.83 6
60<=x<70 0.64 11
70<=x<80 1.0 4
80<=x<90 1.0 2
90<=x<100 1.0 6
x=100 1.0 9
mad: 0.336321935484
rms: 0.400023810824
mar17
Predicted Observed Samples
x=0 0.01 234
0<x<10 0.01 205
10<=x<20 0.06 262
20<=x<30 0.24 291
30<=x<40 0.34 147
40<=x<50 0.53 160
50<=x<60 0.65 98
60<=x<70 0.84 55
70<=x<80 0.92 50
80<=x<90 0.97 32
90<=x<100 0.96 47
x=100 1.0 248
mad: 0.22786440678
rms: 0.323887001132
mar24
Predicted Observed Samples
x=0 0.0 484
0<x<10 0.03 186
10<=x<20 0.08 222
20<=x<30 0.25 170
30<=x<40 0.34 93
40<=x<50 0.51 73
50<=x<60 0.8 54
60<=x<70 0.79 43
70<=x<80 0.98 43
80<=x<90 0.91 22
90<=x<100 0.95 43
x=100 0.98 396
mad: 0.15088354292
rms: 0.267576994126
mar31
Predicted Observed Samples
x=0 0.01 848
0<x<10 0.08 111
10<=x<20 0.06 88
20<=x<30 0.4 58
30<=x<40 0.38 29
40<=x<50 0.4 30
50<=x<60 0.76 21
60<=x<70 0.73 11
70<=x<80 0.85 13
80<=x<90 0.8 5
90<=x<100 0.92 24
x=100 0.97 591
mad: 0.0734882449426
rms: 0.210646096619
Overall:
Predicted Observed Samples
x=0 0.01 1566
0<x<10 0.04 507
10<=x<20 0.07 606
20<=x<30 0.26 537
30<=x<40 0.34 322
40<=x<50 0.51 270
50<=x<60 0.72 179
60<=x<70 0.79 120
70<=x<80 0.94 110
80<=x<90 0.93 61
90<=x<100 0.95 120
x=100 0.98 1244
mad: 0.155843654732
rms: 0.275676432301
Overall, the results are way ahead of random, but I’d like to have a better baseline to compare against.
There’s definitely room for improvement here because it should be possible to get the 0% and 100% bins to be right all of the time, and the calibration shows that there’s an underestimation of teams that are doing well and overestimation of teams that are doing poorly, part of which could be fixed by a model of team skill, or by just adding some calibration tables.
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.