Sykes Event Simulator 2019

#1

This workbook is a tool which can be used to simulate current 2019 events, or to calculate statistics for completed events. Data is pulled from the TBA API in the “data import” sheet. Explanations for how to use each sheet can be found in the “instructions and FAQ” sheet. I changed the name this year to “Sykes Event Simulator” because of the apostrophe in “Caleb’s Event Simulator” causing me issues. Also I’m keeping up the tradition of giving it a new name each year. :slightly_smiling_face:

You can download it from GitHub here:
Sykes Event Simulator 2019

I’m giving a github repo a shot. It seems like a good solution so far, but we’ll have to see. Thanks to wjordan for the suggestion.

As of today, it is missing a lot of the functionality I had at the end of last year. I have hidden the non-functional sheets, so everything visible should be fully functional. I’ve done a fair bit of testing, but there are always bugs, so if you come across one don’t hesitate to reach out. My tentative plan is to have team lookup, match lookup, and current rankings done by tomorrow. Ranking projections probably won’t come until next week.

Here are my simulators from previous years if you are curious:
2017
2018

8 Likes
Houston Ranking Projections
Top bots of Week 1 2019 [Component OPRs]
Help Caleb Get a Job at 538!
Does anyone have scouting data on the daly division
#2

If you do reference the week 0 data, please note that the null hatch panel placement/scoring data provided by FIRST is incorrect for quals 2, and possibly for other matches as well. Somehow the null panels are recorded as attached at the start of the match, but they’re not there at the end. :man_shrugging:

#3

I’ve just uploaded v2019.0.4, key additions are:
Miscellaneous bug fixes
Updated team lookup and match lookup sheets for 2019
Various code restructuring and optimization

No ranking projections quite yet. I might take another pass at it tomorrow. The other thing I’m missing is the “Own Side Bias Points” category, as that requires different logic to implement than all of the other calculated contributions.

#4

Up to 2019.0.5:
Miscellaneous small bug fixes
Added own side bias points
Fixed a major bug that was completely messing up Elo ratings

I hope no one was taking the Elo ratings too seriously since they were completely broken, their only meaning would have come from their start of event seeds lol.

Top bots of Week 1 2019 [Component OPRs]
#5

Just noticed an issue when trying to import stats for event key arli, and bcvi. It keeps saying “Error, invalid team in custom list”

#6

Oh, it’s actually completely broken for all non week 1 events. I think I know the issue so I’ll clear that up tonight or tomorrow at the latest.

Thanks!

#7

Alright, added v0.6. I’ll have another update with better seed values on Thursday.

Changes since v0.5:
Miscellaneous bug fixes
Team Lookup and match lookup macros are now kicked off by data imports
Added playoff matches to team lookup
Added ability to search playoff matches in match lookup
Fixed a bug that did not allow simulation of non week 1 comps

#8

Can you post a raw ELO list somewhere for those of us without access to Excel?

#9

Here you go

Week 1 Elos
Team Elo
2910 1899
1519 1873
330 1860
27 1853
3538 1839
118 1818
346 1815
33 1800
973 1788
610 1783
95 1767
88 1763
1102 1761
5687 1759
614 1758
1023 1756
3309 1748
1747 1734
359 1734
103 1730
3005 1728
3604 1728
5803 1727
245 1722
624 1721
2200 1721
2930 1719
1807 1715
4481 1713
319 1713
379 1711
1506 1710
4451 1704
1262 1701
1418 1700
2363 1700
1718 1699
612 1698
1629 1697
5472 1694
4063 1693
86 1693
1983 1692
5436 1691
3990 1686
3655 1685
2102 1684
2539 1682
2928 1681
3986 1679
2834 1679
360 1678
3847 1677
573 1676
3542 1674
1318 1673
4564 1673
1836 1672
5802 1671
2147 1668
1188 1667
2468 1667
548 1665
3647 1662
3547 1661
4476 1661
1731 1660
2655 1659
3641 1659
6329 1659
1610 1658
1218 1657
4020 1656
6377 1655
2534 1654
4188 1653
5895 1653
3535 1652
1414 1652
1511 1652
1729 1651
5338 1649
4201 1649
5561 1649
316 1645
3824 1645
2607 1644
4680 1644
1250 1643
294 1642
888 1642
5885 1641
4795 1640
263 1639
342 1639
4946 1639
203 1638
3238 1637
6226 1637
1683 1636
2145 1636
1481 1635
4152 1634
4469 1634
2386 1633
4513 1633
832 1631
3494 1631
3663 1631
977 1631
384 1630
5114 1630
4276 1630
4761 1629
5684 1629
3360 1628
166 1627
3974 1626
6705 1625
6618 1624
3414 1624
4153 1623
597 1621
3255 1618
399 1618
709 1617
6334 1616
4454 1616
4099 1615
4907 1614
4079 1613
4242 1612
4414 1612
4512 1611
188 1611
4189 1610
1778 1610
5012 1610
3998 1610
339 1610
2614 1609
5234 1608
2199 1607
5401 1607
3676 1606
1599 1603
4936 1603
811 1602
3958 1602
6340 1602
5260 1602
6802 1601
1648 1601
5181 1600
201 1598
1902 1597
623 1597
3536 1596
6919 1596
3268 1596
108 1596
5467 1594
3679 1594
229 1594
2980 1592
5036 1591
5109 1590
6471 1590
3490 1590
2393 1590
1391 1589
4905 1589
3859 1589
1287 1588
1895 1588
6117 1588
192 1587
3256 1587
4930 1587
4913 1585
6867 1585
3250 1585
3140 1584
3826 1583
21 1583
3489 1581
7491 1581
122 1581
5022 1580
2612 1580
3812 1579
70 1579
4415 1579
8 1579
4965 1577
708 1577
5612 1577
422 1576
1322 1576
2048 1575
3473 1574
4309 1574
4737 1574
2576 1574
418 1573
1051 1573
663 1572
4112 1572
114 1571
1746 1571
5084 1571
3929 1570
1369 1570
131 1570
4738 1569
1647 1569
6948 1568
4005 1565
5199 1564
7447 1563
5562 1562
6328 1562
5150 1562
1413 1562
2543 1562
1793 1562
4994 1561
5457 1561
123 1561
7449 1561
5546 1558
6086 1558
2852 1558
6691 1558
611 1557
619 1557
3544 1557
6357 1556
3467 1556
5901 1555
3568 1555
2028 1554
1719 1554
4505 1553
6144 1553
2106 1552
4525 1552
3455 1552
2998 1552
1721 1552
5067 1551
5137 1551
2881 1550
6081 1550
2559 1550
7605 1549
1293 1548
6135 1547
5090 1547
1143 1547
4951 1547
6882 1546
159 1546
281 1546
2458 1546
2988 1546
94 1546
6570 1546
4947 1545
3566 1545
5804 1545
6072 1545
5683 1545
4610 1545
6646 1544
3550 1543
3341 1543
2994 1542
7501 1542
116 1542
3386 1541
467 1540
4619 1540
5498 1540
6742 1539
4285 1539
6567 1538
6548 1538
5641 1538
7426 1537
5669 1536
3786 1534
6504 1533
1080 1532
4504 1531
4373 1531
2158 1530
816 1530
348 1530
3136 1529
1159 1528
5426 1527
4701 1526
3361 1525
1539 1524
4378 1524
5215 1524
6366 1524
6029 1523
540 1523
3021 1523
5866 1523
5851 1520
1277 1520
6110 1520
1160 1520
283 1520
3258 1518
1515 1518
7053 1517
6616 1517
1247 1516
5756 1516
5048 1516
6155 1516
280 1516
7574 1515
3328 1515
1831 1514
4987 1514
5697 1514
4384 1514
5666 1514
6528 1514
5409 1513
2084 1513
2493 1513
293 1512
7416 1511
1701 1511
900 1511
7152 1510
6808 1510
6869 1510
7865 1510
3581 1509
2848 1509
468 1509
4582 1509
5553 1508
1086 1508
4984 1508
4377 1507
2613 1507
4089 1507
6019 1507
2815 1506
3219 1504
7735 1504
7807 1504
1908 1503
7514 1502
949 1502
2945 1501
4015 1501
5526 1500
4682 1500
6560 1500
5531 1500
835 1499
4568 1498
703 1497
4240 1497
1553 1497
3373 1497
5950 1496
2620 1495
812 1495
1398 1495
4683 1495
7218 1494
5086 1494
3941 1494
5549 1494
7414 1494
2421 1494
5853 1492
4301 1492
4929 1492
5865 1492
4799 1492
647 1491
4509 1491
7196 1491
2605 1490
5317 1490
6543 1490
1522 1490
7540 1490
5805 1490
4958 1489
7107 1489
3952 1489
2234 1489
4748 1489
841 1488
5503 1487
4854 1486
6910 1485
5410 1485
7271 1485
4073 1485
7627 1485
620 1485
2903 1485
4519 1484
6595 1484
5651 1483
6866 1483
5265 1483
4851 1481
6929 1481
1661 1481
3953 1481
4783 1480
3743 1480
5633 1480
2186 1480
5025 1478
6070 1478
2583 1478
4243 1477
5774 1477
2714 1477
7157 1476
3379 1476
3533 1476
5902 1476
2224 1475
6220 1475
5295 1474
7267 1474
5474 1474
6584 1474
6987 1474
5327 1473
2187 1473
7116 1473
6851 1473
6180 1473
7712 1473
1123 1472
5047 1472
6321 1472
4075 1471
3049 1471
5263 1471
6682 1470
6093 1470
815 1470
6091 1470
2984 1469
7750 1469
3366 1469
6514 1468
7489 1468
3976 1468
6872 1468
2673 1467
7837 1466
7072 1466
7872 1465
7137 1465
4594 1465
1512 1465
7511 1465
3359 1464
4652 1464
539 1463
5949 1463
5059 1463
5966 1463
3274 1463
6895 1463
6859 1462
4763 1462
1164 1461
457 1460
5538 1460
7085 1458
5533 1458
7789 1457
6921 1456
7482 1456
1528 1455
7700 1455
4083 1455
5810 1453
7614 1453
7538 1452
1598 1452
5500 1451
5783 1451
7913 1451
6597 1451
4173 1450
3834 1450
5478 1449
1758 1449
3759 1448
2961 1448
4957 1448
5613 1446
5130 1446
7818 1446
7701 1445
7429 1445
7230 1445
4074 1444
6974 1444
4267 1443
2708 1443
7232 1442
7138 1442
4328 1442
7676 1442
7409 1441
4161 1441
6222 1441
4704 1440
4180 1440
7104 1439
7471 1439
4579 1439
6918 1438
5443 1437
6975 1437
3975 1437
7621 1437
7615 1435
2095 1435
7073 1435
5197 1435
7907 1434
7716 1433
4286 1433
5957 1433
5532 1432
3651 1432
5734 1431
7857 1431
5608 1431
7692 1431
7710 1430
2789 1430
2819 1429
5052 1429
5619 1429
7191 1428
6905 1426
3981 1425
4187 1425
7856 1425
5754 1425
3561 1425
7264 1423
7143 1423
4717 1423
6712 1422
7649 1422
6402 1420
3985 1420
6013 1419
6626 1418
6545 1418
3571 1418
585 1418
3988 1417
5103 1416
2626 1416
4056 1415
5527 1415
6499 1413
7060 1412
7533 1412
3532 1411
617 1411
6887 1410
6863 1410
6943 1410
6167 1408
4464 1408
6750 1408
4823 1408
6864 1407
4955 1407
7555 1406
3295 1405
7788 1404
6190 1404
7564 1403
6189 1402
5777 1401
6692 1401
7690 1400
4114 1400
7162 1399
7772 1398
6610 1397
6690 1395
6535 1393
2721 1390
7134 1388
2496 1384
6021 1383
6102 1382
4840 1382
2950 1381
4838 1381
7330 1376
6664 1376
6541 1370
6961 1363
6540 1353
4847 1352
6953 1343
5835 1332
7524 1313
5063 1287
1 Like
#10

Are these only the teams that have been in an event so far? Where would the complete week 1 Elo list be?

#11

You can find it in the “seed values” sheet of my scouting database.

Was going to also copy them here, but that’s too many characters for a post, so here’s a google sheet with them as well.

#12

Just added a week 1 update, I’ll be trying to get future updates out by Wednesdays, but early weeks are always rough as I work through fun new issues.

Changes since v0.6:
Removed the useless metric in placeholder1
updated seed values
Adjusted match predictions to use an average of Elo and Predicted Contribution predictions
Now calculates own side bias in calculated contributions

Hoping to add rank predictions by next week, I think the structure to add them is mostly there.

#13

Thanks so much again, Caleb, for providing your super awesome simulation spreadsheet again! I know it’s a ton of work for you each year. I do appreciate it, as do many others I’m sure.

Hoping to add rank predictions by next week, I think the structure to add them is mostly there.

Is that also why the only option for Settings, “Simulations to Run” is 0 (metrics only, no simulations)?

2 Likes
#14

Correct, no point making imports run more slowly because they’re running simulations that aren’t functional

#15

Alright, I did some work yesterday to get predicted rankings up and running. Those are now available in v 1.1.

Changes since 2019.1.0:
Miscellaneous small bug fixes
Added predicted rankings
Added predicted rankings to team lookup
Added current rankings
Added current rankings to team lookup and match lookup sheets
Added continuous ranking projection sheet
Hid the "seed values" sheet by default. Predicted Contributions will show seed values before the event starts, and if you want to see them again you can unhide the sheet or just run an import starting from before match 1

I’m always terrified of releasing the first round of ranking projections because I’m afraid of bugs, but I have to make the leap sometime so here I go. If you want to help me out and look for bugs, there are a couple of good tools you can use:
First, you can check the “rankings” sheet to verify all the data lines up with official results. The data in my rankings sheet is not directly pulled from TBA, but rather constructed based on match data, so any error there is likely to carry into the rankings projections.
Second, I have given you the option to simulate the event from any point in time by adjusting a cell in “data import”. One simple way to use this is to simulate consecutive matches and see if the ranking projections update in a way you expect for teams that competed in that match.
Lastly, building up on #2, I have a sheet called “continuous ranking projections” which can make cool graphs telling the story of a team’s ranking throughout the event. Everything on those graphs should make intuitive sense if you think about it. For example, here is 2052 at mndu2:


They have some very good matches in q7 and q17 in which the model learns they are competitive and can get the HAB RP consistently. Winning but not getting the HAB RP in q26 doesn’t really affect their top 4, top 8 or top 15 chances since they still won, but their chances of seeding first takes a hit. Through match 51, they lock in their chances of seeding very high, but not much changes in their 1 seed race as both they and 525 continue winning and getting the HAB RP. Everything comes to a head in q52 though, where 2052 comes in as an underdog, but manages to get BOTH bonus RPs and win, and not just against any opponent either, but against 525, who they were racing for the 1 seed. After that, 2052 is 2 full RPs up on 525, and slowly lock in the 1 seed through the end of the event.
For another story, here’s 1410 at okok:

Unlike 2052, who the model would have guessed would rank in the top 8 going into the event, 1410 comes in not even expected to get top 15. They don’t quite have as many flashy matches like 2052, but instead you see a slow but steady climb up the rankings as they win matches but don’t get bonus RPs. There’s a solid jump to the top 15 chances in q31 where they get their sole bonus RP. THere’s also a bump to the top 4, top 8, and top 15 at q59 where they win an underdog match. You can see their top 4 chances getting reasonably high until they lose q74. Their top 4 chances then drop to 0 in q87 when 2996 and 4523 lock in more RPs than 1410, which locks 1410 out of the top 4. One last note, the noisiness of the continuous ranking projections will go down as you select more simulations to run. The above graph for 1410 was created using 400 simulations at each point in time, and the one below was created using 1000:

This is clearly the same graph, but it’s much more smooth and 1410’s matches stand out a little better. So give this a shot for your team and make sure the “story” being shown matches up with what you would expect. I really love these graphs, so I hope you all do as well.

Other good spots to look for errors if you’re hunting:
Surrogate teams
DQed teams
high order ranking sorts
Outlier matches (unicorn matches, high score difference matches, matches with no-shows)
Check that teams’ min and max possible ranks are correct bounds.

3 Likes
#16

Does the match predictor extrapolate trends (up and down) of teams that get better just by playing and others that get worse because of broken components?

Is a performance ceiling calculated? Can one be calculated based upon trend data?

Since traditionally the scores week to week improve, is this also accounted for future competition predictions?

#17

Both skill metrics, Elo and predicted contributions, update with each match. So if a team improves those metrics in a match, the probabilities of winning all future matches will increase. Simulations are still run “cold” though, in the sense that in all simulations, a team’s “true” rating is assumed to be their current rating. Someday I’ll make simulations run hotter, which would let the simulator consider the possibility that the team’s actual ability is greater or worse than their current ability, but that’s a non-trivial problem, and I have other shortfalls I’d like to address first.

Not quite sure what you mean by performance ceiling. Neither Elo nor predicted contributions are bounded, so a team could get either of those ratings ludicrously high theoretically. In practice though, Elos probably won’t get much over 2000 this year and predicted contributions probably won’t get much higher than 60.

Absolute scores don’t really have much impact on the predictions, it’s only relative scores/performance that matter. Since everyone improves at roughly the same rate (or at least, I wouldn’t know how to predict who’s improving faster) I’m unsure how I could use this information to improve future comp predictions.

#18

Those ranking projection graphs are super cool, I love that feature. One of the things our team used the event simulator for last week was to graph how our contributions to certain metric changed throughout the event. This was mainly used post-event, to see how we compared to other teams at the event, and to see where our performance was trending up and down. As a potential future feature suggestion, I think it would be cool to have a chart that you could select from a drop down list of all the metrics, and plot the teams performance over the course of the event. Maybe even overlay two teams on top of each other to see how they compare.

1 Like
#19

Updated seed values through week 2. No other major changes since v1.1.

#20

That sounds pretty cool, I’ve already got most of the structure for that with “continuous ranking projections” so I think I can add this without too much work. I’ll try to add it in sometime soon.

1 Like