Sykes Event Simulator 2019

Elo is cumulative. Rather then averaging the Elo’s from all events, you should use the Elo from the most recent event.

1 Like

Thank you guys … I figured it out and now I match Caleb’s order :slight_smile:

For those playing at home, I did a little research on ELO - the fact that it’s a zero-sum game made me understand that I had to add a function to select the most recent week and then the ELO for that week by team.

Anyway, it’s a pretty cool metric. Not sure how useful it is for prediction, IMHO and I mean no disrespect to team 2056, I just think that 1323 is much more dominant. My metric isn’t better in that 3538 didn’t make my top 25 (#31) and they look very impressive.

First time using this spreadsheet. Love it! Wonderful work.

Just finished QMs in FIM States. During the qualifiers, I used Match Lookup to see win probabilities of upcoming matches. Now, when I go to that tab, I can’t change “qm” to “qf” for upcoming quarter finals. The data import does show those matches, e.g. qf 1 1, qf 1 2, etc. What am I doing wrong?

You can use the dropdown in match lookup to see what your options are. You won’t be able to enter “qf” here, it needs to be “qf1” or “qf3” or the like.

Just so I don’t have to respond to multiple requests, my simulator does not on its own have a way to incorporate the preliminary champs schedules. If TBA updates schedules then it should work from there (but their mo historically has been not to release anything unofficial). Otherwise if I get a data source that returns schedules in the same form as the TBA schedules I can make that adjustment. @wjordan did that last year here: http://wesj.org/docs/houston2018/ and then I just updated the simulator to reference that instead of TBA. He’s busy with stuff right now as am I, but if someone else wants to get his python script in that directory to run, here is what he told me:

Feel free to use the converter.py script to parse CSV scraped from the PDFs. To use it you have to manually scrape the data from the PDF using copy-paste into a CSV file (with the same name as the division). The script expects the CSV file to have the following schema: Match,Red1,Red2,Red3,Blue1,Blue2,Blue3 .

If someone does this, ping me and I can release a simulator update that references your directory.

1 Like

Thanks. The drop down is empty.

I’ve uploaded v6.2 which allows for match data sources other than TBA. Just select your desired data source in “settings”. Big thanks to @AriMB for parsing the Houston preliminary schedules and hosting them. I have his site set as the default data source in this update. Note that if you want live updating match data at Houston, or if you want to simulate non-Houston events, you will need to switch back to the TBA data source. I’m reasonably sure surrogates are being handled properly, but if you see anything weird with them let me know.

https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.6.2.xlsm

3 Likes

Updated seed values through week 7. This should be the last update you need until the end of the season, so I’m setting the outdated warning to be in May. There may still be optional updates though.
https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.7.0.xlsm

Ugh, do I actually have to put in logic to handle matches that are going to be replayed? Why does the api still return results for these matches if they haven’t even “really” been played? Are they at least flagged for replay somehow?

Nope, FMS has no concept of “this match will be replayed at sometime in the future”. Which means whenever the replay happens, the data in the API will be overwritten with the data from the second play of the match.

AFAIK there is no API anywhere (including FIRST’s) that indicates a match will be replayed. If the score got posted, it’s there and official until the replay completes (and the data is overwritten). You can check to see if a replay happened by check for a chance in the actual match time.

(Sniped by Phil… ugh :slight_smile: )

That really sucks, it affects the official rankings as well. I’m going to send an email to FIRST asking them to add a better handling capability. At least they could provide a flag.

7 Likes

Just gonna put this here… I love how passionate you are about all things statistical and FRC :smile:

4 Likes

I’ve also requested that the play count field be exposed in the API, so at least we’d at least be able to tell when a match has been replayed looking backwards.

Having a forward looking flag would be awesome as well (although this would be a larger scoped FMS feature) - I can submit an official ticket for this too

3 Likes

I’ve uploaded the preliminary schedules for Detroit to my “server”, so you can now use the Event Simulator with the Detroit prelim schedules. You’ll need to switch the “data source” in the “settings” tab to “arimb…” for it to work.

2 Likes

Alright, last required update of the season:
https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.9.0.xlsm

You should be able to use this for offseason events without issue. That said, no offseason events currently have team lists, so something might break when we actually get an offseason event with teams. In the meantime, feel free to use the custom event to see what things might look like at your upcoming offseason.

Updated to 9.1: https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.9.1.xlsm

Changes:
Code restructuring/optimizing
Added 5th and 95th percentiles to ranking projections

2 Likes

Updated to 9.2:
https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.9.2.xlsm

Changes:
Added rp data to “predicted rankings” sheet
Added rp predictions to the “team lookup” sheet
Updated some of the fields in “instructions and FAQ”
Now caches all settings changes
Code Restructuring
Added a progress bar to continuous ranking projections, which also allows the continuous rankings operation to be cancelled
Added 4 new graphs to continuous ranking projections
Added a strength of schedule sheet

This was a big feature update that I was pushing to get out before IRI. Here is a more detailed explanation of the changes:

RPs: I had always internally tracked RPs in order to do the ranking projections, but since I had @R.C ask for them directly I realized that I may as well expose them. So I added them into the ranking projections sheet as well as team lookup.

Continuous Ranking Projections: I really liked the ranking probability graphs over time I previously had here, so I decided to make a few more cool graphs. One thing I wanted was the ability to see how the average, 5th/95th percentiles, and current ranks all looked for a team over time, so I added those. I also made similar graphs that look at RPs directly instead of ranks. Finally, like @lbl1731 suggested above, I added a graph that shows how a team’s strength metric can change over time. There’s a lot of cool things you can see in these graphs, and they should also provide more opportunities for people to verify the accuracy of my predictions. For example, here is 2052’s RP projections at MN State champs:


Notice anything off here? Well, if you look at matches 38 through 42, it gives 2052 a 0% chance of getting 17+ RPs, but they do eventually get 17 RPs. Why is that? Well, it’s a poor assumption in my simulator. 2052 goes into match 43 with 13 RPs, and my simulator gives them a 0% chance of getting the rocket RP. When they did end up getting the rocket RP, my simulator is caught off guard. 0% probabilities for theoretically possible outcomes are really dumb. I’m working on alternatives to my bonus RP predictions that don’t provide 0% or 100% predictions. Feel free to use these graphs to spot check my work. The predictions should never go to 0% if the result ends up being true, and likewise for 100% predictions ending up false.
The other much needed feature for the continuous ranking projections is a loading bar that allows process cancellation. Since this operation takes on the order of 15 minutes for me, I would often trap myself running this with no way to get out. Hopefully that won’t happen now, and it’s just nice to know roughly how far along the process is.

Strength of Schedule: Thanks to @bobcroucher for helping to get this one in. I’ve had a couple people ask about schedule strengths for this year, and although you can certainly calculate them without this tab, it’s pretty cumbersome. So I’ve added a new tab which can be used to calculate schedule strengths for any event that has the schedule released. My metric of choice is column E, but there are others as well. Also interesting to see how the projections change for the top 15 seeds before and after schedule releases.

As always, let me know if you see any bugs. Have fun at IRI! Once the schedule is out we can see who got lucky/unlucky

3 Likes

Quick bug fix:
https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.9.3.xlsm

Continuous Ranking Projections weren’t working and now they should.

Another small bug fix. This should make it so the incomplete match data published before the detailed breakdowns doesn’t cause the ranking projections to go whack:
https://github.com/inkling16/2019EventSimulator/raw/master/Sykes_Event_Simulator_2019.9.4.xlsm