Go to Post CHALLENGE: A dozen Krispy Kreamers to the first team to make a three-pointer during a break at an NBA game. - Natchez [more]
Home
Go Back   Chief Delphi > ChiefDelphi.com Website > Extra Discussion
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #16   Spotlight this post!  
Unread 03-15-2018, 08:57 PM
Jacob Plicque Jacob Plicque is offline
Registered User
FRC #0086 (Team Resistance)
Team Role: Mentor
 
Join Date: Apr 2006
Rookie Year: 2001
Location: Jacksonville, Fl
Posts: 61
Jacob Plicque is an unknown quantity at this point
Question Re: paper: Caleb's Event Simulator 2018

I was recently asked to eliminate or adjust two rounds for team 3556 at Orlando. It seems they had a software problems that shut them down. The officials would not replay the two matches. I am guessing that their OPR is about 25% low. Surely other folks that are scouting regionals might benefit by being able to ignore outliers. Any suggestions?
Reply With Quote
  #17   Spotlight this post!  
Unread 03-23-2018, 07:41 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

A week 4 book has been uploaded. The only major change is to update seed values.

Quote:
Originally Posted by Jacob Plicque View Post
I was recently asked to eliminate or adjust two rounds for team 3556 at Orlando. It seems they had a software problems that shut them down. The officials would not replay the two matches. I am guessing that their OPR is about 25% low. Surely other folks that are scouting regionals might benefit by being able to ignore outliers. Any suggestions?
If you send me the matches to ignore I can manually make this adjustment pretty easily. I've had the idea to incorporate something like this in the back of my mind for a while, but it was probably around 6th on my priority list of things to add. I'll move it up a few slots since you requested it, but that's still no guarantee that it'll be out this season.

Last edited by Caleb Sykes : 03-23-2018 at 06:09 PM.
Reply With Quote
  #18   Spotlight this post!  
Unread 03-23-2018, 07:46 AM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,051
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Caleb Sykes View Post
If you send me the matches to ignore I can manually make this adjustment pretty easily. I've had the idea to incorporate something like this in the back of my mind for a while, but it was probably around 6th on my priority list of things to add. I'll move it up a few slots since you requested it, but that's still no guarantee that it'll be out this season.
Doesn't this open up a whole can of worms with people requesting data to be altered? How does this scale beyond a few reports of "robot malfunction?" If you okay with incorporating this "scouting data," why not just ask for scouting data from teams from all events?
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #19   Spotlight this post!  
Unread 03-23-2018, 08:13 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Eugene Fang View Post
Doesn't this open up a whole can of worms with people requesting data to be altered? How does this scale beyond a few reports of "robot malfunction?" If you okay with incorporating this "scouting data," why not just ask for scouting data from teams from all events?
I wasn't thinking of changing any of my published data, that would certainly be a dangerous road to go down. I just meant that I could send him a modified list of contributions for a single event to use as he would like. I personally think it's kind of silly to remove matches from the data set for a specific team, since all teams' ratings would be improved if you ignored their worst matches, but to each his own.
Reply With Quote
  #20   Spotlight this post!  
Unread 03-23-2018, 08:33 AM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,051
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Caleb Sykes View Post
I wasn't thinking of changing any of my published data, that would certainly be a dangerous road to go down. I just meant that I could send him a modified list of contributions for a single event to use as he would like. I personally think it's kind of silly to remove matches from the data set for a specific team, since all teams' ratings would be improved if you ignored their worst matches, but to each his own.
That makes much more sense.
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #21   Spotlight this post!  
Unread 03-30-2018, 07:01 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

I added a week 5 book. The only change is updated seed values.

I'm actually pretty close to having detailed match and ranking predictions functional, but need a little more time to assure they are working properly. I'll likely post an optional update tonight that includes these.

Since right after week 1, I've also been meaning to determine how effective Elo and calculated contributions are at predicting the winners of matches, so I'm going to try my best to do that tonight as well.
Reply With Quote
  #22   Spotlight this post!  
Unread 03-30-2018, 09:44 PM
Schroedes23's Avatar
Schroedes23 Schroedes23 is offline
Registered User
AKA: Noah Schroeder
FRC #1625 (Winnovation)
Team Role: Alumni
 
Join Date: Oct 2013
Rookie Year: 2012
Location: Winnebago
Posts: 11
Schroedes23 is an unknown quantity at this point
Re: paper: Caleb's Event Simulator 2018

Is there any update on if the ranking predictor will be posted tonight?
Quote:
I added a week 5 book. The only change is updated seed values.

I'm actually pretty close to having detailed match and ranking predictions functional, but need a little more time to assure they are working properly. I'll likely post an optional update tonight that includes these.

Since right after week 1, I've also been meaning to determine how effective Elo and calculated contributions are at predicting the winners of matches, so I'm going to try my best to do that tonight as well.
Reply With Quote
  #23   Spotlight this post!  
Unread 03-30-2018, 09:56 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Schroedes23 View Post
Is there any update on if the ranking predictor will be posted tonight?
Working on it.

Past Caleb didn't do a very good job commenting his code or even bothering to structure it in an understandable way, which makes it difficult for me to adapt it for 2018.

I'm glad someone cares though, that's definitely a motivational boost.
Reply With Quote
  #24   Spotlight this post!  
Unread 03-30-2018, 11:20 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Yeah, it's not happening tonight. Sorry to anyone who was looking forward to it. I'll take another crack at it tomorrow.
Reply With Quote
  #25   Spotlight this post!  
Unread 03-31-2018, 05:46 AM
Nathan Streeter's Avatar
Nathan Streeter Nathan Streeter is offline
FIRST Fan(atic)
FRC #1519 (Mechanical MAYHEM)
Team Role: Mentor
 
Join Date: Feb 2008
Rookie Year: 2006
Location: Manchester, NH
Posts: 771
Nathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond reputeNathan Streeter has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Caleb Sykes View Post
Working on it.

Past Caleb didn't do a very good job commenting his code or even bothering to structure it in an understandable way, which makes it difficult for me to adapt it for 2018.

I'm glad someone cares though, that's definitely a motivational boost.
Quote:
Originally Posted by Caleb Sykes View Post
Yeah, it's not happening tonight. Sorry to anyone who was looking forward to it. I'll take another crack at it tomorrow.
I am also interested in it - it has seemed to be a surprisingly effective means of predicting rankings / match outcomes! Definitely better than making educated guesses match-by-match! That said, I really only use the event simulator for events at which my own team is competing... I have a hard enough time just keeping up with watching / checking rankings for other events.
__________________
"If you want to build a ship, don't drum up men to gather wood, divide the work, or give orders. Instead, teach them to yearn for the vast and endless sea." - Antoine de Saint-Exupery
"The fight is won or lost far away from witnesses - behind the lines, in the gym, and out there on the road, long before I dance under those lights." - Muhammad Ali
"Simplicity is the ultimate sophistication." - Leonardo da Vinci


Student: 2006-2010 (#1519)
Mentor: 2011-Present (#1519)


Reply With Quote
  #26   Spotlight this post!  
Unread 03-31-2018, 09:32 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Well, I uploaded a 5.2 version that includes ranking projections. It has almost no model tuning and is probably filled with bugs. The extent of my testing was that the ranking projections looked approximately correct for mnmi.

If you find bugs, feel free to let me know. Use it at your own risk.

Last edited by Caleb Sykes : 03-31-2018 at 09:36 AM.
Reply With Quote
  #27   Spotlight this post!  
Unread 03-31-2018, 09:44 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

I just fixed a major bug that caused red to not get credit for their projected wins. Up to 5.3 now.
Reply With Quote
  #28   Spotlight this post!  
Unread 04-02-2018, 11:44 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

About 4 weeks overdue, I finally went back and found how predictive various methods were of predicting matches. Here are the results for week 4 competitions excluding Israel champs. Values are Brier scores, lower Brier scores mean higher predictive power.

As expected, my Elo model takes a slight but appreciable edge over a simple calculated contribution to total points (OPR) model. I still believe it would be possible to improve upon this calculated contributions model to beat my Elo model, but I have yet to spend sufficient time developing anything that could prove or disprove this theory.

The most powerful predictor we have remains the simple average between calculated contributions and Elo, so this is what I will be using for match predictions moving forward (I've been using Elo only so far just because I hadn't done a proper analysis until now).

To the surprise of no one who follows my work, CCWM predictions are awful as usual relative to the other methods, but I thought I'd throw them in anyway since someone was bound to ask about them.

In comparison to other years, using these methods we have slightly less predictive power than 2010, and slightly more predictive power than 2016.

I'll make a calibration curve for Elo sometime soon, but I'm confident it's well calibrated.
Reply With Quote
  #29   Spotlight this post!  
Unread 04-04-2018, 09:14 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Quote:
Originally Posted by Caleb Sykes View Post
I'll make a calibration curve for Elo sometime soon, but I'm confident it's well calibrated.
Here is the calibration curve for Elo for all matches through week 5. It's a pleasing little graph, maybe even avatar-worthy . There is just a touch of under-confidence in it, but nothing concerning, I'll probably look more into the under-confidence once the season is over.

The predictive power of Elo this year so far is greater than that of Elo in any year on record except 2011, and the playoff predictions this year are the best on record (since 2005). Although to be fair, the ovarall predictive power will probably drop after championships, which is generally the most difficult event to predict due to regional variations in team strength and how the game is played.

Total Brier score is 0.1750, quals Brier score is 0.1774, playoffs Brier score is 0.1634, 73.34% of matches correctly predicted.
Reply With Quote
  #30   Spotlight this post!  
Unread 04-05-2018, 09:48 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,655
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Caleb's Event Simulator 2018

Just uploaded v6.1. Key changes are:
Various bug fixes
Added calculated contribution predictions
Updated seed values
Updated Instructions and FAQ

I still haven't done as thorough of a testing as I would like, so I bet there are still bugs. If anything looks off, let me know. Match win loss predictions should be improved since I'm not only using Elo now. The other ranking points predictions are still probably poor since I haven't had time to calibrate their parameters.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 12:01 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi