OCCRA
Go to Post Woodie Flowers would like to win the Chuck Norris award. - Andy Baker [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 08-07-2018, 08:46 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,677
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
How does the FiveThirtyEight Elo formula work?

Can someone help me to understand the Elo update formula used by FiveThirtyEight? Here is the best article I'm aware of describing how it works. Specifically, I don't understand how they derived their "margin of victory multiplier"

Margin of Victory Multiplier = LN(ABS(PD)+1) * (2.2/((ELOW-ELOL)*.001+2.2))

I don't know where any of the constants in this formula come from, the use of a base e logarithm, the 2.2, and the 0.001. Are these values empirically determined? Or are they related intrinsically to the rest of the model? It also seems like a good tuning parameter that they don't include would be to divide the point differential by some constant A (like maybe the standard deviation of NFL PDs), since there's nothing fundamental about NFL point differentials. Unless something like this is hidden in the other parameters.

I've tried to use this formula before for FRC without success, and I want to take another crack at it, but I think I need a better understanding of how it works first.

Last edited by Caleb Sykes : 08-07-2018 at 08:53 AM.
Reply With Quote
  #2   Spotlight this post!  
Unread 08-07-2018, 09:21 AM
JohnSchneider's Avatar
JohnSchneider JohnSchneider is offline
Registered User
FRC #3310 (Black Hawk Robotics)
Team Role: Mentor
 
Join Date: Mar 2010
Rookie Year: 2010
Location: Dallas
Posts: 887
JohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond reputeJohnSchneider has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

This guy explains it pretty well

http://andr3w321.com/elo-ratings-par...y-adjustments/
Reply With Quote
  #3   Spotlight this post!  
Unread 08-07-2018, 10:26 AM
Unsung FIRST Hero
Karthik Karthik is offline
Registered User
no team
Team Role: Mentor
 
Join Date: Apr 2002
Rookie Year: 1998
Location: Toronto, Ontario, Canada
Posts: 2,599
Karthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by Caleb Sykes View Post
I don't know where any of the constants in this formula come from, the use of a base e logarithm, the 2.2, and the 0.001. Are these values empirically determined? Or are they related intrinsically to the rest of the model?

Based on my understanding they were empirically determined by looking at historical distributions of point differentials. The logarithm was chosen to ensure diminishing returns on large point differentials. (As to why base e was specifically chosen, again it comes to down to historical point distributions) The link that John shared explains this better than I have.
__________________
:: Karthik Kanagasabapathy ::
"Enthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful and you will accomplish your object. Nothing great was ever achieved without enthusiasm" -- R.W. Emerson
My TEDx Talk - The Subtle Secrets of Success
Reply With Quote
  #4   Spotlight this post!  
Unread 08-07-2018, 11:09 AM
Matt G's Avatar
Matt G Matt G is offline
365 --> 4039
AKA: Mattward
FRC #4039 (Makeshift)
Team Role: Mentor
 
Join Date: Apr 2015
Rookie Year: 2009
Location: Ontario
Posts: 54
Matt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud of
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by JohnSchneider View Post
He explains how to get to that equation, but in my opinion doesn't really explain any of the whys about the intricacies of the equation. Still an interesting read though. I also don't have the answer, but I won't let that stop me from speculating based entirely on irrelevant details.

Quote:
MOVM = LN(ABS(PD)+1) * (2.2/((ELOW-ELOL)*.001+2.2))
I suspect that log_e was chosen over log_n because it doesn't make too much of a difference. The curve can be scaled by the second portion of the equation to get the appropriate response.

The form of the second portion allows for essentially two different responses, one for underdog victory and one for favorite victory. It's not a linear function, so underdog victories result in a larger MOVM than favorite victories of the same point differential. This is most significant for abs(ELOW-ELOL) > 500. For -500>ELOW-ELOL>500, it's mostly linear.

The 0.001 and 2.2 aren't really two different constants, they're interdependent (let's call them a and b, a=0.001 and b=2.2). If you take a=0.01 and b=22 you get the same result. The ratio of the two, a/b, determines the non-linearity of the second portion of the MOVM equation, as discussed previously. If you increase a and keep b constant, the function becomes more non-linear, increasing the difference between the reward for underdog victories and favorite victories.

The two also limit the scope of the function. In this case, for (ELOW-ELOL) <= -2200, it breaks. The more non-linear you'd like the function, the smaller of an elo difference it can allow (only for underdog victories of course, Rocky can lose all he wants but as soon as he beats Apollo you have issues).

So my post doesn't answer any of your questions directly, but hopefully it helps with deciding what values would be acceptable for FRC. I suspect log_e is appropriate for pretty much any competition, and the a/b constant is what would actually be tuned. When tuning a/b it's probably necessary to pay attention to the highest conceivable elo difference for an underdog win. Since you sum individual ELOs to achieve an alliance ELO, I suspect it's not unreasonable that you could break this equation given a large upset, but you can probably look at historical data to make sure.

If a and b aren't guess+check, I would think that they depend on the largest conceivable underdog win ELO difference and the prevalence of underdog victories. If underdog victories are common I would expect a more linear fit, but if they are uncommon I would expect a more non-linear fit. As to how to translate that into a number, , good luck!
Reply With Quote
  #5   Spotlight this post!  
Unread 08-07-2018, 12:08 PM
Unsung FIRST Hero
Karthik Karthik is offline
Registered User
no team
Team Role: Mentor
 
Join Date: Apr 2002
Rookie Year: 1998
Location: Toronto, Ontario, Canada
Posts: 2,599
Karthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by Matt G View Post
The form of the second portion allows for essentially two different responses, one for underdog victory and one for favorite victory. It's not a linear function, so underdog victories result in a larger MOVM than favorite victories of the same point differential. This is most significant for abs(ELOW-ELOL) > 500. For -500>ELOW-ELOL>500, it's mostly linear.

Yup, exactly. This is explained by Silver at the link in the OP.

Quote:
Originally Posted by Nate Silver
Autocorrelation is the tendency of a time series to be correlated with its past and future values. Let me put this into football terms. Imagine I have the Dallas Cowboys rated at 1550 before a game against the Philadelphia Eagles. Their rating will go up if they win and go down if they lose. But it should be 1550 after the game, on average. That’s important, because it means that I’ve accounted for all the information you’ve given me efficiently. If I expected the Cowboys’ rating to rise to 1575 on average after the game, I should have rated them more highly to begin with.
It’s true that if I have the Cowboys favored against the Eagles, they should win more often than they lose. But the way I was originally designed, I can compensate by subtracting more points for a loss than I give them for a win. Everything balances out rather elegantly.
The problem comes when I also seek to account for margin of victory. Not only do favorites win more often, but when they do win, they tend to win by a larger margin. Since I give more credit for larger wins, this means that their ratings tend to get inflated over time.

Quote:
Originally Posted by Matt G View Post
I suspect log_e is appropriate for pretty much any competition, and the a/b constant is what would actually be tuned. When tuning a/b it's probably necessary to pay attention to the highest conceivable elo difference for an underdog win.

Probably not a coincidence that the natural logarithm was chosen, given that e^2~=7, e^3 ~= 20, and that a "close" game is usually defined as MoV <=7, with diminishing returns are usually seen after MoV > 21 in the NFL. (e.g. The difference between 7 and 14 point win in the NFL is massive, while a 21 and 28 point win is fairly minor)
__________________
:: Karthik Kanagasabapathy ::
"Enthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful and you will accomplish your object. Nothing great was ever achieved without enthusiasm" -- R.W. Emerson
My TEDx Talk - The Subtle Secrets of Success

Last edited by Karthik : 08-07-2018 at 12:13 PM. Reason: Clarity on y = e^x
Reply With Quote
  #6   Spotlight this post!  
Unread 08-07-2018, 01:05 PM
Citrus Dad's Avatar
Citrus Dad Citrus Dad is offline
Business and Scouting Mentor
AKA: Richard McCann
FRC #1678 (Citrus Circuits)
Team Role: Mentor
 
Join Date: May 2012
Rookie Year: 2012
Location: Davis
Posts: 1,285
Citrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by Karthik View Post
(As to why base e was specifically chosen, again it comes to down to historical point distributions)
Natural logs have mathematical and statistical properties that fit with statistical theory (that I've now forgotten after 2 decades...)
__________________
Reply With Quote
  #7   Spotlight this post!  
Unread 08-07-2018, 01:35 PM
Matt G's Avatar
Matt G Matt G is offline
365 --> 4039
AKA: Mattward
FRC #4039 (Makeshift)
Team Role: Mentor
 
Join Date: Apr 2015
Rookie Year: 2009
Location: Ontario
Posts: 54
Matt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud ofMatt G has much to be proud of
Re: How does the FiveThirtyEight Elo formula work?

I hadn't read the posted article all the way through, only skimmed it. I read the second post article, but I definitely should have read the first one all the way through instead of just looking for numbers and equations.

Quote:
Originally Posted by Karthik View Post
Probably not a coincidence that the natural logarithm was chosen, given that e^2~=7, e^3 ~= 20, and that a "close" game is usually defined as MoV <=7, with diminishing returns are usually seen after MoV > 21 in the NFL. (e.g. The difference between 7 and 14 point win in the NFL is massive, while a 21 and 28 point win is fairly minor)
That's good reasoning that I hadn't considered, however:

since ln(x) = 2.303*log(x), I can rewrite Mr. Silver's equation as:

MOVM = LOG(ABS(PD)+1) * (5.0666/((ELOW-ELOL)*0.002303+5.0666))

and I should get the same values. I think the selection of the base of the logarithm can be accounted for by selecting appropriate constants for the second portion of the equation.

Which led me to think that maybe the equation would be easier to interpret with constants of 1 and some number, rather than 2.2 and some number:

MOVM = Log_1.575(ABS(PD)+1) * (1/((ELOW-ELOL)*.0004545+1))

I like the form of 1/x+1, but it makes the logarithm pretty messy. And it leaves us wondering why Log_1.575 and 0.0004545 are significant... They're pretty ugly numbers.

Log_1.575(7) = 4.3
Log_1.575(21) = 6.7


For Caleb: Nate Silver is relatively accessible via twitter, so you could probably ask him there and have a shot at a reply. Although I'd probably wait until after today's primaries/special election are over, I bet he's pretty busy right now. There's also an email for him on fivethirtyeight.com, so you could try that as well... If you do ever get an answer from him, please post it here, I'm pretty interested.
Reply With Quote
  #8   Spotlight this post!  
Unread 08-07-2018, 01:45 PM
Unsung FIRST Hero
Karthik Karthik is offline
Registered User
no team
Team Role: Mentor
 
Join Date: Apr 2002
Rookie Year: 1998
Location: Toronto, Ontario, Canada
Posts: 2,599
Karthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond reputeKarthik has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by Citrus Dad View Post
Natural logs have mathematical and statistical properties that fit with statistical theory (that I've now forgotten after 2 decades...)

Quote:
Originally Posted by Matt G View Post

That's good reasoning that I hadn't considered, however:

since ln(x) = 2.303*log(x), I can rewrite Mr. Silver's equation as:

MOVM = LOG(ABS(PD)+1) * (5.0666/((ELOW-ELOL)*0.002303+5.0666))

and I should get the same values. I think the selection of the base of the logarithm can be accounted for by selecting appropriate constants for the second portion of the equation.

Which led me to think that maybe the equation would be easier to interpret with constants of 1 and some number, rather than 2.2 and some number:

MOVM = Log_1.575(ABS(PD)+1) * (1/((ELOW-ELOL)*.0004545+1))

I like the form of 1/x+1, but it makes the logarithm pretty messy. And it leaves us wondering why Log_1.575 and 0.0004545 are significant... They're pretty ugly numbers.

Log_1.575(7) = 4.3
Log_1.575(21) = 6.7

Yup, the choice of the natural logarithm is quite possibly a result because the numbers fall into approximately the right space and because it's so readily used in traditional statistics/science.
__________________
:: Karthik Kanagasabapathy ::
"Enthusiasm is one of the most powerful engines of success. When you do a thing, do it with all your might. Put your whole soul into it. Stamp it with your own personality. Be active, be energetic, be enthusiastic and faithful and you will accomplish your object. Nothing great was ever achieved without enthusiasm" -- R.W. Emerson
My TEDx Talk - The Subtle Secrets of Success
Reply With Quote
  #9   Spotlight this post!  
Unread 08-07-2018, 09:04 PM
cobbie's Avatar
cobbie cobbie is offline
Head Scout and Data Analyst
AKA: Evan
FRC #1425 (Error Code Xero)
Team Role: Tactician
 
Join Date: Mar 2017
Rookie Year: 2015
Location: West Linn, OR
Posts: 107
cobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant futurecobbie has a brilliant future
Re: How does the FiveThirtyEight Elo formula work?

Here is an article about how they use it for the NBA.
__________________
Thanks to our 2018 alliance partners:
2471 1359 2046 5468 6445 16 418 4795

Thanks to our event winning alliance partners:
2007 Las Vegas 254 1983 2009 Seattle 488 1569 2012 Seattle Olympic 2557 2928
2013 Central Washington 1983 753 2014 Wilsonville 4488 2002 2014 Oregon St. 2811 4457
2016 Wilsonville 4488 3711 2016 Auburn 4469 3786 2017 CAIS 4488 6445
2018 Wilsonville 2471 1359 2018 Auburn Mountainview 2046 5468
Reply With Quote
  #10   Spotlight this post!  
Unread 08-09-2018, 07:58 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,677
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: How does the FiveThirtyEight Elo formula work?

Quote:
Originally Posted by cobbie View Post
Here is an article about how they use it for the NBA.
Here's the margin of victory multiplier from footnote 2 from that link, the general structure looks pretty similar to the NFL multiplier except using a power funtion instead of a logarithm.


Here's a graph of the MOV versus the multiplier for each of the two formulas for elo_diff = 200:


Note that it's kind of silly to directly compare NFL margins of victory against NBA MOVs since they have different units, but since the average MOV for each sport is ~10 points we can do at least a rough comparison. My guess is that the NFL has higher multipliers near 0 because NFL teams who are winning early can control the pacing of the game much better than NBA teams up by similar margins early can. So if NFL teams were actually playing to maximize margin of victory instead of win probability, teams with leads would probably play more aggressively than they presently do. That still doesn't tell us how either of these multipliers were constructed or tuned though.

I'll be sending Nate Silver an email soon, I'll share the whole email chain here after we're done.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 05:57 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi