OCCRA
Go to Post "We used to shut the robot up in a box and send it away from home. Now we just suffocate it." - GaryVoshol [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
View Poll Results: I am interested in...
Making and submitting predictions 2 3.13%
Viewing other people's predictions 20 31.25%
Both! 36 56.25%
Neither. Predictions can make underperforming teams feel bad. 6 9.38%
Voters: 64. You may not vote on this poll

Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 08-15-2018, 06:21 PM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,058
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Need Your Input -- FRC Prediction Competition Website

Match result and ranking prediction contests are a fun way to learn about the interesting methods the FRC community has come up with for predicting match results, helpful for finding close or blowout matches, and an opportunity for playful banter. I’d love to see prediction contests for more than just a small handful of events each year, so I’m thinking about building an automated FRC prediction competition website.

The general idea is that people would submit predictions to the website via an API on a per-event basis. Predictions for a given event could be submitted and updated as many times as desired, but updated predictions for matches that have already started would be ignored. All predictions and some aggregate of them would be displayed publicly for each match. A stats page could show how accurate everyone’s predictions are.

Some nice things we get from this setup:
  • People no longer have to manually deal with spreadsheets (thanks, everyone who has organized a prediction contest!)
  • Algorithms that adapt their prediction throughout the course of an event can be better showcased
  • The FRC community can more easily browse predictions compared to viewing a spreadsheet

My question for you: Would you either be interested in participating in making predictions or viewing predictions made by the community?

If so, there are still a few things to be decided. Here are a few big ones:
  • Prediction API granularity: Do we care just about win/loss? Probability of win/loss? Probability of getting bonus RP? Final match score? Probability distribution of final match score? Final match subscores (e.g. auto score & teleop scores separately)? Final game state (e.g. everything that shows up in the “Detailed Results” table on TBA)? The list can go on and on...
  • Do we want to including ranking predictions? If so, we again need to decide on the granularity of data (e.g. average rank vs. rank probability distribution).
  • What stats should we show to rank how accurate predictions are? Since it’s not required to submit predictions for every event, we may need separate stats that rank by the whole season vs. single event.
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #2   Spotlight this post!  
Unread 08-15-2018, 06:38 PM
AriMB's Avatar
AriMB AriMB is offline
The Philadelphian emigrant
AKA: Ari Meles-Braverman
FRC #5987 (Galaxia)
Team Role: Mentor
 
Join Date: Mar 2015
Rookie Year: 2012
Location: Haifa, Israel
Posts: 1,657
AriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

It would make everything a bit more complicated, but in theory you could have all of the prediction granularity options at once. If someone only submits win/loss predictions, they are only "competing" in the win/loss rankings.

If they submit win/loss probabilities, they can be entered for that ranking's "competition" and the system can calculate the implicit win/loss prediction based on the probabilities and enter them in that "competition" also. And so on for all of the categories, where your prediction counts for the "competition" you entered in, as well as all of the less-granular ones.

This way, people with really complex prediction schemes can show off their granularity*, while beginners just trying to predict who will win also get a shot.


*not sure this is the right word here...you get what I mean though
__________________
Studying MechE at the Technion - Israel Institute of Technology
2017-present: FIRST Israel CSA/FTAA
2017-present: FRC 5987 Technical Mentor 18isr2 18isr4 18isrcmp 18carv
2012-2016: FRC 423 Member 15njtab 15padre 16paphi
Reply With Quote
  #3   Spotlight this post!  
Unread 08-15-2018, 06:56 PM
Billfred's Avatar
Billfred Billfred is offline
Never look back, no, never say die
AKA: Always attack with the warrior's cry...
FRC #1293 (Pandamaniacs); FTC #11444 (Garnet Squadron) (Also, CD mod.)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: The Shadow Realm
Posts: 9,540
Billfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by AriMB View Post
It would make everything a bit more complicated, but in theory you could have all of the prediction granularity options at once. If someone only submits win/loss predictions, they are only "competing" in the win/loss rankings.

If they submit win/loss probabilities, they can be entered for that ranking's "competition" and the system can calculate the implicit win/loss prediction based on the probabilities and enter them in that "competition" also. And so on for all of the categories, where your prediction counts for the "competition" you entered in, as well as all of the less-granular ones.

This way, people with really complex prediction schemes can show off their granularity*, while beginners just trying to predict who will win also get a shot.


*not sure this is the right word here...you get what I mean though
Basically came here to say this.

ALSO: a low-friction way to submit human-generated predictions would be appreciated. Bonus points for mobile-friendly, so as to work in the stands or at the hotel during an event. Maybe a web interface attracts more guessers than data scientists, but I think there's value in that once you get enough people weighing in.
__________________
William "Billfred" Leverette - Gamecock/Jessica Boucher victim/Owner/Director of Bricks 4 Kidz in Columbia, SC.

2004-2006: FRC 1293 (D5 Robotics) - Student, Mentor, Coach
2007-2009: FRC 1618 (Capital Robotics) - Mentor, Coach
2009-2013: FRC 2815 (Los Pollos Locos) - Mentor, Coach - Palmetto '09, Peachtree '11, Palmetto '11, Palmetto '12
2010: FRC 1398 (Keenan Robo-Raiders) - Mentor - Palmetto '10
2014-2016: FRC 4901 (Garnet Squadron) - Co-Founder and Head Bot Coach - Orlando '14, SCRIW '16
2017: FRC 5402 (Iron Kings) - Mentor, Coach (oh, and I worked at AndyMark during this time)
2018-: FRC 1293 (Pandamaniacs) - Mentor

113 events (more than will fit in a ChiefDelphi signature), 15 seasons, over 74,500 miles, and still on a mission from Bob.
Rule #1: Do not die. Rule #2: Be respectful. Rule #3: Be safe. Rule #4: Follow the handbook.
Reply With Quote
  #4   Spotlight this post!  
Unread 08-15-2018, 07:02 PM
IndySam's Avatar
IndySam IndySam is offline
Registered User
FRC #0829 (Digital Goats)
Team Role: Mentor
 
Join Date: Nov 2005
Rookie Year: 2004
Location: Indy
Posts: 3,597
IndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond reputeIndySam has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

you need to add "I really don't care"
__________________
"Champions are champions not because they do anything extraordinary but because they do the ordinary things better than anyone else." —Chuck Knoll


Reply With Quote
  #5   Spotlight this post!  
Unread 08-15-2018, 07:10 PM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,058
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by AriMB View Post
It would make everything a bit more complicated, but in theory you could have all of the prediction granularity options at once. If someone only submits win/loss predictions, they are only "competing" in the win/loss rankings.

If they submit win/loss probabilities, they can be entered for that ranking's "competition" and the system can calculate the implicit win/loss prediction based on the probabilities and enter them in that "competition" also. And so on for all of the categories, where your prediction counts for the "competition" you entered in, as well as all of the less-granular ones.

This way, people with really complex prediction schemes can show off their granularity*, while beginners just trying to predict who will win also get a shot.


*not sure this is the right word here...you get what I mean though
That's a good idea. If we go down this route, we still need a reasonable cutoff for granularity. I highly doubt that anyone will actually do final game state predictions, so implementing that would be a waste of time. I'm thinking win/loss, prob win/loss, and prob bonus RP is sufficient, at least for the first year we try this. We can get fancier in the future.

Quote:
Originally Posted by Billfred View Post
ALSO: a low-friction way to submit human-generated predictions would be appreciated. Bonus points for mobile-friendly, so as to work in the stands or at the hotel during an event. Maybe a web interface attracts more guessers than data scientists, but I think there's value in that once you get enough people weighing in.
The interface would be straightforward and would call the same backend API. Have to think about how this scales. I'm sure we can handle a few dozen "data scientists," but accepting predictions from hundreds (thousands?) of students might get very $$$ very quick.

Quote:
Originally Posted by IndySam View Post
you need to add "I really don't care"
Mod please add?
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #6   Spotlight this post!  
Unread 08-15-2018, 07:26 PM
Billfred's Avatar
Billfred Billfred is offline
Never look back, no, never say die
AKA: Always attack with the warrior's cry...
FRC #1293 (Pandamaniacs); FTC #11444 (Garnet Squadron) (Also, CD mod.)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: The Shadow Realm
Posts: 9,540
Billfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Eugene Fang View Post
Mod please add?
Just checked, we can't edit them.
__________________
William "Billfred" Leverette - Gamecock/Jessica Boucher victim/Owner/Director of Bricks 4 Kidz in Columbia, SC.

2004-2006: FRC 1293 (D5 Robotics) - Student, Mentor, Coach
2007-2009: FRC 1618 (Capital Robotics) - Mentor, Coach
2009-2013: FRC 2815 (Los Pollos Locos) - Mentor, Coach - Palmetto '09, Peachtree '11, Palmetto '11, Palmetto '12
2010: FRC 1398 (Keenan Robo-Raiders) - Mentor - Palmetto '10
2014-2016: FRC 4901 (Garnet Squadron) - Co-Founder and Head Bot Coach - Orlando '14, SCRIW '16
2017: FRC 5402 (Iron Kings) - Mentor, Coach (oh, and I worked at AndyMark during this time)
2018-: FRC 1293 (Pandamaniacs) - Mentor

113 events (more than will fit in a ChiefDelphi signature), 15 seasons, over 74,500 miles, and still on a mission from Bob.
Rule #1: Do not die. Rule #2: Be respectful. Rule #3: Be safe. Rule #4: Follow the handbook.
Reply With Quote
  #7   Spotlight this post!  
Unread 08-16-2018, 01:29 AM
Brian Maher's Avatar
Brian Maher Brian Maher is offline
#JusticeForPinecone
FRC #2791 (Shaker Robotics)
Team Role: College Student
 
Join Date: Apr 2014
Rookie Year: 2012
Location: Troy, NY
Posts: 1,214
Brian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond reputeBrian Maher has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

I will post more thoughts eventually, but I found having an easy to use web interface helps a lot with engaging participants. The invite prediction contest had a Google form where participants could just vote yes or no on each team. 75 people entered, many of whom didn't seem to be especially involved in the FRC data scene.

I think an important question to ask is, what is the goal here: engage new/casual players exploring predictions and data, or optimize for power users (@SLFF crew and Caleb)?
__________________
2016-present, Mentor, FRC 2791 - Shaker Robotics
2018: Central New York Winner (2791, 340, 5030, 7081), Tech Valley Finalist (2791, 20, 3624, 5123), Darwin Division QF (4946, 2791, 2834, 3461), IRI Invite
2017: Tech Valley Winner (333, 2791, 5952), NYC SF (2791, 2383, 1880), Curie Division QF (2481, 2791, 203, 1807), IRI Invite, Robot Rumble Winner (2791, 5881, 1880)
2016: Tech Valley SF (5236, 2791, 3624), Finger Lakes SF (5254, 2791, 2383), Battlecry Winner (195, 2791, 501), Robot Rumble Winner (2791, 195, 6463)

2012-2015, Student, FRC 1257 - Parallel Universe
2015: Mount Olive QF (1257, 1923, 1811) , North Brunswick Finalist (11, 193, 1257)
2014: Clifton Winner (1626, 869, 1257), MAR CMP QF (1257, 293, 303)
2012: Mount Olive QF (204, 303, 1257)

2017-present, SLFF - Questionable Decisionmakers
2018: Regular Season Finalist, Championship 3rd Place, Fantasy FIM Finalist
2017: Regular Season Winner, Championship Winner, Fantasy FIM Finalist, IRI Winner
Reply With Quote
  #8   Spotlight this post!  
Unread 08-16-2018, 01:53 AM
MikLast's Avatar
MikLast MikLast is online now
Eats Tableau for Breakfast
AKA: Mikal Dieatrick
no team (Volunteer)
Team Role: Alumni
 
Join Date: Jan 2015
Rookie Year: 2014
Location: Medical Lake, WA
Posts: 1,374
MikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond reputeMikLast has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Brian Maher View Post
I think an important question to ask is, what is the goal here: engage new/casual players exploring predictions and data, or optimize for power users (@SLFF crew and Caleb)?
Considering SLFF isint optimized in any way for new players, i kinda hope the former will be the answer...
__________________


Join the FRC Discord!
FRC4513 (Circuit Breakers)
*Student: 2014-2017
*Drive Coach: 2016-2017

Volunteer
A/V: West Valley, PNW DCMP 2018
Reply With Quote
  #9   Spotlight this post!  
Unread 08-16-2018, 07:37 AM
BrennanB's Avatar
BrennanB BrennanB is offline
CD Off-season...
AKA: Brennan Bibic
FRC #4476 (W.A.F.F.L.E.S.)
Team Role: Coach
 
Join Date: Jan 2012
Rookie Year: 2006
Location: Kingston, ON, Canada
Posts: 2,049
BrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond reputeBrennanB has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by MikLast View Post
Considering SLFF isint optimized in any way for new players, i kinda hope the former will be the answer...
Same thoughts, really a lot of focus on SLFF should be put towards to making it easier to draft/see who got picked/easily accessible scoring.

I suspect it's very similar for something like this. Easy to submit predictions, what the predictions are, and what the results were/who is good at predicting.
__________________
@b_bibic :::: Brennan Bibic :::: 2014 DLF :::: brennanbibic.com
Fantasy First 2014-2018 (TBC) Discord - 10 years of inspiration compilation

92 events - W.A.F.F.L.E.S. #4476 YouTube Channel - 69,315.2 Kilometers
FRC|> 2009-2012 (K-Botics #2809) | 2013-2018 (W.A.F.F.L.E.S. Community Robotics #4476) <|FRC
FLL|> 2006-2018 (W.A.F.F.L.E.S. #105) <|FLL | VRC|> 2010-2013, 2015-2016 (W.A.F.F.L.E.S. #4476) <|VRC
“Skills are cheap. Passion is priceless.”Gary Vaynerchuk

Reply With Quote
  #10   Spotlight this post!  
Unread 08-16-2018, 07:52 AM
wesbass23 wesbass23 is offline
Registered User
FRC #1732 (Hilltopper Robotics)
Team Role: Coach
 
Join Date: Mar 2012
Rookie Year: 2011
Location: Milwaukee
Posts: 182
wesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud ofwesbass23 has much to be proud of
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by IndySam View Post
you need to add "I really don't care"
Also known as not replying.

This is a neat idea, would be cool to have a site used for pickem or fantasy leagues beyond ChiefDelphi or custom spreadsheets. Not sure if this is quite what you are looking to do but I could see it evolving into that.
__________________
2017 Wisconsin Regional Winners with 1675 and 2194
2017 St. Louis Regional Winners with 1706 and 537
2014 Wisconsin Regional Winners with 2481 and 2202
2014 Midwest Regional Finalists with 2338 and 3936
2014 Midwest RCA Winner
2013 Wisconsin Regional Winner with 111 and 2338
2013 MARC Winners with 1718 and 1023
2012 Boilermaker RCA Winner
Reply With Quote
  #11   Spotlight this post!  
Unread 08-16-2018, 08:29 AM
Eugene Fang's Avatar
Eugene Fang Eugene Fang is offline
The Blue Alliance
FRC #0604 (Quixilver)
Team Role: Alumni
 
Join Date: Jan 2007
Rookie Year: 2000
Location: Bay Area, CA -> Pittsburgh, PA
Posts: 1,058
Eugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond reputeEugene Fang has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Brian Maher View Post
I think an important question to ask is, what is the goal here: engage new/casual players exploring predictions and data, or optimize for power users (@SLFF crew and Caleb)?
I didn't really consider having a bunch of people casually vote on who they think will win until the idea was brought up in this thread. My original intended goal was to optimize for power users in terms of prediction submission and make viewing predictions accessible to everyone. That way, we can have high quality predictions (unlike the TBA predictions page I never had time to properly update for 2018) that are more accessible than Caleb's Event Simulator (which is awesome), while automating the spreadsheet-based prediction contests.

Quote:
Originally Posted by wesbass23 View Post
This is a neat idea, would be cool to have a site used for pickem or fantasy leagues beyond ChiefDelphi or custom spreadsheets. Not sure if this is quite what you are looking to do but I could see it evolving into that.
I definitely do not intend for this to become a fantasy league platform for FRC.
__________________
Eugene Fang
2010 Silicon Valley Regional Dean's List Finalist

Various FLL Teams - Student (2000-2006), Mentor (2007-2010)
FRC Team 604 - Student (2007-2010), Mentor/Remote Advisor (2011-Present)
FRC Team 1323 - Mentor/Remote Advisor (2011-2014)

The Blue Alliance | TBA GameDay | TBA Android App | TBA Blog | TBA Swag
Reply With Quote
  #12   Spotlight this post!  
Unread 08-16-2018, 09:21 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,694
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Eugene Fang View Post
[*]Prediction API granularity: Do we care just about win/loss? Probability of win/loss? Probability of getting bonus RP? Final match score? Probability distribution of final match score? Final match subscores (e.g. auto score & teleop scores separately)? Final game state (e.g. everything that shows up in the “Detailed Results” table on TBA)? The list can go on and on...
I would say WL and probability of WL for sure, and maybe the other RPs as well. Anything more than that seems like overkill at this point, although I could see adding more things in the future.

Quote:
[*]Do we want to including ranking predictions? If so, we again need to decide on the granularity of data (e.g. average rank vs. rank probability distribution).
I'd say no, keep it simple in year 0 and see where people want to see things go from there. I didn't get around to ranking projections in my simulator until after I had spent a ton of time on other things. We'll likely find other uses for this along the way, so no need to force anything this big so early.

Quote:
[*]What stats should we show to rank how accurate predictions are? Since it’s not required to submit predictions for every event, we may need separate stats that rank by the whole season vs. single event.[/list]
For each event and for the season, I personally would like Brier scores, number of matches predicted, and maybe some measure of over-confidence (either a single number or a calibration curve) if you have a big enough sample size. Probably also good to split quals and playoff matches. I'm not a big fan of "matches correctly predicted", but I get that this is more intuitive than Brier scores, so I guess putting it in is fine so long as there's a big enough sample size.

You could also show Brier scores only for matches that were predicted, but it might be better to default any non-entries to a prediction of 50%. That way no one can just predict the obvious matches and get an artificially good Brier score. If you want to get a single score to evaluate who had the "best" predictions at an event or set of events, I would recommend a Brier score with a 50% prediction inserted for any matches not predicted.


I'm pretty excited about this. I know my excel workbooks can sometimes be cumbersome to deal with, so this would be an awesome way to better share my work.
Reply With Quote
  #13   Spotlight this post!  
Unread 08-16-2018, 09:26 AM
Billfred's Avatar
Billfred Billfred is offline
Never look back, no, never say die
AKA: Always attack with the warrior's cry...
FRC #1293 (Pandamaniacs); FTC #11444 (Garnet Squadron) (Also, CD mod.)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2004
Location: The Shadow Realm
Posts: 9,540
Billfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond reputeBillfred has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Eugene Fang View Post
I didn't really consider having a bunch of people casually vote on who they think will win until the idea was brought up in this thread. My original intended goal was to optimize for power users in terms of prediction submission and make viewing predictions accessible to everyone. That way, we can have high quality predictions (unlike the TBA predictions page I never had time to properly update for 2018) that are more accessible than Caleb's Event Simulator (which is awesome), while automating the spreadsheet-based prediction contests.
Is there a less-resource-intensive way to collect casual votes and present them as one "The Internet" prediction model? I don't know if that gives people all of their jollies, but it would be a good chunk of them. (There may not be such a way, I'm spitballing.)
__________________
William "Billfred" Leverette - Gamecock/Jessica Boucher victim/Owner/Director of Bricks 4 Kidz in Columbia, SC.

2004-2006: FRC 1293 (D5 Robotics) - Student, Mentor, Coach
2007-2009: FRC 1618 (Capital Robotics) - Mentor, Coach
2009-2013: FRC 2815 (Los Pollos Locos) - Mentor, Coach - Palmetto '09, Peachtree '11, Palmetto '11, Palmetto '12
2010: FRC 1398 (Keenan Robo-Raiders) - Mentor - Palmetto '10
2014-2016: FRC 4901 (Garnet Squadron) - Co-Founder and Head Bot Coach - Orlando '14, SCRIW '16
2017: FRC 5402 (Iron Kings) - Mentor, Coach (oh, and I worked at AndyMark during this time)
2018-: FRC 1293 (Pandamaniacs) - Mentor

113 events (more than will fit in a ChiefDelphi signature), 15 seasons, over 74,500 miles, and still on a mission from Bob.
Rule #1: Do not die. Rule #2: Be respectful. Rule #3: Be safe. Rule #4: Follow the handbook.
Reply With Quote
  #14   Spotlight this post!  
Unread 08-16-2018, 09:33 AM
Bryce2471's Avatar
Bryce2471 Bryce2471 is offline
Alumnus
AKA: Bryce Croucher
FRC #2471 (Team Mean Machine)
Team Role: Mentor
 
Join Date: Feb 2013
Rookie Year: 2006
Location: Camas, WA
Posts: 663
Bryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond reputeBryce2471 has a reputation beyond repute
Re: Need Your Input -- FRC Prediction Competition Website

Quote:
Originally Posted by Eugene Fang View Post
What stats should we show to rank how accurate predictions are? Since it’s not required to submit predictions for every event, we may need separate stats that rank by the whole season vs. single event.
I think for the win probability predictions you should use log loss. It penalizes over confidence a logical amount, when compared to something like error squared (Otherwise known as Brier scores).
Using 50% default predictions seems like a good idea. In the scope of a competition anyway.
I also think "matches correctly predicted" makes a lot of sense for W/L predictions and season long scores because it encourages people to predict more events.
__________________
2006-2011:
FLL Team Future imagineers
2011-????:
FRC Team Mean Machine

Last edited by Bryce2471 : 08-16-2018 at 10:03 AM.
Reply With Quote
  #15   Spotlight this post!  
Unread 08-16-2018, 09:45 AM
Lil' Lavery Lil' Lavery is offline
Hungry Dawgs Run Faster
AKA: Sean Lavery
FRC #1712 (DAWGMA)
Team Role: Mentor
 
Join Date: Nov 2003
Rookie Year: 2003
Location: Philadelphia, PA
Posts: 7,626
Lil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond reputeLil' Lavery has a reputation beyond repute
Send a message via AIM to Lil' Lavery
Re: Need Your Input -- FRC Prediction Competition Website

Would this have any integration with the predictions portion of TBA Blog?

Quote:
Originally Posted by IndySam View Post
you need to add "I really don't care"
That's called "not voting"
__________________
Being correct doesn't mean you don't have to explain yourself.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 09:18 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi