View Full Version : Week3 cumulative Twitter stats & OPRs
Stats and OPRs based on Twitter Qual match data as of Fri 3-14-2014 22:30:53 ET.
The usual Twitter data caveats apply.
NOTE: LOOK FOR UPDATED DATA TO BE POSTED LATER IN THIS THREAD
Jim Zondag
14-03-2014, 23:53
Excerpt from the above post:
win foul points awarded:
20 average (mean)
220 max
lose foul points awarded:
5 average (mean)
110 max
The above is a very simple assessment of how the foul points affect this game. 4X as many foul points to the winning alliance....this means that fouls swing a large percentage of matches.
50 point foul is effectively a automatic loss most of the time.
Tragic.
Worst game design since 2003, yet no real action to correct....
Tragic.
Thad House
14-03-2014, 23:58
20 foul points on average for the winning alliance. That is insane. What makes it worse, is that by the look at average scores, they are not a ton different then last year, but the fouls are worth a ton more. I think the game designers expected higher average match scores to make up for higher foul scores.
Nathan Streeter
15-03-2014, 09:28
Excerpt from the above post:
win foul points awarded:
20 average (mean)
220 max
lose foul points awarded:
5 average (mean)
110 max
The above is a very simple assessment of how the foul points affect this game. 4X as many foul points to the winning alliance....this means that fouls swing a large percentage of matches.
50 point foul is effectively a automatic loss most of the time.
Tragic.
Worst game design since 2003, yet no real action to correct....
Tragic.
Agreed that the impact of foul points is tragic... and that foul points swinging the outcome of the match is likely a significant contributor to this statistic; however, I do think there is another possible/probable cause.
Fouls "take advantage" of less experienced, lower-performing teams. The teams that are more likely to get careless HP fouls, struggle to pickup a ball when inbounding, play hard defense that involves a manipulator flipping out and getting inside an offensive bot, or pin for too long are the teams that are less familiar with the rules, haven't kept up with the Q&A, and haven't designed as effective an offensive bot.
The fact that the teams most likely to suffer from the heavy fouls are also the teams that FIRST is having the hardest time retaining isn't a good thing for FIRST growing at or above average next year.
Consider me more and more on the side of "bring foul values down to 10 and 30." Keep in mind this is still 333% and 150% the value of fouls in 2013.
Big, Bad, Bobth Post! (#319)
Gaurav27
15-03-2014, 11:25
Excerpt from the above post:
win foul points awarded:
20 average (mean)
220 max
lose foul points awarded:
5 average (mean)
110 max
The above is a very simple assessment of how the foul points affect this game. 4X as many foul points to the winning alliance....this means that fouls swing a large percentage of matches.
50 point foul is effectively a automatic loss most of the time.
Tragic.
Worst game design since 2003, yet no real action to correct....
Tragic.
Agreed. Quite sad to see some really good strategies play out and then that one tech foul swings it the other way.
I think a 30 point value should be the max assigned to the tech foul. Hopefully, based on average win/loss margins it will have a smaller swing effect on match outcomes.
Twitter Qual Match Stats
based on Twitter Qual match data as of Sat 15 Mar 2014 20:13:09 ET.
The usual Twitter data caveats apply.
I've removed the Quartile data from the stats report. In their place I will be posting histograms later this evening. Also Twitter-based OPRs. So check back later.
Unpenalized Winning Margin Histogram
based on Twitter Qual match data as of Sat 15 Mar 2014 20:13:09 ET.
The usual Twitter data caveats apply.
OPR for Final, Foul, Auto, & TeleOp
based on Twitter Qual match data as of Sat 15 Mar 2014 20:56:33 ET.
The usual Twitter data caveats apply.
Data is shown for teams which have played at least 6 matches.
Very well presented, your picture (http://www.chiefdelphi.com/forums/attachment.php?attachmentid=16567&d=1394931119) is worth a thousand words! Would you mind if I combine your charts with my questionnaire results (https://docs.google.com/spreadsheet/pub?key=0ApW29QW2kbRcdDZfX2pMMXNSVUpYR2Y3c0VzbVBvc lE&single=true&gid=1&output=html) for submission to FIRST tomorrow?
The questionnaire can be found here (http://goo.gl/PbxyPf).
[would it be possible to calculate the percentage of games finished within +/- 50 points? i.e. a tech foul call or non-call would have swung the results of that game?]
The fact that the teams most likely to suffer from the heavy fouls are also the teams that FIRST is having the hardest time retaining isn't a good thing for FIRST growing at or above average next year.
^^^ This says it all. Thank you Nathan. ^^^
Very well presented, your picture (http://www.chiefdelphi.com/forums/attachment.php?attachmentid=16567&d=1394931119) is worth a thousand words! Would you mind if I combine your charts with my questionnaire results (https://docs.google.com/spreadsheet/pub?key=0ApW29QW2kbRcdDZfX2pMMXNSVUpYR2Y3c0VzbVBvc lE&single=true&gid=1&output=html) for submission to FIRST tomorrow?
You have my permission.
[would it be possible to calculate the percentage of games finished within +/- 50 points? i.e. a tech foul call or non-call would have swung the results of that game?]
See attached chart. Of the 3022 non-tied qual matches in the Twitter data dated 3/15/2014 20:13:09 ET, there were 1619 matches with final score (including awarded foul points) margin less than or equal to 50.
The usual Twitter data caveats apply.
Alliance Score Residuals
based on Twitter Qual match data as of Sat 15 Mar 2014 20:56:33 ET.
Example graph interpretation:
81 - 18 = 62% of Alliance Unpenalized Scores were within +/- 20 points of the "OPR" predicted value.
66 - 32 = 35% of Alliance Unpenalized Scores were within +/- 10 points of the "OPR" predicted value.
The usual Twitter data caveats apply.
Ian Curtis
16-03-2014, 13:01
50 point foul is effectively a automatic loss most of the time.
Tragic.
Worst game design since 2003, yet no real action to correct....
Tragic.
The fact that the teams most likely to suffer from the heavy fouls are also the teams that FIRST is having the hardest time retaining isn't a good thing for FIRST growing at or above average next year.
This can be a double whammy for District teams on the cusp of making it to the District Championship. The points for winning qualifying matches can function as the tie-breaker for determining who makes the cut-off and who doesn't. Your team can do everything right, but if a partner makes a mistake or doesn't know the rule, there goes 2 points. (And vice versa, you can do everything wrong and still get 2 points!) We do not have the manpower or experience to pull a 51 and record every foul and coach other teams through their problem areas. At our next event we will make a point to bring it up before every match so everyone knows.
At the end of the season I would be curious to see how many teams make or do not make the DCMP by foul points.
This can be a double whammy for District teams on the cusp of making it to the District Championship. The points for winning qualifying matches can function as the tie-breaker for determining who makes the cut-off and who doesn't. Your team can do everything right, but if a partner makes a mistake or doesn't know the rule, there goes 2 points. (And vice versa, you can do everything wrong and still get 2 points!) We do not have the manpower or experience to pull a 51 and record every foul and coach other teams through their problem areas. At our next event we will make a point to bring it up before every match so everyone knows.
At the end of the season I would be curious to see how many teams make or do not make the DCMP by foul points.
I agree that would be an interesting metric, although the match data doesn't show how deserving the fouls were or how consistently they were called.
As you are probably aware the twitter data (caveats apply) shows 1778 lost one match and won another at Mt Vernon due to fouls, so no net impact.
Ian Curtis
16-03-2014, 15:46
As you are probably aware the twitter data (caveats apply) shows 1778 lost one match and won another at Mt Vernon due to fouls, so no net impact.
Yep, and that make me wonder if they are uniformly enough distributed so that it is a net zero effect. I think the safest thing to do is assume that they are deserved. At Mr. Vernon I thought the refs did a great job.
Red wins 1935; Blue wins 1722; Ties 26
based on Twitter Qual and Elim match data as of Sun 16 Mar 2014 15:32:18 ET.
The usual Twitter data caveats apply.
Coach Norm
16-03-2014, 16:53
Red wins 1935; Blue wins 1722; Ties 26
based on Twitter Qual and Elim match data as of Sun 16 Mar 2014 15:32:18 ET.
The usual Twitter data caveats apply.
Does this include eliminations? If so, that would make sense.
Does this include eliminations? If so, that would make sense.
The long term trends have been that Qual performance for red is 50% and elimination is ~65%.
The long term trends have been that Qual performance for red is 50% and elimination is ~65%.
Elims only: red wins 415 (66.8%); blue wins 206 (33.2%); ties 3
Quals only: red wins 1520 (50.1%); blue wins 1516 (49.9%); ties 23
Twitter data 3/16 15:32:18
Joseph1825
16-03-2014, 19:59
I thought I understood what these numbers meant, but I'm confused, so could someone explain somthing to me? I'm with team 1825 and the spreadsheet says we have an OPR of over 80 points (the blue alliance agrees.) but our CCWM is -40, could someone explain what those number mean? Thanks.
I dont think you took out duplicate matches from replays.
I dont think you took out duplicate matches from replays.
Are you addressing Joseph1825 or me?
Are you addressing Joseph1825 or me?
You.
You.
Your post had no context, and it was linked to Joseph1825's post #20.
Could not tell to whom the pronoun "you" was referring.
I dont think you took out duplicate matches from replays.
"The usual Twitter data caveats apply (https://www.google.com/#q=Ether+caveats+Twitter+site%3Achiefdelphi.com)"
Twitter data is known to have omissions and redundancies and possibly a few errors.
"The usual Twitter data caveats apply (https://www.google.com/#q=Ether+caveats+Twitter+site%3Achiefdelphi.com)"
Twitter data is known to have omissions and redundancies and possibly a few errors.
Then this truly isnt the correct OPR calculations.
And your post before said I linked my other post to that Joseph's post, but I did so such thing.
your post before said I linked my other post to that Joseph's post, but I did so such thing.
Perhaps unknowingly, but yes you did (http://www.chiefdelphi.com/forums/attachment.php?attachmentid=16576&stc=1&d=1395016067).
You are reading that wrongly. That shows me posting after him. If you look to see that one of your posts comes all the way down from mine, that is one where you respond directly to me.
All I did was click Post Reply at the very bottom of the page.
Is the twitter data any different from the FRC-Spy (http://www.chiefdelphi.com/forums/frcspy.php?) data?
GearsOfFury
16-03-2014, 20:47
Unpenalized Winning Margin Histogram
based on Twitter Qual match data as of Sat 15 Mar 2014 20:13:09 ET.
The usual Twitter data caveats apply.
The bi-modal nature of this graph is really very surprising to me. I cannot think of why there would be such a discrete break around 0... What am I missing?
Great graph, in any case! Thank you for posting it.
The bi-modal nature of this graph is really very surprising to me. I cannot think of why there would be such a discrete break around 0... What am I missing?
It looked a bit strange to me at first, so I re-created the graph manually directly from the Twitter data and got the same result. I can post the spreadsheet later this evening if you're interested.
Is the twitter data any different from the FRC-Spy (http://www.chiefdelphi.com/forums/frcspy.php?) data?
Same thing.
You are reading that wrongly. That shows me posting after him.
No, I am reading it correctly. The threaded view is a hierarchy. The intent is to show which post you are responding to.
All I did was click Post Reply at the very bottom of the page.
That's the problem. When you do that, it links your post in the threaded view to whatever is the most recent post in the thread, even if that isn't the post you are responding to. That might be OK if you include context in your post like quoting a section of the post to which you are responding or use screen names instead of pronouns. Otherwise there is no way to tell who you are addressing.
If you look at the threaded view, you will see this post linked to your post in the hierarchy (http://www.chiefdelphi.com/forums/attachment.php?attachmentid=16580&stc=1&d=1395019608), even though your post is not the most recent one as I am typing this.
The way to get the links correct is to use the "Reply with quote" or "Quick reply to this message" buttons on the post to which you are responding (http://www.chiefdelphi.com/forums/attachment.php?attachmentid=16578&stc=1&d=1395019320).
Then this truly isnt the correct OPR calculations
I can assure you that the algorithm I use to compute the L2 norm of the simple linear combination of team scores (what we're calling OPR) is correct.
But that leaves two questions:
1) What is the "correct" data to use, and
2) Is the L2 norm of a linear combination of team scores the "correct" algorithm to get the most meaningful and useful metric?
Question 2 has been discussed in various threads here on CD in the past. I won't beat that horse here.
Question 1 is especially problematic this year because of the high value and erratic enforcement of fouls (I am not blaming the refs: this is a difficult game to ref and score). To get a truer measure of performance arguably requires that the foul points be removed from the score before computing the OPR. The problem is, you can't do this with the official data. You need to use the Twitter data to remove the foul points.
Ed Law maintains a spreadsheet in which he makes every effort to "repair" the Twitter data whenever possible and deciding when and when not to integrate it with the official FIRST Match Results and Team Standings data. It's apparently a labor of love for Ed and he spends many hours getting it right, for which we are in his debt. If I tried to do that, I'd probably introduce more errors than I corrected.
So if you're looking for the "official" OPR, rather than the "correct" OPR, I would say that Ed Law's spreadsheet is the de facto standard.
Mike Bortfeldt
16-03-2014, 21:58
The bi-modal nature of this graph is really very surprising to me. I cannot think of why there would be such a discrete break around 0... What am I missing?
Mark,
I suspect the break around zero occurs due to penalties. All the matches with negative win margins were decided by penalties points and the number of matches where this occurs is a small subset of all the data. All the data with positive win margins include matches with no penalties, offsetting penalties or matches where the penalties did not affect the final outcome.
Mike
I suspect the break around zero occurs due to penalties.
Correct.
All the matches with negative win margins were decided by penalties points...
Correct.
...and the number of matches where this occurs is a small subset of all the data.
Some would say "Not small enough."
All the data with positive win margins include matches with no penalties, offsetting penalties or matches where the penalties did not affect the final outcome.
Correct.
I think Mark's question was why the curve changes so abruptly at the Y axis. I've been trying to come up with a good intuitive way to explain it. Any takers?
I thought I understood what these numbers meant, but I'm confused, so could someone explain somthing to me? I'm with team 1825 and the spreadsheet says we have an OPR of over 80 points (the blue alliance agrees.) but our CCWM is -40, could someone explain what those number mean? Thanks.
I'll try. What spreadsheet are you referring to?
I thought I understood what these numbers meant, but I'm confused, so could someone explain somthing to me? I'm with team 1825 and the spreadsheet says we have an OPR of over 80 points (the blue alliance agrees.) but our CCWM is -40, could someone explain what those number mean? Thanks.
I believe the spreadsheet you are referring to is OPR based on Twitter data 3-14-2014 203053 ET revB.XLS.
The numbers are calculated correctly. However: 1) that spreadsheet was prepared from Twitter data from Friday night before you had finished quals (on Saturday), and 2) Twitter contained only 5 of the 8 matches you played on Friday. For such a small data sample size, the OPR may be misleading.
To answer your questions:
Column B is the "standard" OPR, based on final score (which includes awarded foul points).
Column E is unpenalized CCWM... "unpenalized" in this context meaning the awarded foul points are removed from the final score, and CCWM meaning the opposing alliance's score (with awarded foul points removed) is subtracted from your alliance's score for each match. Many folks think removing the foul points gives a better metric for this year's game because of the high value and erratic enforcement* of penalties.
Note: AIUI, Ed does not use the Twitter data to remove foul points if it is not complete for that event. So you won't see unpenalized CCWM for MOKC in his spreadsheet.
update:
4) Quite a few regional/districts this week have incomplete twitter data so adjusted OPRs cannot be calculated for those events..
* I am not faulting the refs. This is a very difficult game to ref and score.
I think Mark's question was why the curve changes so abruptly at the Y axis. I've been trying to come up with a good intuitive way to explain it. Any takers?
I thought Mike Bortfeldt explained that well enough.
We're not seeing a break so much as a graph representing two populations of different sizes.
Everything to the right of the Y-axis shows the fairly expected distribution of winning margins. This represents just under 90% of all matches.
The data to the left of the Y-axis shows a similar albeit reflected pattern. It is scaled down in frequency since it comes from the ~11% of matches that would have a different winner without the penalties.
There are some other effects due to penalties being larger and more quantized than the point value of scoring objectives, but the main cause is due to sub-population size differences.
GearsOfFury
17-03-2014, 06:30
It looked a bit strange to me at first, so I re-created the graph manually directly from the Twitter data and got the same result. I can post the spreadsheet later this evening if you're interested.
I would be interested, yes, thanks.
In the meantime, can you overplot this graph with the penalized version? And (separately) the histogram of penalties?
GearsOfFury
17-03-2014, 06:37
I thought Mike Bortfeldt explained that well enough.
We're not seeing a break so much as a graph representing two populations of different sizes.
Everything to the right of the Y-axis shows the fairly expected distribution of winning margins. This represents just under 90% of all matches.
The data to the left of the Y-axis shows a similar albeit reflected pattern. It is scaled down in frequency since it comes from the ~11% of matches that would have a different winner without the penalties.
There are some other effects due to penalties being larger and more quantized than the point value of scoring objectives, but the main cause is due to sub-population size differences.
The plot is showing frequency, not percentage. In a "normal" world, the winning margin would be normally distributed about some mean, and so would the penalties. When the penalties are applied, the winning margin would "shift" to the right - but there would not be a large, discrete jump at 0.
In our "less normal" world, we probably have a large skew to the left - the winning margin is far more likely to be small than large. But, I would expect it to still be more or less continuous (as you point out, there are quantization effects because of the scoring objectives... just as scores of say 4 or 5 in football are unlikely). I would expect the penalty point distribution to be continuous, as well, but with even larger gaps between likely values.
When the penalties are subtracted from the penalized score, I expect the resulting distribution to be continuous. The jump right at zero is not expected.
I'll withhold my tin foil hat theories as to why this is until I can take a look at Ester's raw data.
Mike Bortfeldt
17-03-2014, 07:36
...
Mike,
Thank you for a better explanation. Perhaps a different way to look this graph is to plot the data as two separate sets. The first being all matches where there were no foul points. The second with the matches that had foul points.
Mike
Twitter winning margin histograms
As requested, attached is an Excel XLS spreadsheet using non-tied qual matches from Twitter data 3/16 16:44:46. Included are histograms of frequency (counts) versus:
wm: winning margin (with awarded foul points included)
wmu: unpenalized winning margin (with foul points removed)
wfm: winning foul margin (winner's awarded foul points minus loser's awarded foul points)
All the necessary raw data is in the spreadsheet, as well as the derived data and the formulas used to compute it. You can play around with it to see if there's a better way to present it.
Other than removing ties, I made no further effort to modify the Twitter data.
The usual Twitter data caveats apply.
Andrew Schreiber
19-03-2014, 17:20
Ether, maybe I'm just blind as a bat (likely) but have you posted raw data for twitter? I'm trying to avoid going to the effort of scraping it if I can.
Ether, maybe I'm just blind as a bat (likely) but have you posted raw data for twitter? I'm trying to avoid going to the effort of scraping it if I can.
http://www.chiefdelphi.com/forums/showpost.php?p=1350317&postcount=11
Thank you, Brandon :)
Here's an XLS version with practice matches removed
List of 430 Qual & Elim Matches whose outcome was affected by fouls
From Twitter data 3/16/2014 16:44:46
The usual Twitter data caveats apply.
Alliance Score Residuals
based on Twitter Qual match data as of Sat 15 Mar 2014 20:56:33 ET.
Example graph interpretation:
81 - 18 = 62% of Alliance Unpenalized Scores were within +/- 20 points of the "OPR" predicted value.
66 - 32 = 35% of Alliance Unpenalized Scores were within +/- 10 points of the "OPR" predicted value.
The usual Twitter data caveats apply.
Attached is a cumulative percent histogram using updated Twitter data 16Mar2014 16:44:46
81.5 - 18.7 = ~63% of Alliance unpenalized scores were within +/-20 points of the "OPR predicted" (L2 norm) value.
List of 430 Qual & Elim Matches whose outcome was affected by fouls (http://www.chiefdelphi.com/forums/showpost.php?p=1361482&postcount=46)
Key:
rw: TRUE if red won the match, i.e. rfinal>bfinal
bw: TRUE if blue won the match, i.e. bfinal>rfinal
tie: TRUE if the final score was tied, i.e. rfinal==bfinal
ru: red final score with awarded foul points subtracted, i.e. rfinal-rfpts
bu: blue final score with awarded foul points subtracted, i.e bfinal-bfpts
~rw: TRUE if red would NOT have won if there were no foul points awarded, i.e. ru<=bu
~bw: TRUE if blue would NOT have won if there were no foul points awarded, i.e. bu<=ru
~tie: TRUE if the score would NOT be tied with foul points removed, i.e. ru!=bu
rwa: TRUE if Red's official win would have been altered by removing all foul points, i.e rw && ~rw
bwa: TRUE if Blue's official win would have been altered by removing all foul points, i.e bw && ~bw
ta: TRUE if the official tie would have been altered by removing all foul points, i.e tie && ~tie
fao: TRUE if match outcome was affected by foul points, i.e. rwa || bwa || ta
Team-by-team analysis of foul points on Win/Loss/Tie
based on Qual & Elim matches from Twitter data Sun 16 Mar 2014 16:44:46
Top "beneficiaries":
Team wil wit liw lit til tiw WLTnet
4500 6 12
1801 5 10
4779 4 1 9
56 4 8
453 5 1 8
470 4 8
701 5 1 8
1159 4 8
2388 4 8
3070 4 8
3245 4 8
4089 4 8
5065 3 1 7
171 3 6
537 3 6
900 4 1 6
919 4 1 6
1287 4 1 6
2411 3 6
2550 3 6
2811 3 6
3236 3 6
3562 3 6
3619 3 6
4048 3 6
4061 3 6
4114 3 6
4153 4 1 6
4216 3 6
4320 3 6
4375 3 6
4511 3 6
4690 3 6
4970 3 6
5149 3 6
5290 3 6
Top "victims":
Team wil wit liw lit til tiw WLTnet
245 5 -10
399 5 -10
4776 4 1 1 -10
599 4 1 -9
910 1 5 -9
360 4 -8
488 1 5 -8
1208 4 -8
4654 4 -8
4784 4 -8
5273 4 -8
1094 1 4 1 -7
1806 1 4 1 -7
2642 3 1 -7
116 3 -6
126 1 3 2 -6
144 3 -6
192 3 -6
219 3 -6
435 3 -6
494 3 -6
854 3 -6
955 1 4 -6
1164 3 -6
1302 2 5 -6
1318 3 -6
1477 3 -6
1619 3 -6
1714 3 -6
2073 1 4 -6
2994 3 -6
3063 3 -6
3250 3 -6
3936 1 4 -6
4090 3 -6
4206 3 -6
4245 3 -6
4683 3 -6
4794 3 -6
4815 3 -6
5159 3 -6
5215 3 -6
5252 3 -6
Complete list attached.
The usual Twitter data caveats apply.
http://www.chiefdelphi.com/forums/showpost.php?p=1350317&postcount=11
Thank you, Brandon :)
Here's an XLS version with practice matches removed
Has anyone located a lookup table with district event codes? and which region each district event is in?
Thanks!
Dan Hudnut, mentor
Team 885, Randolph Ctr., VT
Has anyone located a lookup table with district event codes? and which region each district event is in?
http://frclinks.frclinks.com/
Joe Ross
23-03-2014, 14:20
Has anyone located a lookup table with district event codes? and which region each district event is in?
Besides FRCLinks, you can download the Alliance Selection Spreadsheet (https://docs.google.com/spreadsheet/ccc?key=0ArVM96D1kMDzdE82Y2lCOW9yekJRc3EtUTZvZzZma 3c&usp=drive_web#gid=0) and do a lookup there. It might be more convenient then parsing a webpage. For district events, the begining of the name contains the area, ie MI, NE, PNW, or MAR.
Week4 Twitter data analyses available here (http://www.chiefdelphi.com/media/papers/2985). More to come later this evening.
Besides FRCLinks, you can download the Alliance Selection Spreadsheet (https://docs.google.com/spreadsheet/ccc?key=0ArVM96D1kMDzdE82Y2lCOW9yekJRc3EtUTZvZzZma 3c&usp=drive_web#gid=0) and do a lookup there. It might be more convenient then parsing a webpage. For district events, the begining of the name contains the area, ie MI, NE, PNW, or MAR.
Thank you both!
Has anyone located a lookup table with district event codes? and which region each district event is in?
Thanks!
Dan Hudnut, mentor
Team 885, Randolph Ctr., VT
If you need a historical one, I have one posted in CD-Media: Event Codes for FRC Events 2002-2014 (http://www.chiefdelphi.com/media/papers/2864)
vBulletin® v3.6.4, Copyright ©2000-2017, Jelsoft Enterprises Ltd.