Chief Delphi - New FRC Stat: aWAR

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Rules/Strategy (http://www.chiefdelphi.com/forums/forumdisplay.php?f=6)

- - New FRC Stat: aWAR (http://www.chiefdelphi.com/forums/showthread.php?t=126992)

Re: New FRC Stat: aWAR

Finally got it. Turned out it was my school firewall blocking the download with no popup warning.

Changed to a different computer not behind the firewall and it worked. Although one more thing I discovered for anyone else who may be having trouble, the page did not want to open in Chrome but opened immediately in Explorer...how's that for weird? :]

Re: New FRC Stat: aWAR

Any chance of saving a copy as .xls (2003 era)? It's too big for Google Docs, and I won't have access to newer Excel until Monday :P

Re: New FRC Stat: aWAR

Quote:

Originally Posted by IKE (Post 1347716)

A correlation study would be interesting to dig into.

I did a quick linear regression using the aWAR data from 2008-2012 to predict aWAR in 2013. The R^2 was 0.50.

Code:

        Coefficients        Standard Error        t Stat                P-value

Intrcpt        0.3038132        0.052869809        5.746440231        1.23044E-08

2008        -0.000281711        0.032586229        -0.008645104        0.993104123

2009        0.070683871        0.031397538        2.251255225        0.02459942

2010        0.058408153        0.034406836        1.697574055        0.089918829

2011        0.250272535        0.035138483        7.12246277        2.10545E-12

2012        0.427316251        0.033320186        12.82454585        8.15505E-35

This shows that data from 5 years ago is not statistically significant, and 3-4 years old is minimally significant.

Re: New FRC Stat: aWAR

Quote:

Originally Posted by wilsonmw04 (Post 1347652)

going to go through it this weekend and see what shakes out. I find it humorous that the file that is 32 megs is called "slimdown" :-)

Thanks! I'd appreciate having more eyes on it! Yeah, the "slimdown" doesn't drop that much file size... it does drop the extra functionality to easily rank different events that Dan Niemitalo had made... I hadn't yet adapted all the formulas to work with aWAR so I dropped it for now. I would like to add it back in though, as it is really cool!

Quote:

Originally Posted by Joseph Smith (Post 1347705)

I noticed an interesting thing in the team lookup form, there are several teams (just from the few that I checked) where their Championship stats (OPR, rank, win/loss record, etc.) is a direct duplicate of their State/Regional Championship stats. For example, team 469, their Archimedes division stats are a direct duplicate of their MSC stats. The same is true for team 33, team 217, and team 245, but not for ALL MSC teams... I'm not sure if I see a pattern, but I'm really confused. Is this some element of the system that I'm not understanding?

Thanks for pointing out the bug... there's probably an issue with the formulas somewhere. Feel free to look into it yourself... I will later too tough.

Quote:

Originally Posted by I-DOG (Post 1347710)

Hands down, this is the coolest thing I've seen all season.

As someone who loves ranking everything, this rankings list is like candy to me.

Thanks! :-)

Quote:

Originally Posted by XaulZan11 (Post 1347712)

Unless I'm mistaken, couldn't the aWAR of 4 be a 4-8 team that won Chairmans award, though? I'd be curious to see the rankings based only on on-field preformance.

Yes, that could be... I don't think winning Chairmans would have *that* much of an impact... but I do agree that it would be interesting to have a robot-only version. That was my original goal but it got tabled... the intention was that it would be called rWAR, though. I'll add it in before long... it's an easy-enough addition.

Quote:

Originally Posted by IKE (Post 1347716)

I find the yearly ranking element interesting:
(32%, 29%, 23%, 16%).

On simialr efforts, using a fraction to the exponent of years produces a neat result:
For instance 1/2^year would be (1/2, 1/4, 1/8, 1/16 or 0.50, 0.25, 0.13, 0.06)... This weighting tends to favor last year's performance heavily with a quick roll-off on history. The neat thing about it is you can use all of history and still not hit 1.0. The bad thing about it is that past 4 years has very little impact.

Using (2/3)^year gets 0.667, 0.444, 0.296, 0.198 which normalized to a sum of 1 would be 0.417, 0.275, 0.185, 0.116. Using this algorithm tends to favor teams with longevity ans consistent high performance, but can also keep a team in the spot-light possibley a year or two after their prime if they have a very strong legacy.

A correlation study would be interesting to dig into.

Agreed that the correlation study is worthwhile... thanks Joe Ross!

Quote:

Originally Posted by rsegrest (Post 1347728)

It opened for me in Chrome... maybe not as slow though? Glad you ended up getting it downloaded!

Quote:

Originally Posted by Christopher149 (Post 1347743)

Any chance of saving a copy as .xls (2003 era)? It's too big for Google Docs, and I won't have access to newer Excel until Monday :P

I can do that... maybe tomorrow.

Quote:

Originally Posted by Joe Ross (Post 1347765)

I did a quick linear regression using the aWAR data from 2008-2012 to predict aWAR in 2013. The R^2 was 0.50.

Code:

        Coefficients        Standard Error        t Stat                P-value

Intrcpt        0.3038132        0.052869809        5.746440231        1.23044E-08

2008        -0.000281711        0.032586229        -0.008645104        0.993104123

2009        0.070683871        0.031397538        2.251255225        0.02459942

2010        0.058408153        0.034406836        1.697574055        0.089918829

2011        0.250272535        0.035138483        7.12246277        2.10545E-12

2012        0.427316251        0.033320186        12.82454585        8.15505E-35

This shows that data from 5 years ago is not statistically significant, and 3-4 years old is minimally significant.

This would probably argue for only doing the past three years... so maybe we should drop it to 3. I'd be curious if 2009 or 2010 are outliers though... given how much of a drop there is from 2011 to 2010. I have some more thoughts related to this correlation study... I'll comment more when I have time later.

Practice 'bot to finish... :-)

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Joe Ross (Post 1347765)

I did a quick linear regression using the aWAR data from 2008-2012 to predict aWAR in 2013. The R^2 was 0.50.

Code:

        Coefficients        Standard Error        t Stat                P-value

Intrcpt        0.3038132        0.052869809        5.746440231        1.23044E-08

2008        -0.000281711        0.032586229        -0.008645104        0.993104123

2009        0.070683871        0.031397538        2.251255225        0.02459942

2010        0.058408153        0.034406836        1.697574055        0.089918829

2011        0.250272535        0.035138483        7.12246277        2.10545E-12

2012        0.427316251        0.033320186        12.82454585        8.15505E-35

This shows that data from 5 years ago is not statistically significant, and 3-4 years old is minimally significant.

If I'm interpreting your P-value results correctly, 2008 and 2010 aWARs were really bad predictors of 2013 success, while 2009 aWARs were reasonably good? From the games, that's pretty much the opposite of what I'd expect. Or am I misinterpreting your stat program's outputs?

Re: New FRC Stat: aWAR

Something that really disappoints me about ranking statistics (including this one) is that they do not attempt to capture a team's improvement over the season.
A team that improves over the season should rank higher than a team that start their season off really well and then drops off. This stat really needs to have some sort of weighting system for doing well at harder regionals/districts (and the inverse for weaker events).
I also really want to see how removing non-competitive awards would affect the rankings.

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Orion.DeYoe (Post 1347825)

If you do OPR mapping of 1st event to second event to third event, you will find some really neat trends. For open scoring games (non-2011), the event to event increase is pretty impressive for the first 3 events for most teams with some "flat-lining" after that. Some of the teams that practice more will "flat-line" earlier.

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Nathan Streeter (Post 1347807)

Thanks for pointing out the bug... there's probably an issue with the formulas somewhere. Feel free to look into it yourself... I will later too tough.

It's a very simple problem. Most cells in the 5th row (/event) for each year has the same formulas as the 4th. You just need to shift the column references.

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Orion.DeYoe (Post 1347825)

some sort of weighting system for doing well at harder regionals/districts (and the inverse for weaker events).

Of course, then you'd need a way to identify stronger and weaker events...

Re: New FRC Stat: aWAR

Quote:

Originally Posted by DampRobot (Post 1347820)

2009 was statistically significant, but not necessarily what a normal person would call a good predictor on its own. It made up 5% of the prediction, which only predicted about 50% of the 2013 results.

Re: New FRC Stat: aWAR

I've been looking through the aWAR numbers and having a lot of fun checking it out! Thanks for putting this out there!

Quote:

Unless I'm mistaken, couldn't the aWAR of 4 be a 4-8 team that won Chairmans award, though? I'd be curious to see the rankings based only on on-field preformance.

You can actually do that with the spreadsheet. Go to the Point System tab and change all of the award values to zero (or any other value you like), then check the new totals in the aWAR tab. The Team Lookup tab isn't going to update, because it's looking at a table of static values, but the aWAR tab will have the numbers you want.

Speaking more broadly, you can also change the point system in various other ways in that tab if you're into playing with numbers.

Quote:

Any chance of saving a copy as .xls (2003 era)? It's too big for Google Docs, and I won't have access to newer Excel until Monday

The spreadsheet uses functions that aren't available in Excel 2003, so here are just the aWAR numbers for each team:
Attachment 16379

Quote:

I did a quick linear regression using the aWAR data from 2008-2012 to predict aWAR in 2013. The R^2 was 0.50.

Thanks for running those numbers! One of these summers I want to go more in depth into this type of thing.

Re: New FRC Stat: aWAR

One thing I don't really like about the weighted multi-year average is that teams of age < length of your stat rolloff are crippled because they have an aWAR of zero for the years before they existed.

In that case, I think their weighted average should not include the years they didn't exist. (this affects newer strong teams like 4001, 4334, 4814, 4451, 3990, etc)

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Racer26 (Post 1349071)

I tend to agree with this, but they also have the advantage of winning rookie all star and rookie inspiration, which will boast their numbers. Teams that have won Chairmans at the championship also are disadvantaged.

Re: New FRC Stat: aWAR

Quote:

Originally Posted by Racer26 (Post 1349071)

I'd love to figure out the best way to predict future competitive success based on the combination of years of experience, number of events per year, success in each event, consistency of success, awards, and OPR.

One gets into some tricky gray areas. For example, which team is likely to do better next year: the one that went quarterfinalist & finalist in two events, or the one that was a finalist in a single event? I've been thinking about how to organize all of the data to make it easier to study questions like that, but that's something that won't happen until the summer.

I tend to think that teams that playing in 2+ events correlates with better success, and that teams in their 3rd year or more will tend to perform better than rookies and second year teams. For that reason, I think it makes sense to give extra credit for the seasons that are 3 or 4 years ago, even if it's a small amount. But as to the exact amount it should be, I don't know. Needs to be studied.

Quote:

Teams that have won Chairmans at the championship also are disadvantaged.

On the contrary, I looked them all up and gave them 10 points in each year after they won the Chairman's Award. And one can adjust that figure to whatever value one wants in the Point System worksheet. I guessed that a Hall of Fame team is more likely to do well than a non HoF team in a given year, so I think it is a good adjustment. It's one more thing that would be interesting to study in a regression analysis.

Re: New FRC Stat: aWAR

Looking at the numbers for 857, aWAR looks pretty accurate at first blush:

2008: 0.17 - poor robot, driving was its best quality
2009: 1.2 - we were pretty good, got into elims at one event
2010: -0.1 - that was such a horrible bot *shudders*
2011: 0.7 - we may have been a #7 captain, but we were a bit mediocre
2012: 0.2 - poor robot, didn't really get anywhere until near the end of 2nd district event, didn't move several matches
2013: 1.46 - fantastic robot, semis and quarters and MSC, scored every match I think