pic: Alliance Scores Over the 2011 FRC Season

[Ian Curtis]

Originally Posted by Chris is me

Ian, I think you're failing to account for the fact that FLR was an event that very quickly caught on to the 6v0 strategy. That would explain such a large scoring discrepancy.

Can you recommend a BAE-esque regional for replacement?

EDIT: Yup, Chris is right, other regionals in 2010 are significantly more skewed. I'll replace FLR with another regional tomorrow.

So, here is WPI then. It is skewed slightly, the mean and the median differ by .25. However, the OPR predicted distribution and the actual one match up quite well.

For this set the quartiles line up pretty well, with just a couple of outliers in the real world case. Chris, do you know if these were 6v0, or just exceptional performances?

[Chris is me]

Originally Posted by Ian Curtis

Can you recommend a BAE-esque regional for replacement?

WPI in Week 2 had similar depth to BAE and only one repeating team (20). Very little 6v0 was played (mostly by us but we had a reasonably accurate OPR anyway)

[Chris is me]

Originally Posted by Ian Curtis

For this set the quartiles line up pretty well, with just a couple of outliers in the real world case. Chris, do you know if these were 6v0, or just exceptional performances?

I don't want to say 6v0 because that implies 2791 contributed to the point spread, but one of the 10 point matches had us playing "helpful defense"...

I would say the rest of the high points could be attributed to 230 being allowed to run the field without interference, scoring points even without a ball magnet.

[IKE]

Your OPR Alliance score estimator will always create a more normal distribution than the actuals because it is using an average value, and not a scoring distribution. the minibot is a great example of this, and you 30/0/30/0 vs. 15/15/15/15 hits the nail right on the head. Both of these scenarios have the same average and thus would add to the OPR scoring algorithm the same way. From an Actuals though, the 30/0 would lead to 2 groupings which is more accurrate for the 2011 scoring.

The point to my comment was that the "average" or median robot does score significantly less than most people would estimate.

Our kids did an estimate on the VEX game this year. I think the max possible score was on the order of 60 pts. I then let them work on an estimate of what a good score would be. Initially they came up with a figure in the 40s. We did some refining techniques and their new estimate was much closer to 24-25 points. At their tournament this past weekend, that was pretty much exactly where the "good" scores came in. One alliance hit a 29 during a match.

For 2010, the average alliance score was around 4 points, but this was partially skewed by having the higher scorers contribute to more alliance scores because eliminations data was used as well (16 elims matches at FLR relative to 74 Qual matches with the best of the best playing in 50% of those elims matches). If you only use Qualification data, the average alliance score will be slightly lower than 3 pts. which means the "average" contribution would/should be just under 1 pt. The Median being slightly below this. To put this into perspective, if you started in the home zone, and just scored the 1 ball in the home zone every match, you would be better than 50% of the 2010 field. If you could hang (worth 2 pts.) 100% of the time, you would be over 2X the national "average". If you could put 1 ball in and hang, then you would make it to 3 pts. and be able to outscore about 50% of alliances, all by yourself. At an event like FLR, this would put you in the top 7 or so of teams. Top 7 and you are only pushing 1 ball in the goal, and hanging at the end... If your goal was to be an alliance captain or picked, targeting those easy 2-3 points is a very reasonable target. Notice the strategic difference though between these 3 points (which can be accomplished in the home zone) versus 3 points from a different zone. 3 points kicking balls means moving 3 balls into the home zone. Then moving the robot into the home zone, and then re-collecting and scoring the 3 balls. By my count this is a minimum of 7 actions to get 3 points (if you consider acquire, and then transfer seperate moves, it can be as many as 13 moves). Versus the original strategy which is 2-3 actions for 3 points...

For 2011, similar analysis shows the average score for an alliance being under 30 pts. It also showed that minibots were frequently not launched at all. Doing a post season analysis, If you simply had a good reliable minibot system, (not even a sub 2 second minibot) you would win most of your matches. At an absolute minimum, a scoring minibot was worth 10 points which was again more than the "average" contribution and well above the Median. Compare this to scoring tubes. Top row tubes are worth 3 points. 2x if you make a logo. If you hang an ubertube, its 6 in Auto, and up to an additional 6 points if you make a logo over it. In other words, in order to score 30 points in tubes, you would need to score an Ubertube in Autonomous, acquire and hang 3 different shaped tubes, in the right order (one of which would be difficult as you are hanging it over a ubertube). Again, this is 7 actions just to get to 30 points, versus essentially 2 actions for the minibot (align to tower and launch minibot). using the minibot minimum of 10 points, you would still need to score and uber tube and then acquire an hang another tube over it in order to beat the minibot minimum. If you don't have an autonomous, then you would have to hang a minimum of 3 different tubes top row creating a logo (6 actions) or 4 tubes top row not creating a logo (8 actions) just to beat the minimum minibot contribution...
************************************************** ******
This season:
1. Do a scoring analysis (all the way to get and block points, and then prioritize the way to get those points with the fewest distinct actions).
2. Do some field analysis. The best way to be playing in elims is to win qualifications and be an alliance captain. Be realistic on what a real alliance score will be. Understand that only about 25% of teams will get autonomous bonus points, and only about 25% of teams will hit most end game bonuses. Being able to get one of those bonuses every time will usually move you towards the top of the field.
3. Be realistic in your goals, and relentless in hitting them.

[Jim Zondag]

The attached Graph shows the distribution of individual team's season OPRs for the 2011 Season.
The trend you see here is pretty typical and is important when doing game analysis and strategy: Typcially about 25% of FRC population has a season contribution of 0 or less (26% in 2011). The 50% population point is much lower than you think. This has been the case in 2008, 2010, and 2011 since the GDC got "penalty happy". (2009 was an exception with only about 5% being negative, but the distributions are the same, just shifted to the right). The average scores per team increases quickly as teams play more events: Last year the OPR average by experience trend was 6.1, 18.0, 27.7, 34.4, 39.2, for 1-5 events played. Notice that it nearly triples going from 1 event to 2.
The performance distibution follows a roughly Gamma distirbution for all the teams and is very assymetrical. Last year 532 teams have net contribution at or below zero, while only 112 teams were at 30 or higher.
However, this function changes dramatically the more teams play.
If you can achieve half of the season maximum at your first event (OPR of 35 last year), you will be in the top 5% or so in the world at the beginning. If you keep this same level of performance, by your 3rd event you will only be barely above average relative to other teams with the same level of experience.

[Ian Curtis]

Jim, this is incredible! I hope lots of people get a chance to look at this and IKE's comments and realize that they are better off aiming low and hitting their goals than shooting for the moon and coming up way short. It also speaks volumes for having a robot done early so you can practice.

Since I assume 33 has pretty decent points per robot per match data, have you ever matched up the OPR with the actual points a robot is worth per match? Can you make any comments as to how good that fit is?

[Joe Ross]

Originally Posted by Ian Curtis

Jim, this is incredible! I hope lots of people get a chance to look at this and IKE's comments and realize that they are better off aiming low and hitting their goals than shooting for the moon and coming up way short. It also speaks volumes for having a robot done early so you can practice.

Since I assume 33 has pretty decent points per robot per match data, have you ever matched up the OPR with the actual points a robot is worth per match? Can you make any comments as to how good that fit is?

The fit changes every year based on how linear the scoring is and how dependent it is on your alliance partners. 2008 was the best year for OPR. Here's how our scouting matched with OPR in 2008. http://www.chiefdelphi.com/forums/sh...g#pos t732115

[IKE]

Originally Posted by Ian Curtis

Since I assume 33 has pretty decent points per robot per match data, have you ever matched up the OPR with the actual points a robot is worth per match? Can you make any comments as to how good that fit is?

We do a comparison at the events we are at to see how good of an indicator OPR is. We don't run stats on it, but mostly do visual checks to see if we are missing something or to look for trends. This probably would be a good thing to run stats on though...

Like Joe Ross said, 2008 correlated incredilby well. Especially if you used the top 2 offensive teams OPR. Unfortunately, negative OPRs also correlated well that year.*

2009 didn't work well at all. Over 50% of moonrocks were scored by humans, and many teams rotated the human player position.
2010 was pretty good. With good teams though, 2+2+2 =8 at MSC, the scores inflated drastically as teams could do zone play because there were good teams that could pick up the slack. 2010 also had some unique strategies that would underpredict certain teams. 67, 254, and 1114 were so good that even though OPR had them at 8-10, they would frequnetly score that many points, plus attempt a couple of points for the other teams! In reality, in a close match, those guys could do 10-12. OPR would often underpredict good alliances that year. Our qualifying match against 254 had an OPR predictor of I believe 16 to 12. The actual match turned out to be 20 to 18 (still one of my favorite FRC qualifying matches even though we lost).

2011 was interesting. OPR ws a reasonable predictor, but due to digressive scoring, it had the opposite effect as 2010 had. In 2011, 50+60+45 = 120-130 was not uncommon. There wasn't enough room for 3 good robots to put up a good score, and there were only 2 minibot poles.

************************************************** *****

*One partner in 2008 had a 6WD with omnis on the corners. Every lap the driver would inadvertently spin the wrong way when changes direction while doing a lap and would spin back over the line and get a penalty. I convinced a the team to zip-tie grip-mat to their omni wheels for a our match, and they only got 1 penalty that match.

Examples like this are where scouting can pay dividends. Often a team with a negative OPR is either breaking a rule or driving poorly. If you can observe their issue, and point it out to them, you can frequently get a few more points, ah er, not loose a few points that you likely would have without the comments. I was suprised that there were not more DQs in 2011 with all the red-card opportunities the GDC had in the rules.

[Jim Zondag]

Same data graphed by Percentage lets you see the trends a little better:
You can see how the population center moves to the right as experience increases
A little bit about the data set and this method. I have a database which has all the OPRs for all the team at each event they play, spanning many years. I take all of the OPRs and group them into catagories, this year it is in segments of 5 points per segement. I have used the "20 slices" method since 2006 to allow me to overlay data from several years worth of competition onto the same chart for analysis of multi year trends even though the games often have very different scoring systems.
Included in the 2011 data set:
Teams who played at least one event = 2053
Teams who played at least two events = 800
Teams who played at least 3 events = 244
Teams who played at least 4 events = 45
Beyond this, the popluations are too small to be relevant.

You can see from the chart some of the things IKE mentions: At 4 events, the teams are clearly limiting one anothers' total performance as indicated by the big peak at 40-45. With multiple robots of this caliber, the per team score actually goes down.