OCCRA
Go to Post week 7: check to see if the pets are still alive, ask significant others to take us back, do 6 weeks worth of make-up homework, and ask my boss for another extension. - Tim Baird [more]
Home
Go Back   Chief Delphi > ChiefDelphi.com Website > Extra Discussion
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 05-11-2018, 07:11 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
paper: Miscellaneous Statistics Projects 2018

Thread created automatically to discuss a document in CD-Media.

Miscellaneous Statistics Projects 2018 by Caleb Sykes

This whitepaper is a continuation of my miscellaneous statistics projects whitepapers from last year. For those not familiar, here is a summary of why I do this:
I frequently work on small projects that I don't believe merit entire threads on their own, so I have decided to upload them here and make a post about them in an existing thread. I also generally want my whitepapers to have instructions sheets so that anyone can pick them up and understand them. However, I don't want to bother with this for my smaller projects.

I have decided to make a new thread this year in order to not overload my other thread with too many whitepapers, and because I will be analyzing 2018 specific things here. As always, feel free to provide feedback of any kind, including pointing out flaws in my data or my analysis.
Reply With Quote
  #2   Spotlight this post!  
Unread 05-11-2018, 07:24 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

My first book for this year is an investigation of what my Elo model might look like if I tried to incorporate non-WLT RPs. This idea was spawned by posts 41-44 in this thread. This workbook has identical data as my normal FRC Elo book for 2005-2015, but from 2016-2018, I make adjustments to incorporate the other ranking points. I was unable to find a nice data set to use for 2012 coop RPs, if someone knows of one, let me know and I might try to do this same analysis for that year. For each year in 2016-2018, there were two additional non-WLT ranking points available in each quals match. In 2016 and 2017, the tasks required to achieve these ranking points were also worth bonus points in playoff matches.

The concern that spawned this effort is that, in quals matches, many/most teams are not strictly trying to win, but rather are trying to maximize the number of ranking points they earn. Without some kind of RP correction, this means that teams who are good at earning these RPs might be under-rated by Elo, since they might be more likely to win matches if they werenít expending effort on the RPs. Additionally, since playoffs had different scoring structures than quals in 2016-2017, the teams that do well earning these RPs in quals will presumably be even more competitive in playoffs due to the bonuses.

My approach for this effort was to find the optimal value to assign to the qualification RPs, and to add this value to teamsí winning margins for the quals matches in which they achieve this RP. I wanted to find the optimal value for each of the six types of RPs between 2016-2018. Although there are other approaches to incorporating RP strength into an all-encompassing team rating, I always prefer to use methods which can be used to maximize predictive power over methods that donít, since I can justify why I chose the values I did over just taking guesses about how much different things are worth. There are a few different metrics I could have chosen to optimize, but I settled on overall playoff predictive power over the full period 2016-2018. I chose to optimize for playoff performance since in playoff matches teams are almost strictly just trying to maximize their winning margin (or win). This contrasts with quals matches where teams may have other considerations for the match, potentially including going for RPs or showing off so they are more likely to be selected. I also chose to maximize predictive power over the full 2016-2018 instead of each year separately since Elo ratings carry over some between years, so the optimal value for 2016 RPs when maximizing predictive power for 2016 alone will be a bit different than the optimal value when maximizing over all three years since the latter will look at how well the rating carries over between years.

Here were the optimal (+-20% or 1 point, whichever is greater) values I found for each of the 6 RPs, measured in units of their respective yearís points:
2016 Teleop Defenses Breached: 2
2016 Teleop Tower Captured: 8
2017 kPa Ranking Point Achieved: 80
2017 Rotor Ranking Point Achieved: 40
2018 Auto Quest Ranking Point: 7
2018 Face The Boss Ranking Point: 45
All of these values are positive, which indicates that on average teams that get these RPs in quals are more likely to do better in playoffs than similar teams who do not. You can see the effects of these adjustments by looking at the attached book and looking at the ďAdjusted Red winning marginĒ column. This value should be equal to the red score minus the blue score with additional additions/subtractions depending on the RPs both alliances received. For example, in 2018 Great Northern qm 31, blue wins 305 to 288, so redís unadjusted winning margin is -17. Red got the auto RP and blue got the climb RP in this match though, so after accounting for these, redís adjusted winning margin is -17+7-45=-55.

Here are my probably BS rationalizations of why these RPs have the values above:
2016 Teleop Defenses Breached: It really doesnít surprise me that this value is so low. Teams tended to deal with the defenses in quals in much the same way they dealt with them during playoffs. Although there was a 20 point bonus in playoffs for the breach, any alliance worth their salt was going to get this anyway, so a team that got this RP consistently in quals wasnít set up to do that much better in playoffs than a similar team who got this RP less consistently.
2016 Teleop Tower Captured: I donít want to analyze this RP too much since its definition changed for championships, an event where teams were getting this RP much more frequently than a standard regional/district. I wouldnít have expected this value to exceed 10, since it generally took at least a pair of competent scorers to get 8 or 10 balls, and the 20 point playoff bonus divided by 2 is 10. I donít think teams would have played much differently in quals if this RP had not existed, except maybe being more conservative in the last few 30 seconds to make sure everyone surrounded the tower.
2017 kPa Ranking Point Achieved: This is by far the RP that had the most value. There are a couple of reasons I think it is so high. To start, there was a 20 point playoff bonus for this task that was unavailable in quals, and unlike the teleop tower captured in 2016, getting this RP was generally an individual effort, so a team that gets this RP consistently in quals should be worth at least 20 points more in playoffs than a similar team that does not. On top of this, because there were so few ways to score additional points in playoffs, the 40-70 fuel points scored in playoffs are in a sense more valuable than the points scored with other methods. There were diminishing returns on gear scoring after getting the third rotor, and no value at all in scoring gears after the fourth rotor, and thereís not much teams could do to get more climbing points except potentially lining up a bit earlier to avoid mistakes. Fuel points though were unbounded, so a team that consistently got the kPa RP in quals was going to be so much better off in playoffs just because they could get 60-90 points that were unachievable for a non-fuel opposing alliance.
2017 Rotor Ranking Point Achieved: Similar to 2016 teleop tower captured, I think most of the value of this RP comes from the playoff bonus of 100 points. This task required at least two competent robots to perform, which means I would have expected the value of this RP to be bounded above by 50. I donít think the strategy changed much in playoffs due to this RP, since the goal of 40 points + RP in quals is comparatively lucrative to 140 points in playoffs.
2018 Auto Quest Ranking Point: I expected this RP to be worth around 5 points and I was correct. Teams likely opted for higher risk and higher average reward autonomous modes in playoffs than they did in quals because they could afford to have one robot miss out on the crossing or be okay with not getting the switch if they could get one more cube on the scale. This wasnít a huge effect but it does exist.
2018 Face The Boss Ranking Point: I expected the value of this RP to be around 20 points because there is no playoff bonus for this task and I didnít think the opportunity cost was particularly high, although certainly higher than the auto RP. This was the value that most surprised me at 45 points. In my original analysis, I was thinking of the opportunity cost of going for the climb RP, not the extra value of a team implied by said team achieving the climb RP. I think the distinction is important because relatively few teams were able to consistently achieve the climb RP, and the teams that did so were generally very competitive teams. This means that in the playoffs they can afford to spend a few more seconds scoring elsewhere in the field before going for the climb, and can climb much faster on average than teams that were not consistently getting the climb RP in quals. If I had thought about it more from this perspective, I might have predicted this RP to be worth around 30 points instead of 20. The remaining 15 still surprises me though, one possible explanation is that this value is over-rated since we havenít had the 2019 season yet, so the model doesnít properly account for teamsí future success.


Overall, this was an interesting analysis, but I will almost certainly not be incorporating a change like this into my Elo ratings moving forward for the following reasons:
The adjustments made here do not provide enough predictive power for me to consider them worthwhile. These adjustments improved the Brier score for playoff matches in 2016-2018 by about 0.001. I would have needed it to be at least 0.003 to consider it worthwhile, since I am reasonably sure there exist other improvements to my model which can provide this much or more improvement.
We have no guarantee that future games will have similar RP incentives. I try hard to keep the number of assumptions in my model to a minimum. I do this because I want my model to be valuable even when we get thrown a curveball for some aspect of the game like we did this year for time-dependent scoring. Assuming we will continue getting games with this RP structure is just not a very good assumption in my opinion.
There isnít a clear way to find good values to use for the RPs during the season in some years. I am back-fitting data right now so I have a good sample size of quals matches where teams get the RPs. However, if we get a game like 2017 again, where we didnít get above a 2% success rate for either RP until week 4, there just wouldnít be a good sample size of matches to use to find good values until late in the season.
Reply With Quote
  #3   Spotlight this post!  
Unread 05-14-2018, 07:31 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

Continuing with my investigation I did last year of autonomous mobility. I thought it would be interesting to look at auto mobility rates for every year since 2016. I have attached a book titled "2016-2018_successful_auto_movement" which provides a summary of this investigation. For each team that competed in 2018, it shows their matches played, successful mobilities, and success rates for each year 2016-2018. It also contains these metrics in aggregate over all of these years as well as a reference to the first match in which the team missed auto mobility. I counted both "Crossed" and "Reached" in 2016 as successful mobilities.

Note that this is using data provided by the TBA api, which pulls directly from FIRST. So there are certainly some matches where teams are incorrectly assigned auto mobility or not. There are many possible reasons for this, but one of the ones I identified last year was that referees at some events were entering mobilities based on team positions and not team numbers.

Here's a fun graph of 2017 auto mobility rates versus 2018 auto mobility rates:


Here are the teams that have competed each year 2016-2018, and have never missed auto mobility points according to this dataset:
Code:
team	matches
1506	149
5554	112
4550	86
4050	77
5031	69
3061	61
6175	59
6026	55
1178	48
3293	45
4462	39
6054	36
6167	35
5119	35
6155	35
5508	34
884	32
3511	32
4630	30
4728	29
6164	28
1230	28
2264	28
4054	27
4648	26
5171	26
The only other per-team metric available is climb rates, so I might do an investigation of that in the future. Climbing was wildly different each year though, whereas just moving forward a few feet is essentially the same each year, so it wouldn't be as easy of a comparison.
Reply With Quote
  #4   Spotlight this post!  
Unread 06-08-2018, 07:39 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

I'm looking to make predicted schedules soon to use for a couple of projects. I would like the capability to do this even before the total number of matches at the event is known. With this in mind, I have attached a simple book which looks at, for every 2018 event, the number of teams at the event versus the number of qual matches/team.

Here is a plot for all events:


And here is a plot for regional events:


I'll likely just set district events (including district champs) to 12 matches/team, champs events to 10 matches/team (although I may change this depending on the structure of champs in future years and the game), and regional events matches according to the formula matches/team = 17.0-(0.13*teams).
Reply With Quote
  #5   Spotlight this post!  
Unread 07-15-2018, 04:01 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

I just uploaded a workbook called "2018_schedule_strengths_v1".

I'm planning to make a new thread soon to discuss "strength of schedule" in FRC, so I made this book to hopefully inform that discussion. I labeled it v1 because I imagine I'll need to go back and calculate other metrics as the discussion in my upcoming thread progresses.

Essentially, all I did was run my event simulator at each event twice, once before the schedule was released and once after. By looking at each team's ranking distribution change in this time period, we can pinpoint what effect exactly the schedule had. At least that's the idea anyway. I have some summary statistics of each team's ranking predictions for both of these periods, as well as the changes between them included.

Additionally, I have what is my first pass at a "strength of schedule" metric. I calculate this by finding the probability that the given team will seed better with the actual schedule than they would have with a random schedule. So a "schedule strength" of 0% means that you will never seed higher with the existing schedule than you would have with a random schedule, and a "schedule strength" of 100% means that you are guaranteed to seed higher with the actual schedule than you would have with a random schedule.

What I like about this metric:
It compares the given schedule against other hypothetical schedules
It is customized for each team, that is, it compares your hypothetical results with a random schedule with your hypothetical results with the given schedule. I'm not the biggest fan of team-independent metrics since, for example, a schedule full of buddy climb capable partners is amazing for a team without a buddy climber, but just alright for a team that has a good buddy climber, and team-independent metrics would have to give the schedule a single score for both of these teams.
It's on an interpretable scale (0% to 100%) and has meaningful significance
It's able to be calculated before the event occurs (I don't like metrics that require hindsight unless maybe we want to use SoS as a tiebreaker for something)

What I don't like about this metric:
Requires a full event simulator to calculate
Teams that are basically guaranteed to seed first (like 1678 at their later regionals) will inevitably be shown to have bad schedules, since there is no schedule that would give them much of a better chance of seeding higher than their expectation (1st). Switching to greater than or equal ranks just flips the problem to high scores instead of low scores for these scenarios
Average value is 48.1% instead of 50%


Anyway, feel free to use this as proof of how bad your schedule was. The worst schedule this year according to my metric were (excluding the expected 1 seeds):
2096 on Hopper
4065 at Orlando
6459 on Roebling

And the best schedules were:
2220 on Archimedes
5104 on Newton
1806 on Turing

Last edited by Caleb Sykes : 07-15-2018 at 04:05 PM.
Reply With Quote
  #6   Spotlight this post!  
Unread 07-15-2018, 06:38 PM
AriMB's Avatar
AriMB AriMB is offline
The Philadelphian emigrant
AKA: Ari Meles-Braverman
FRC #5987 (Galaxia)
Team Role: Mentor
 
Join Date: Mar 2015
Rookie Year: 2012
Location: Haifa, Israel
Posts: 1,667
AriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond reputeAriMB has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

Quote:
Originally Posted by Caleb Sykes View Post
I just uploaded a workbook called "2018_schedule_strengths_v1".

I'm planning to make a new thread soon to discuss "strength of schedule" in FRC, so I made this book to hopefully inform that discussion. I labeled it v1 because I imagine I'll need to go back and calculate other metrics as the discussion in my upcoming thread progresses.

Essentially, all I did was run my event simulator at each event twice, once before the schedule was released and once after. By looking at each team's ranking distribution change in this time period, we can pinpoint what effect exactly the schedule had. At least that's the idea anyway. I have some summary statistics of each team's ranking predictions for both of these periods, as well as the changes between them included.

Additionally, I have what is my first pass at a "strength of schedule" metric. I calculate this by finding the probability that the given team will seed better with the actual schedule than they would have with a random schedule. So a "schedule strength" of 0% means that you will never seed higher with the existing schedule than you would have with a random schedule, and a "schedule strength" of 100% means that you are guaranteed to seed higher with the actual schedule than you would have with a random schedule.

What I like about this metric:
It compares the given schedule against other hypothetical schedules
It is customized for each team, that is, it compares your hypothetical results with a random schedule with your hypothetical results with the given schedule. I'm not the biggest fan of team-independent metrics since, for example, a schedule full of buddy climb capable partners is amazing for a team without a buddy climber, but just alright for a team that has a good buddy climber, and team-independent metrics would have to give the schedule a single score for both of these teams.
It's on an interpretable scale (0% to 100%) and has meaningful significance
It's able to be calculated before the event occurs (I don't like metrics that require hindsight unless maybe we want to use SoS as a tiebreaker for something)

What I don't like about this metric:
Requires a full event simulator to calculate
Teams that are basically guaranteed to seed first (like 1678 at their later regionals) will inevitably be shown to have bad schedules, since there is no schedule that would give them much of a better chance of seeding higher than their expectation (1st). Switching to greater than or equal ranks just flips the problem to high scores instead of low scores for these scenarios
Average value is 48.1% instead of 50%


Anyway, feel free to use this as proof of how bad your schedule was. The worst schedule this year according to my metric were (excluding the expected 1 seeds):
2096 on Hopper
4065 at Orlando
6459 on Roebling

And the best schedules were:
2220 on Archimedes
5104 on Newton
1806 on Turing
How do you calculate the schedule strength metric? I'm assuming it's at least partially based on the changes due to schedule numbers, but I don't see any direct correlation (team A having a larger positive change in average rank than team B does not seem to imply that team A's schedule strength will be lower than team B's, or vice versa).
__________________
Studying MechE at the Technion - Israel Institute of Technology
2017-present: FIRST Israel CSA/FTAA
2017-present: FRC 5987 Technical Mentor 18isr2 18isr4 18isrcmp 18carv
2012-2016: FRC 423 Member 15njtab 15padre 16paphi
Reply With Quote
  #7   Spotlight this post!  
Unread 07-15-2018, 10:24 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

Quote:
Originally Posted by AriMB View Post
How do you calculate the schedule strength metric? I'm assuming it's at least partially based on the changes due to schedule numbers, but I don't see any direct correlation (team A having a larger positive change in average rank than team B does not seem to imply that team A's schedule strength will be lower than team B's, or vice versa).
The schedule strength is the probability that the given team will seed higher with the actual schedule than they would have with a random schedule according to the simulator. This is found by the following formula:



Where r and q are ranks and are summed over all ranks and all ranks greater than r respectively. Changing the second summation to over all q>=r would provide a very similar metric, just that it would err on the high side instead of the low side.

For example, say that before the schedule is released a team is predicted to have a 20% chance of seeding first, 30% second, and 50% third. After the schedule is released, they have a 5% chance of seeding first, 25% second, and 70% third. Their schedule strength would then be:
0.05*(0.3+0.5)+0.25*(0.5)=0.04+0.125=0.165 or 16.5%.



Looks like a pretty strong correlation to me. Excepting the teams which are heavy favorites to seed first it seems to be doing it's job.
Reply With Quote
  #8   Spotlight this post!  
Unread 07-25-2018, 10:38 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

As I'm delving more into schedule analysis, I decided I should probably do a proper comparison of the Cheesy Arena schedules, which is what I have been using instead of generating my own schedules, and the actual schedules used this year, generated according to the IdleLoop algorithm at each event. I am unfamiliar with exactly how this algorithm is run at each event, but I have heard that the scorekeeper generally will select a minimum match gap based on a number of factors, and then run the algorithm at the "best" (5 million attempts) setting. How exactly the minimum gap is selected is unknown to me, and likely varies across events.

For all 174 events which had qual matches, I compared the actual schedule for the event with the cheesy arena schedule for the same number of teams and the same number of matches per team. The metrics I was looking at to determine how good the schedule were:
number of surrogates used
red/blue balance for each team
gap between matches for all consecutive pairs of matches for all teams
repeated partners for all teams
repeated opponents for all teams

In some cases, these criteria will interfere with each other, most notably, a large minimum match gap will tend to cause more duplicate partners and opponents than a small minimum match gap. I also made the decision to treat surrogates the same as normal teams. Although for some metrics it might make sense to ignore surrogates, it seems to me that, if you were up against 254 in 4 different matches, it would be of little consolation to you if 254 was a surrogate in one of them, so I treated surrogates as normal teams.

The results show that overall, the cheesy arena schedules and the IdleLoop schedules are very similar. There were 11 events this season for which the Cheesy Arena schedules were as good or better than the IdleLoop schedules across all categories according to my metrics, and 6 events for which the IdleLoop schedules were as good or better than the Cheesy Arena schedules across all metrics. Here is a breakdown by category:
Surrogates: The number of surrogates used was identical for all events, this isn't surprising since surrogate usage should be bounded above by 6 for any combination of teams and matches/team.
Red/blue balance: There were 80 events for which the red/blue balance was better achieved by the cheesy arena schedules than the Idle Loop schedules, and 64 for the reverse.
Match gap: There were 85 events for which the match gap was better in the Cheesy Arena schedules than the Idle Loop schedules, and 89 for the reverse.
Duplicate partners: There were 14 events for which the Cheesy Arena schedules were better at avoiding duplicated partners than the Idle Loop schedules, and 9 for the reverse.
Duplicate opponents: There were 70 events for which the Cheesy Arena schedules were better at avoiding duplicated opponents than the Idle Loop schedules, and 88 for the reverse.

Overall, it seems that the Cheesy Arena schedules might be slightly better at red/blue balance and avoiding duplicate partners, whereas the IdleLoop schedules might be slightly better at having larger match gaps and avoiding duplicate opponents, although the sample sizes might be too small to draw too much of a conclusion.

The full results can be seen in my IdleLoop_and_Cheesy_Arena_comparison.xlsx file. The "summary" page shows a quick look at which schedule was "better" according to my metric for each category. The "raw data" sheet shows a more in depth look at each event. I will walk through the results for the Great Northern Regional here in order to explain them. You can stop reading now if you're not interested in a deeper dive into the data:
Columns 2-5 show information about the event. Columns 8-36 show data on the actual (Idle Loop) schedule used for this event. Columns 58-86 show data on the Cheesy schedule with the specified number of teams and matches/team. Columns 108-136 show the difference between the actual schedule and the cheesy schedule.
Starting in column 8, we see that the actual schedule had 5 surrogates used, and in column 58 we see the cheesy schedule also had 5 surrogates used. The difference between these in column 108 is 0.
In columns 9-17, we see the red/blue balance for teams in the actual schedule. There are 0 teams at this event that have an even number of red/blue matches, which is unsurprising given that each team had 11 matches. There are 40 teams that had either 1 more red matches than blue matches or 1 more blue matches than red matches, which are as balanced as you can get with 11 matches. There were 5 teams (the surrogate teams) that had a red/blue difference of 2 matches. Finally, there were 2 teams that had a 3 match red/blue difference. The cheesy schedule is similar, except that 4 of the surrogate teams had 0 red/blue difference (columns 59-67), this difference can be seen in cols 109-117, with the +4 value in the 2 match difference indicating that the actual schedule had 4 more instances of 2 match red/blue difference than the cheesy schedule did. Since the worst red/blue balance is worse for the actual schedule than the cheesy schedule, I mark in the "summary" sheet that the cheesy arena schedule is better for red/blue balance.
Moving on in a similar way to the next category of gap between matches, we see that with the actual schedule, no teams had back-to-back matches or consecutive matches with only a single match between their matches. There were however 38 occurrences of teams that had consecutive matches that were 3 matches apart. In the cheesy schedule though, there were only 34 of these occurrences. This gives the cheesy schedule the edge in "match gap" in my summary.
For pairs of partners, both schedules had 1044 occurrences of teams having another partner only once. There were no duplicated partners in either schedule. 1044 is to be expected since there are 87 matches, 2 alliances per match, 3 teams per alliance, and 2 partners per team, and 87*2*3*2 = 1044. Neither schedule is superior by this metric.
Finally, looking at opponent pair occurrences, we see that the actual schedule had 2 instances where a team had to face another team in 3 separate matches (2500 facing 7257 and 7257 facing 2500). In contrast, the cheesy schedule has 6 such occurrences. This gives the Idle Loop schedule the edge in my summary sheet.

Anyway, let me know if you see any flaws in my analysis or have any questions.
Reply With Quote
  #9   Spotlight this post!  
Unread 07-28-2018, 01:07 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

In response to requests for more model validation, I decided to pull all of the top 4, top 8, and top 12 predictions my current simulator would have made on the week 3-4 events at 4 different points at the event, before the schedule was released, after the schedule was released, after 1/3 of qual matches had been played, and after 2/3 of qual matches had been played. The results can be seen in my "week3-4_calibration_results.xlsx" book. This book contains the raw data, a sheet that has calibration results, and a sheet of a bunch of graphs showing these results.

I'm still toying around with how best to represent the calibration results. I've done this format in the past, but this is limited by arbitrary bin sizes and inability to show how many of a type of prediction I make. One idea I had was to use my normal calibration curve format, but with smaller bins, and with dot sizes corresponding to the number of points that fall into the bin. This way, people can see how many of a type of prediction I make, as well as how well calibrated those predictions are. Here's one of these graphs:


Although this is a pretty cool format to look at, there's not much actionable information I can pull out of it. So I also made simple scatter plots of the predicted probabilities versus actual results, here's one of those graphs:


This isn't very visually intuitive, but the linear regression tells me both how much predictive power I have (according to the R^2 value) and how well calibrated my model is (looking at the slope of the line). In the shown graph, the R^2 value of 0.30 indicates that my model can explain about 30% of the variance in the top 12 seeds after the schedule is released, but before any matches have been played. The slope of 0.82 indicates that, if I wanted to be well calibrated for top 12 predictions, I would need to mean-revert all of the top 12 predictions at this point in the event by about 18%.

The results are pretty much what I would have expected. I'm aware that my model has a small but appreciable overconfidence problem, particularly early in the event, you can see though from the graphs that the calibration gets better and better as the event goes on. The main spots in my model that I need to inject more uncertainty are:
The non-WLT RPs, I threw together the predictions for these really quickly this year and didn't get around to any proper calibration of them, so I expect them to be pretty overconfident
The second and third order RP sorts. I don't have any uncertainty in these predictions at all, they are completely deterministic (that is, no variation from one simulation to another).
Simulations running "cold" instead of "hot". I don't have any accounting for the possibility that a team will consistently perform better or worse than their metrics would suggest. This is fine near the end of the event, since all of the team's skill levels are pretty well known, but early on in the event this is a poorer assumption.

I could throw a blanket mean-reversion on my predictions to fix the over-confidence, but I'd prefer to fix the above problems first and the over-confidence should largely be taken care of as a result of the added uncertainty I introduce with the changes.

Not really sure how to respond to accusations that I personally am overconfident, since my model isn't particularly overconfident in a statistical sense. Any major uncertainties I have are either already introduced into my model, or listed above and planned to be improved upon in the future. I don't see my posting of my predictions as any different from predicting a coin flip has a 50% chance of heads. Sure, if you knew more properties of the coin, the surrounding air, the person flipping the coin, and the local gravitational field, you could make a prediction that is better than the 50% prediction, but that doesn't mean the 50% prediction is bad. It's just a well calibrated prediction that recognizes what it knows and doesn't know. That's all I try to do with my predictions, to maximize predictive power given the known bounds of the metrics available to me. I'm nowhere remotely close to predicting everything perfectly, but I'm doing the best I can with the information I have available.
Reply With Quote
  #10   Spotlight this post!  
Unread 08-06-2018, 11:28 AM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

I'm going to try soon to add some normalization between years for my Elo ratings. Presently, I find start of season Elos by taking 70% of the previous season's Elo + 30% of the end of season Elo from 2 seasons ago. I then revert this sum toward 1550 by 20%. My concern with this method is that I don't think it's fair to directly sum Elos from different seasons since the Elo distributions vary so greatly year to year based on the game. If we had the same game every year, this wouldn't be a problem.

To start, I measured the average, stdev, skew, and kurtosis for the end of season Elo distributions in each year. The results are shown in this table:


The average hovers right around 1500 each year, but this is due to how I designed my Elo ratings, and doesn't actually tell us much. Some actual measure of "true skill" would probably have higher averages in recent years, since most would agree the average robot in 2018 is much better than the average robot was in 2002.
stdevs move around each year, likely due to the game structure. 2018 had the highest stdev on record by a pretty solid margin. I have previously speculated that this could be due to the snowballing scoring structure of powerup.
The skewness is interesting, for those of you unfamiliar with skewness, a positive skewness indicates a larger positive "tail" on the distribution than the negative "tail". Every year on record has had a positive skew, which indicates that there are always more "outlier" good teams than "outlier" bad teams. Some years have had much higher skews than others though. For example, 2015 had an incredibly positive skew, which means there were a large number of very dominant teams. 2017 in contrast had one of the smallest skews on record. This is probably due to the severely limited scoring opportunities for the strong teams after the climb and 3 rotors, as well as the fact that the teams that lacked climbing ability were a severe hindrance to their alliances. The difference in skews between 2015 and 2017 can be seen in histograms of their Elo distributions. Notice how much longer the 2015 positive tail is than the 2017 one.



I also threw in kurtosis, kurtosis is a rough measure of how "outlier-y" or "taily" a distribution is. Kurtosis tracks very closely with skew every year. This means that the "outlier" teams driving the high kurtosis in some years are "good team" outliers and not "bad team" outliers. A high kurtosis with low skew would indicate that there are lots of good team and bad team outliers. Plots of stdev vs skew and skew vs kurtosis can be seen below.



Next, I'll be trying to normalize end of season Elos so that I can get better start of season Elos. We've now had two years in a row of games that have low skew/kurtosis, which means that without adjustment the 2019 start of season Elos will also have low skew/kurtosis even though the 2019 game likely will not. It'll all come down to predictive power though, if I can get enough predictive power increase I'll add it in, otherwise I won't.
Reply With Quote
  #11   Spotlight this post!  
Unread 08-11-2018, 04:33 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

I'm currently working on analyzing the awesome timeseries data from TBA. I'll have plenty more to come, but I'm at a point where I got some really sweet graphs, so I thought I'd share, and describe a rough outline of my live model at the same time.

I am currently analyzing the ~1500 matches that have the best timeseries data. It's possible that I'll go back later and clean up the messier data, but I wanted to focus my early analysis on data I could have high trust in. What I'm currently working on is a way to predict the match winner in real-time based on this data. Here is a Brier score graph of my current model:


The first five seconds of the match just use my pre-match Elo win probability, but from then on, I begin incorporating the real-time scoring (in conjunction with the pre-match Elo prediction) to create win probabilities. The Brier score is basically steady for the first 5 seconds (when I'm not incorporating match data) but also from ~19 to ~23 seconds, which is probably because teams have by this point scored their first set of cubes and are picking up the second set. Also, note that even at t = 150 seconds, the Brier score is not zero because the actual final score can differ from the last score shown on the screen.

I mentioned that I incorporate Elo ratings into the predictions, here is a graph showing how much weight I give to Elo versus live match data at each second:


This and the following graphs you will see were created by tuning my prediction model, so the values that you see were the most predictive ones I found. After the first 5 seconds, the importance of Elo drops sharply down to ~65% by the end of auto, where it holds roughly steady for the same 19-23 second interval described above. This makes sense, since if there isn't much scoring, we wouldn't expect the live scoring to increase in importance much. After that, the Elo weight decays roughly exponentially down to 0.

The general form of my model (excluding Elo) is red win probability = 1-1/(1+10^((current red winning margin)/scale)) where "scale" is how much of a lead red would need to have a 10/11 ~=91% chance of winning the match at that point in the match. Let's call this "scale" the "big lead" amount so as not to be confused with the scale on the field. If a team is up by 40 at a point in the match where the "big lead" value is 40, that team has a 91% chance of winning, but if they are up by 80 (two big leads), that team has a 99% chance of winning. Obviously, what is considered a big lead will vary over the match, so here is a graph showing that change over time:


I excluded the first 5 seconds since the values there are indeterminate since I don't incorporate match data then. The next few seconds of auto are also a bit weird, probably since not much happens at this time in most matches, and even in the matches that do have things happen, a "big lead" of 30+ points is not very intuitive since there is no way a team could even have this much of a lead this early (excluding penalties). By the end of auto though, we see the big lead value settle at around 20, which sounds about right, teams who are up by 20 after auto are probably feeling pretty good since they probably have control of both the switch and the scale. After auto, what is considered a big lead increases steadily until peaking at around 60 points at around 60 seconds in. This seems to make sense, because a team up by 20 after auto should be up by 60 40 seconds later if they control the scale the whole time and nothing else changes. After this, the "big lead" holds steady until 110 seconds, when it sharply drops but then recovers at ~122 seconds. I don't know the explanation for this, but my gut tells me this has to do with climbing positioning. After that, the "big lead" drops until ending at 29 points. This means that, if you see your score on the screen at the end of the match at 30 points, you are about 90% likely to win the match in the final score, and if you are up by 60 points, you are about 99% likely to win the match.

I mentioned that the form of my model uses the red winning margin, but that's not precisely true. In fact, I use an adjusted red winning margin where I account for ownership of the scale and of the switch. Basically, I found how much "value" to give to switch and scale ownership at each point in the match. What I mean by "value" is this, if red is down by X points but controls the scale, what is the value of X such that red and blue have an equal chance of winning the match? Here is a graph of value versus time for the scale:


Again, skipping the first 10 seconds of auto, we see scale ownership to be worth ~30 points after auto. It then drops in value to a min of 25 points at 23 seconds. This drop might be due to the initial scuffle for the scale. By 45 seconds, the scale has peaked in value at 49 points, and then it has a jittery drop until the end of the match. The same dip seen in "big lead" also appears in scale value at around 120 seconds. Interestingly, scale value does not go to 0 at the end of the match, but rather ends at 8 points. Perhaps scale ownership provides some indication of climb success?

Here is a similar graph for switch value:


Most of the same trends as in the previous graph also appear in this one. The biggest difference though is that switch value actually does go to 0 by the end of the match.


Let me know if you have any questions. I'll have more to come soon, including win probability graphs and match "excitement" and "comeback/upset" scores.
Reply With Quote
  #12   Spotlight this post!  
Unread 08-17-2018, 03:21 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

Quote:
Originally Posted by Caleb Sykes View Post
I'll have more to come soon, including win probability graphs and match "excitement" and "comeback/upset" scores.
I've just uploaded a sheet called "live_win_probabilities" which has win probabilities at every second of the match for each match in my data set. Only about 10% of matches are in this data set, I still might go back and clean up the messier ones if people are interested, but idk. I also calculated a couple of other metrics which could be used to determine how "good" a match is, here's a summary of them:
team quality: this is just the pre-match average Elo of the competing teams
stakes: This is just a simple combination of the match type and the match number, qual matches have values of zero, and the first matches in the quarterfinals have values of 1, and the second matches of the quarterfinals have values of 2. This continues up until the highest stakes of final 3, which have values of 9. This is just a rough measurement obviously.
upset/comeback score: This is the probability that the winning alliance will win at their weakest point in the match. A value of 98% indicates the wining team had, at their worst point in the match, a 2% chance of winning.
excitement score: This looks at how much the win probability changes for an alliance over the course of a match. The units are a bit hard to grasp, but essentially a value of 1 indicates that, during the match, there was a full swing from a red win to a blue win or vice versa. So a value of 5 indicates 5 full swings of the expected match outcome.

According to these metrics, here are the matches in my data set that had the most excitement:
Tesla Division qm 1 29
Tesla Division qf 2 3
San Francisco Regional qm 1 33
Finger Lakes Regional qm 1 68
Idaho Regional qm 1 35

Here are the matches that had the biggest comeback or upset:
PNW District SunDome Event qf 2 2
Canadian Rockies Regional qm 1 64
ONT District Western University, Western Engineering Event f 1 1
Tesla Division qm 1 105
Minnesota North Star Regional qm 1 41

I also added a "lookup" sheet to the workbook which can be used to generate win probability graphs. For example, here's the win probability graph for the last comeback I mentioned, Minnesota North Star Regional qm 1 41:


I had a lot of fun looking at this data. Live win probabilities are something I've been dreaming about for FRC ever since I saw them for other sports. I hope to be able to analyze the 2019 data before the season ends, but no promises.
Reply With Quote
  #13   Spotlight this post!  
Unread 08-18-2018, 05:09 PM
Caleb Sykes's Avatar
Caleb Sykes Caleb Sykes is offline
Knock-off Dr. Strange
AKA: inkling16
no team
 
Join Date: Feb 2011
Rookie Year: 2009
Location: Minneapolis, Minnesota
Posts: 1,697
Caleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond reputeCaleb Sykes has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

For anyone that already wants to start thinking about 2019, I've uploaded a book which shows the 2019 start of season Elo ratings for all teams. I'm not currently planning to change my Elo model again before next season (although note that I'm bad at evaluating myself). Results are copied here:
Code:
Team	2019 start of season Elo
254	1900
2056	1876
1678	1865
118	1828
2481	1820
195	1817
1114	1817
2046	1817
694	1806
148	1801
3310	1791
2590	1789
3309	1788
2767	1785
1323	1782
125	1778
2910	1778
1574	1775
1538	1774
2122	1774
225	1770
1619	1768
1986	1767
3130	1764
842	1761
5406	1760
1241	1760
217	1758
971	1756
1796	1755
16	1751
4613	1750
133	1750
27	1749
4539	1744
5172	1744
180	1742
25	1741
4003	1740
3538	1740
624	1740
3357	1740
2168	1740
1690	1739
987	1739
67	1739
230	1738
3478	1737
2471	1734
340	1732
1629	1732
179	1731
3452	1730
1806	1728
319	1727
1318	1727
3005	1725
330	1724
399	1722
5050	1722
141	1722
2337	1721
494	1721
85	1719
3476	1718
2614	1718
2791	1718
2478	1715
1519	1715
5460	1715
3707	1715
33	1714
4488	1714
3339	1713
1325	1712
1747	1712
3512	1711
365	1709
4028	1708
4917	1708
4237	1706
1756	1705
177	1705
4910	1705
176	1705
2642	1704
3959	1704
1706	1703
1425	1703
5818	1703
973	1703
5687	1701
610	1701
1640	1700
1918	1700
5190	1699
291	1695
525	1693
1876	1692
5895	1692
1533	1692
868	1691
604	1690
56	1690
2403	1690
233	1690
2834	1689
234	1689
1712	1689
2059	1689
3937	1689
3539	1688
359	1688
4451	1686
2052	1685
4362	1685
302	1684
269	1683
71	1683
1506	1683
1024	1683
865	1681
364	1680
1923	1680
95	1679
6329	1679
3015	1678
4564	1678
3255	1678
384	1678
910	1677
303	1676
1259	1676
5199	1676
3674	1675
1731	1675
4513	1675
5985	1675
1730	1674
6763	1673
2992	1673
1023	1673
1741	1672
2557	1672
1648	1672
744	1671
341	1671
175	1670
870	1669
1102	1669
379	1668
2848	1668
5987	1668
2410	1668
4253	1668
3098	1668
2630	1667
4618	1667
4265	1666
1726	1666
4946	1665
1492	1665
4063	1665
4143	1664
346	1663
1011	1663
876	1663
4476	1663
3683	1663
1736	1663
294	1663
3802	1662
2130	1661
1391	1660
4911	1660
3646	1660
1305	1659
957	1659
3663	1659
3128	1659
6705	1659
3990	1659
832	1659
5803	1658
368	1658
846	1658
2451	1658
78	1657
4655	1657
2075	1657
1477	1656
188	1656
4188	1656
4334	1655
1025	1655
3641	1655
3616	1655
4976	1654
4678	1654
1746	1654
503	1654
862	1654
5505	1654
1339	1653
5802	1653
5254	1653
456	1652
1622	1652
5472	1651
4522	1651
2877	1651
488	1651
5053	1651
4272	1651
772	1651
5012	1651
2522	1650
20	1650
3546	1650
3824	1650
639	1650
2013	1649
3316	1649
1507	1649
2974	1649
2502	1649
1723	1649
4587	1648
2383	1648
1559	1648
5572	1648
3230	1648
1018	1648
1983	1647
1065	1647
1772	1646
1296	1646
4561	1646
2826	1646
2194	1645
70	1645
2960	1645
1577	1645
1684	1645
3314	1644
2485	1644
314	1644
6672	1643
4004	1643
1262	1643
2706	1643
115	1643
1885	1642
1197	1642
135	1642
5842	1640
6579	1640
5883	1640
649	1639
3536	1639
88	1639
5436	1639
3656	1639
3620	1638
5501	1638
4388	1638
4405	1638
1073	1638
2200	1638
4189	1638
2771	1637
4635	1637
834	1637
2137	1637
930	1637
4145	1636
3750	1636
1414	1636
2996	1636
333	1636
3547	1636
2708	1635
3847	1635
3542	1635
5567	1635
6201	1635
2930	1635
2976	1634
2231	1634
1836	1634
2054	1633
2556	1633
4776	1632
3419	1632
5036	1632
6025	1631
5813	1631
2655	1631
2987	1631
5026	1631
245	1631
3835	1631
2659	1630
1421	1630
6090	1630
1718	1630
469	1629
4980	1629
2363	1629
1156	1629
2175	1629
1255	1629
2682	1629
2474	1629
386	1628
5404	1628
2338	1628
5046	1628
1360	1628
103	1628
4213	1628
2990	1628
4541	1627
6328	1627
1572	1627
5152	1627
3276	1627
2144	1627
66	1627
6334	1627
3489	1626
4391	1626
5675	1626
6418	1626
1322	1625
1744	1625
4469	1625
1599	1625
1058	1625
5817	1625
687	1624
3534	1624
3250	1623
316	1623
4967	1623
5155	1623
4944	1623
4855	1623
829	1623
1100	1622
5603	1622
4146	1622
597	1622
3929	1622
5509	1622
2783	1622
1710	1621
708	1621
2486	1621
422	1621
1218	1621
4290	1621
3481	1621
1319	1621
3770	1621
4959	1621
2202	1621
997	1620
5114	1620
1987	1620
578	1619
3461	1619
237	1619
1189	1619
2611	1619
3328	1618
5431	1618
1732	1618
238	1618
2164	1617
2468	1617
1625	1617
1540	1616
6502	1616
75	1616
4533	1616
5584	1616
1418	1616
4499	1615
4276	1615
3997	1615
4550	1615
4415	1615
1816	1615
3197	1615
222	1615
343	1615
4795	1614
5986	1614
3374	1614
2811	1613
3695	1613
226	1613
4027	1613
1778	1613
4908	1613
4039	1613
236	1612
3256	1612
3042	1612
63	1612
2537	1612
5530	1612
6117	1611
5434	1611
4061	1611
3623	1611
5499	1610
3010	1610
2199	1610
1595	1610
3218	1609
1002	1609
968	1609
3647	1609
5576	1609
342	1609
3360	1609
58	1608
348	1608
2102	1608
1592	1608
4525	1608
3635	1607
5338	1607
888	1607
4020	1606
1523	1606
5654	1606
4087	1606
207	1606
4623	1606
2152	1606
3737	1606
167	1606
287	1605
1658	1605
6077	1605
5663	1605
4472	1605
358	1605
1071	1605
3277	1605
5614	1604
3667	1604
159	1604
120	1604
59	1603
614	1603
5846	1603
4198	1603
2851	1603
1807	1603
3313	1602
2377	1602
51	1602
1511	1602
670	1602
836	1602
1596	1602
2359	1601
3711	1600
3196	1600
4646	1600
3179	1600
948	1600
4392	1600
433	1600
3236	1600
6429	1600
5013	1600
3940	1599
5420	1599
6411	1599
1902	1599
1683	1599
3260	1599
3931	1598
1410	1598
4468	1598
999	1598
2765	1598
3284	1598
1729	1598
2823	1598
1792	1598
4183	1598
383	1598
4557	1598
4041	1597
2977	1597
2702	1597
4361	1597
418	1597
2386	1597
5528	1597
5823	1597
1306	1597
2342	1596
623	1596
86	1596
4216	1596
2883	1596
360	1596
2607	1596
4450	1596
6800	1595
4965	1595
2928	1595
126	1595
3928	1595
3039	1595
3140	1595
3668	1594
3171	1594
5437	1594
1405	1594
5010	1594
6548	1594
4026	1594
4930	1594
5188	1594
2875	1594
5546	1594
3883	1594
3184	1593
612	1593
4239	1593
3970	1593
2832	1593
3223	1593
5234	1593
2181	1593
4531	1593
1699	1593
4961	1593
5544	1593
2170	1593
3492	1592
7021	1592
1251	1592
5427	1592
3535	1592
144	1592
1675	1592
6193	1592
156	1591
4779	1591
4984	1591
6886	1590
2352	1590
558	1590
1939	1590
2601	1590
2959	1590
498	1589
2090	1589
2658	1589
1720	1589
1880	1589
3164	1589
93	1589
1126	1588
228	1588
2040	1588
1676	1588
191	1588
2512	1588
5314	1588
1768	1588
4096	1588
1553	1588
5587	1587
131	1587
4688	1587
6378	1587
2016	1587
1817	1587
4611	1587
2509	1587
1501	1587
5216	1587
4325	1587
6474	1587
5150	1586
1178	1586
74	1586
5517	1586
3459	1586
5889	1586
4079	1585
3293	1585
1825	1585
4818	1585
2169	1585
5674	1585
4181	1585
991	1585
5554	1584
1310	1584
4639	1584
8	1584
4508	1584
4941	1584
4077	1584
4085	1584
2415	1584
3244	1584
395	1584
201	1583
1671	1583
5945	1583
5801	1583
747	1583
2457	1583
1108	1583
2609	1583
6406	1583
21	1583
5205	1583
2521	1583
5976	1582
3624	1582
263	1582
5735	1582
5492	1582
353	1582
2579	1582
5453	1582
3853	1582
4176	1582
1551	1582
1721	1582
3882	1582
5099	1582
4786	1581
5285	1581
2846	1581
4909	1581
2283	1581
1257	1581
3238	1581
111	1581
533	1581
181	1581
1657	1580
108	1580
5407	1580
1317	1580
6647	1580
527	1580
6445	1580
663	1580
6104	1580
4653	1580
5458	1580
173	1580
4638	1579
60	1579
5193	1579
192	1579
61	1579
1708	1579
1481	1579
107	1579
5291	1579
3793	1579
2252	1578
5990	1578
2445	1578
1711	1578
5618	1578
1099	1578
5410	1578
3026	1578
2145	1578
4159	1578
3550	1578
894	1578
2526	1577
847	1577
467	1577
4607	1577
4103	1577
4206	1577
3324	1577
447	1577
2084	1577
3472	1577
2062	1577
2710	1577
2491	1577
2225	1577
1557	1577
696	1576
2638	1576
3604	1576
4586	1576
1982	1576
3137	1576
102	1575
1014	1575
548	1575
5804	1575
5414	1575
2073	1575
5550	1575
6933	1575
6055	1575
4750	1574
5740	1574
5596	1574
4738	1574
5496	1574
4256	1574
4055	1574
219	1574
79	1573
4342	1573
3242	1573
5907	1573
5000	1573
4905	1573
3767	1573
3572	1573
6582	1572
1208	1572
7179	1572
1745	1572
2830	1572
3008	1572
2067	1572
3838	1572
5030	1571
2907	1571
292	1571
6508	1571
6415	1571
5951	1571
3490	1571
5015	1571
4904	1571
6121	1571
3473	1571
4400	1571
229	1571
4537	1571
6009	1571
5635	1570
3653	1570
580	1570
5843	1570
4915	1570
3637	1570
4814	1570
2180	1570
4970	1570
2147	1570
3100	1570
702	1569
3602	1569
3414	1569
3574	1569
5413	1569
5847	1569
5148	1569
4983	1569
3335	1568
5086	1568
3986	1568
4230	1568
4462	1568
2501	1568
1626	1568
2539	1568
3792	1568
703	1567
4859	1567
4536	1567
470	1567
2227	1567
7039	1567
203	1567
5442	1566
3618	1566
5588	1566
4050	1566
5039	1566
3034	1566
1662	1566
3003	1566
6424	1566
190	1565
1989	1565
4565	1565
4920	1565
3082	1565
5763	1565
1937	1565
123	1565
2836	1565
223	1565
3932	1565
1	1565
1155	1564
3880	1564
281	1564
2472	1564
3038	1564
2657	1564
1610	1564
308	1564
4242	1564
1072	1564
3239	1564
6753	1564
2637	1564
323	1563
5690	1563
1466	1563
3418	1563
5426	1563
354	1563
3072	1563
1661	1563
3410	1562
4384	1562
5524	1562
5468	1562
6618	1562
1279	1562
2341	1562
3999	1562
5024	1562
668	1562
5522	1562
1413	1561
4458	1561
5914	1561
293	1561
3996	1561
5006	1561
231	1561
3322	1561
1518	1561
401	1561
4322	1561
611	1561
7225	1561
5145	1560
1369	1560
5137	1560
4	1560
4130	1560
4774	1560
272	1560
6344	1560
4174	1560
3998	1560
5204	1560
4592	1560
3814	1560
4486	1560
599	1560
5429	1559
1701	1559
6907	1559
2648	1559
5805	1559
3504	1559
4355	1559
1089	1558
5102	1558
812	1558
818	1558
1285	1558
4819	1558
5638	1558
5826	1558
2264	1558
3258	1558
6493	1557
3506	1557
4122	1557
801	1557
1250	1557
907	1557
1287	1557
4485	1557
4150	1557
2605	1557
3603	1557
3494	1557
1243	1557
3161	1557
122	1557
2839	1557
3009	1557
4481	1557
5561	1557
5819	1557
5612	1557
2158	1557
3965	1557
3024	1557
5892	1556
4931	1556
250	1556
2333	1556
2903	1556
5222	1556
501	1556
2508	1556
2220	1556
5553	1556
329	1556
1868	1555
3630	1555
601	1555
4192	1555
2594	1555
4939	1555
2035	1555
4584	1555
6613	1554
1810	1554
492	1554
3593	1554
2612	1554
6343	1554
2587	1554
2370	1554
1311	1554
5712	1554
2438	1554
369	1554
4001	1554
5653	1554
5822	1554
2395	1554
1124	1553
5278	1553
1522	1553
5590	1553
1038	1553
1895	1553
3044	1553
1795	1553
114	1553
2081	1553
5124	1553
5462	1553
5910	1552
2635	1552
3840	1552
967	1552
5424	1552
620	1552
6024	1552
554	1552
4296	1552
1860	1552
5839	1552
6340	1552
2053	1551
2534	1551
3844	1551
4401	1551
1997	1551
5950	1551
3735	1551
3354	1551
5507	1551
2582	1551
2543	1551
4376	1551
4680	1550
2503	1550
3826	1550
5927	1550
871	1550
145	1550
2619	1550
5608	1550
4048	1550
48	1549
5449	1549
457	1549
1495	1549
279	1549
41	1549
6831	1549
2106	1549
6359	1549
5450	1549
4201	1549
4381	1549
4470	1549
441	1549
11	1549
5422	1549
4409	1548
6224	1548
23	1548
839	1548
4936	1548
1153	1548
4656	1548
2212	1548
1051	1548
4495	1548
3330	1547
4293	1547
3158	1547
6110	1547
4990	1547
1111	1547
3755	1547
2869	1547
213	1547
1289	1547
6615	1547
6517	1547
3953	1547
68	1547
935	1547
1094	1547
4994	1547
5736	1546
6072	1546
4131	1546
3386	1546
2890	1546
5511	1546
6630	1546
4135	1546
2586	1545
3021	1545
3581	1545
1764	1545
4603	1545
3676	1545
3925	1545
6237	1545
2679	1545
3132	1545
3176	1545
5676	1545
6027	1545
3556	1545
6803	1545
3655	1545
4918	1545
3555	1544
3926	1544
5811	1544
5428	1544
4069	1544
3688	1544
6352	1544
5331	1544
6100	1544
4014	1544
3299	1544
1091	1544
3660	1544
3274	1544
2412	1544
7048	1544
4769	1543
6377	1543
2643	1543
4512	1543
5563	1543
5724	1543
587	1543
5052	1543
3075	1543
701	1543
2518	1543
5547	1543
5980	1543
4698	1542
5439	1542
4207	1542
157	1542
3011	1542
172	1542
5535	1542
3875	1542
5860	1542
5045	1542
4075	1542
5480	1542
1388	1542
3219	1542
6829	1542
5179	1541
4125	1541
6386	1541
4669	1541
3294	1541
3495	1541
1884	1541
5317	1541
3691	1541
852	1541
3598	1540
3526	1540
3863	1540
94	1540
3303	1540
6978	1540
3951	1540
5560	1540
1468	1540
1775	1540
1561	1540
4935	1540
1777	1539
3633	1539
4956	1539
2465	1539
2584	1539
461	1539
3275	1539
337	1539
6200	1539
5523	1539
6390	1538
815	1538
5913	1538
1164	1538
949	1538
3117	1538
1444	1538
3480	1538
3200	1538
2340	1538
5417	1538
4521	1538
5104	1538
6538	1537
6479	1537
932	1537
4906	1537
1248	1537
2373	1537
2354	1537
2393	1537
977	1537
2405	1537
835	1537
1750	1537
4264	1537
4711	1537
6366	1537
4246	1537
5948	1537
3777	1537
6043	1536
4241	1536
4375	1536
4627	1536
4089	1536
2852	1536
4523	1536
1280	1536
662	1536
6417	1536
3588	1536
4395	1536
449	1536
4737	1536
6500	1536
5467	1536
6357	1536
6550	1536
5242	1536
3966	1536
4175	1535
174	1535
6022	1535
3679	1535
5464	1535
7160	1535
4459	1535
3206	1535
2606	1535
5952	1535
5181	1534
6323	1534
6880	1534
857	1534
5403	1534
4903	1534
3974	1534
4810	1534
571	1534
3991	1534
3247	1534
4952	1534
247	1534
121	1534
101	1534
900	1534
4991	1534
2141	1533
3007	1533
3692	1533
4330	1533
1584	1533
4947	1533
6510	1533
5901	1533
5096	1533
3753	1533
3501	1533
116	1533
2729	1533
5577	1533
5502	1533
2165	1533
4356	1533
6020	1533
1389	1533
4320	1533
4775	1533
5658	1532
3654	1532
2980	1532
4619	1532
5232	1532
334	1532
7034	1532
6451	1532
3341	1532
1086	1532
1138	1532
283	1532
2085	1532
5209	1532
288	1532
5112	1532
3379	1532
6544	1531
1758	1531
5940	1531
7154	1531
6434	1531
3487	1531
246	1531
6155	1531
6026	1530
2375	1530
2972	1530
178	1530
931	1530
1188	1530
1493	1530
7200	1530
5508	1530
5084	1530
271	1530
2028	1530
1967	1530
4009	1530
2449	1530
3511	1530
2662	1530
3484	1529
3160	1529
6421	1529
2191	1529
486	1529
6003	1529
2399	1529
5526	1529
5623	1529
4073	1529
6131	1529
7127	1529
5715	1529
4343	1529
5866	1528
484	1528
2143	1528
4121	1528
2500	1528
6300	1528
5997	1528
2733	1528
3749	1528
540	1528
2797	1528
6591	1528
4064	1528
1261	1528
6425	1528
6052	1528
2230	1528
244	1528
5109	1528
509	1528
378	1528
514	1528
2022	1527
4778	1527
4456	1527
1635	1527
3187	1527
6420	1526
5827	1526
1266	1526
1660	1526
5401	1526
2172	1526
2487	1526
2135	1526
3289	1526
5903	1526
4226	1526
3950	1526
2989	1526
2549	1526
3760	1526
5160	1526
568	1526
4761	1525
4060	1525
3381	1525
7329	1525
858	1525
138	1525
4576	1525
6225	1525
4500	1525
4972	1525
716	1525
3941	1525
4982	1524
5810	1524
2600	1524
3871	1524
5930	1524
1569	1524
1714	1524
3018	1524
573	1524
7178	1524
2849	1524
6573	1524
5173	1524
5696	1524
1403	1524
3958	1524
2080	1524
3146	1524
5230	1523
6336	1523
753	1523
1160	1523
1763	1523
6314	1523
1245	1523
1539	1523
5533	1523
1351	1522
6085	1522
4546	1522
4999	1522
3345	1522
3145	1522
1225	1522
5299	1522
5451	1522
2815	1522
5716	1522
980	1522
4377	1522
6177	1522
5461	1522
6996	1522
2096	1522
3389	1522
4153	1522
2574	1522
2656	1521
3606	1521
2493	1521
4624	1521
28	1521
2148	1521
5891	1521
4454	1521
1797	1520
2993	1520
4398	1520
5409	1520
6869	1520
3408	1520
3267	1520
2443	1520
1984	1520
2856	1520
7250	1520
2626	1520
6172	1520
2640	1520
4582	1520
1080	1520
4728	1520
5182	1520
3933	1520
6122	1520
5627	1520
7079	1520
6217	1519
5504	1519
2221	1519
2421	1519
3245	1519
6679	1519
6490	1519
5276	1519
2994	1519
3229	1519
3467	1519
5484	1519
6590	1519
3397	1518
3355	1518
7140	1518
5865	1518
6458	1518
2723	1518
4306	1518
6443	1518
4845	1518
6941	1518
5212	1518
619	1518
751	1518
7109	1518
2183	1518
4021	1518
1891	1518
6530	1518
4590	1517
6704	1517
4010	1517
3570	1517
3455	1517
6361	1517
6637	1517
5327	1517
1458	1517
5998	1517
4664	1517
4099	1517
5032	1517
6945	1517
4622	1516
1246	1516
1143	1516
2499	1516
6802	1516
5775	1516
3136	1516
6112	1516
5970	1516
3848	1516
2344	1516
5238	1516
5332	1516
5856	1516
6758	1516
3954	1516
2560	1516
2374	1515
3482	1515
3658	1515
1600	1515
2186	1515
6036	1515
6695	1515
3859	1515
6717	1515
4255	1515
4585	1515
6056	1515
2240	1515
2795	1515
3058	1515
6153	1515
6988	1515
3464	1515
4327	1515
3537	1515
4263	1515
4035	1515
2335	1515
2861	1515
4514	1514
5632	1514
6033	1514
3036	1514
5031	1514
3211	1514
4735	1514
5176	1514
3627	1514
2583	1514
5659	1514
5115	1514
3721	1514
4076	1514
3488	1514
4473	1514
691	1514
3522	1514
3719	1514
3070	1514
1735	1514
2345	1514
6098	1514
2827	1513
5443	1513
1228	1513
5531	1513
2498	1513
4509	1513
6731	1513
1647	1513
5119	1513
5809	1513
6081	1513
3142	1513
4571	1513
1991	1513
7187	1513
2789	1512
5568	1512
4209	1512
6394	1512
6722	1512
6065	1512
4277	1512
5347	1512
6054	1512
5899	1512
6951	1512
2973	1512
706	1512
7117	1512
5864	1512
6150	1512
648	1512
5851	1512
4160	1512
3061	1512
6836	1512
4403	1512
3083	1512
1646	1512
5900	1512
2559	1511
1912	1511
2190	1511
5224	1511
686	1511
5295	1511
5684	1511
5872	1511
6342	1511
3081	1511
3234	1511
6471	1511
5702	1511
2882	1511
7054	1511
1740	1511
2473	1510
4482	1510
6317	1510
569	1510
5920	1510
7074	1510
3023	1510
6909	1510
2576	1510
4267	1510
6465	1510
4191	1510
5816	1510
547	1510
581	1510
2204	1510
6852	1510
435	1509
7162	1509
3189	1509
4054	1509
6628	1509
5090	1509
2641	1509
5072	1509
4329	1509
6067	1509
4112	1509
4951	1509
6113	1509
3205	1509
2914	1509
6919	1509
4534	1509
4663	1509
6766	1509
7026	1509
5924	1509
1757	1509
5525	1509
6175	1508
1294	1508
1329	1508
3405	1508
4080	1508
3337	1508
6546	1508
2634	1508
2129	1508
3163	1508
2078	1508
5419	1508
5171	1508
2881	1508
4731	1508
1209	1508
5806	1508
5183	1508
4074	1508
5196	1508
1517	1508
6542	1508
5059	1508
4632	1507
6908	1507
4573	1507
5274	1507
4011	1507
6087	1507
5348	1507
4012	1507
6696	1507
4593	1507
7051	1507
6337	1507
3560	1507
2879	1507
4914	1507
5293	1507
100	1507
2367	1507
6483	1507
6823	1507
4572	1507
2876	1507
321	1507
4749	1507
3359	1507
5624	1506
4643	1506
5669	1506
296	1506
4505	1506
6084	1506
5709	1506
5098	1506
6484	1506
5085	1506
5993	1506
6050	1506
4285	1506
709	1506
5093	1506
69	1506
1334	1506
4471	1506
6305	1506
4215	1506
904	1506
698	1505
811	1505
6001	1505
4243	1505
7213	1505
5933	1505
6353	1505
5599	1505
5113	1505
5038	1505
3513	1505
2523	1505
5078	1505
5982	1505
6322	1505
5781	1505
1781	1505
6824	1505
2398	1505
2051	1505
5754	1505
4782	1505
810	1505
4057	1505
6957	1505
4091	1505
4467	1505
6358	1504
3019	1504
2079	1504
5126	1504
2551	1504
1515	1504
2805	1504
1288	1504
7231	1504
3609	1504
6086	1504
2425	1504
5688	1504
6174	1504
6325	1504
6843	1504
4338	1504
6936	1503
5110	1503
3216	1503
5730	1503
6556	1503
5411	1503
1482	1503
6621	1503
6168	1503
3759	1503
6324	1503
3734	1503
3202	1503
4600	1503
5836	1503
2867	1502
7287	1502
4610	1502
4601	1502
2444	1502
2550	1502
4501	1502
5960	1502
1965	1502
5748	1502
1591	1502
2773	1502
6827	1502
5881	1502
7111	1502
4353	1502
4758	1502
5217	1502
3544	1502
2647	1501
3166	1501
2353	1501
5136	1501
4630	1501
451	1501
4608	1501
6191	1501
4926	1501
6432	1501
6422	1501
2667	1501
830	1500
7110	1500
7169	1500
5975	1500
34	1500
3388	1500
6569	1500
7239	1500
1076	1500
4169	1500
6354	1500
4013	1500
224	1500
3936	1500
1785	1500
5506	1500
339	1500
5518	1500
2713	1499
1793	1499
2495	1499
6348	1499
1212	1499
2458	1499
2704	1499
1157	1499
6709	1499
4595	1499
945	1499
5162	1499
1510	1499
7157	1499
4979	1498
4748	1498
972	1498
1771	1498
3138	1498
5555	1498
6856	1498
6527	1498
6744	1498
6878	1498
5261	1498
5515	1498
3288	1498
677	1498
295	1498
7144	1498
5248	1498
4019	1497
3402	1497
3612	1497
6813	1497
4516	1497
6047	1497
1672	1497
5229	1497
3745	1497
2232	1497
6044	1497
3147	1497
4765	1497
2613	1497
2036	1497
3491	1497
3591	1497
4799	1496
7211	1496
1972	1496
6845	1496
6063	1496
5510	1496
5134	1496
1382	1496
6401	1496
5928	1496
4416	1496
5048	1495
2411	1495
1075	1495
5774	1495
1450	1495
3004	1495
6170	1495
6032	1495
6861	1495
6070	1495
5885	1495
3055	1495
4829	1495
2916	1495
7047	1495
6392	1495
4262	1494
6331	1494
4648	1494
6659	1494
6391	1494
3102	1494
5465	1494
1847	1494
6097	1494
3134	1494
6875	1494
3220	1494
6318	1494
4107	1494
6133	1494
6321	1494
555	1494
3839	1494
3366	1494
5516	1494
2171	1494
5855	1494
6404	1493
6574	1493
3799	1493
1370	1493
5915	1493
6019	1493
5908	1493
1422	1493
5837	1493
5753	1493
7116	1493
4049	1493
5263	1493
4123	1493
6898	1493
3930	1493
2228	1493
6442	1493
5339	1493
4580	1493
2896	1492
5857	1492
6694	1492
5082	1492
3596	1492
5681	1492
3577	1492
4598	1492
6093	1492
2813	1492
193	1492
4594	1492
6364	1492
6002	1492
5144	1492
6518	1492
1818	1492
4314	1492
6199	1492
3043	1492
6398	1492
4118	1492
4005	1492
5959	1492
2437	1492
3225	1492
2207	1492
6720	1491
6671	1491
3881	1491
2978	1491
5542	1491
5953	1491
6901	1491
4104	1491
4418	1491
6558	1491
6657	1491
1504	1491
4998	1491
4284	1491
5418	1491
5056	1491
5022	1491
955	1491
4578	1491
5934	1490
3597	1490
5720	1490
4704	1490
3182	1490
1159	1490
7331	1490
5253	1490
6489	1490
5159	1490
5973	1490
3180	1490
5243	1490
4309	1490
6949	1490
6194	1490
3035	1490
7120	1490
5964	1490
5421	1490
3566	1490
7142	1489
5260	1489
4328	1489
6622	1489
1985	1489
7043	1489
4065	1489
1432	1489
4180	1489
4692	1489
7278	1489
6476	1489
3319	1489
5069	1489
5831	1489
3329	1489
4119	1489
981	1489
7013	1489
3173	1488
3103	1488
6929	1488
6160	1488
4097	1488
4270	1488
4152	1488
6844	1488
6071	1488
7322	1488
199	1488
6111	1488
4950	1488
1665	1488
5067	1488
3559	1488
7262	1488
2530	1488
6351	1488
6106	1487
4809	1487
1247	1487
2358	1487
1872	1487
4151	1487
4964	1487
5498	1487
6164	1487
216	1487
6668	1487
4397	1487
5641	1487
6226	1487
1927	1487
31	1487
6482	1487
6023	1487
6341	1487
4511	1486
6888	1486
6403	1486
988	1486
2935	1486
4006	1486
5752	1486
7222	1486
1727	1486
6729	1486
6441	1486
4043	1486
5686	1486
7137	1486
3212	1486
6381	1486
5941	1486
5161	1486
2554	1486
3796	1486
589	1486
6914	1486
3533	1485
5142	1485
5016	1485
1378	1485
6746	1485
5514	1485
1123	1485
5830	1485
7028	1485
5202	1485
3648	1485
5943	1485
6132	1485
3006	1485
6842	1485
2177	1485
6514	1485
4504	1485
6166	1485
3669	1485
6075	1485
7004	1485
6808	1485
4575	1485
6239	1485
6721	1484
7226	1484
2532	1484
3975	1484
6771	1484
4212	1484
1293	1484
5678	1484
5519	1484
2604	1484
2077	1484
6655	1484
7123	1484
5708	1484
4269	1484
6088	1484
4942	1484
6440	1483
5682	1483
3751	1483
7190	1483
848	1483
2484	1483
5622	1483
6990	1483
3510	1483
5168	1483
2964	1483
5966	1483
4717	1483
7069	1483
3157	1483
5613	1483
5935	1483
3957	1483
6934	1483
4506	1483
4332	1483
2872	1483
5923	1483
6702	1482
6423	1482
2854	1482
6326	1482
6004	1482
2577	1482
6202	1482
5852	1482
6801	1482
5988	1482
2439	1482
2895	1482
4662	1482
1769	1482
6188	1482
496	1482
5459	1482
5578	1482
1277	1482
7267	1482
3407	1482
6038	1482
117	1482
6464	1482
6060	1481
3947	1481
885	1481
6851	1481
4336	1481
6068	1481
5074	1481
6708	1481
3352	1481
695	1481
743	1481
5729	1481
7214	1481
7153	1481
3693	1481
45	1481
4141	1481
6567	1481
5470	1481
6076	1481
7053	1480
3619	1480
4373	1480
4496	1480
3740	1480
5593	1480
3096	1480
6146	1480
4992	1480
4932	1480
6204	1480
6664	1480
97	1480
2489	1480
2010	1480
6107	1480
640	1480
5494	1480
5829	1480
423	1480
6970	1480
1258	1480
5066	1480
4187	1480
5057	1480
7148	1480
6833	1479
5937	1479
4652	1479
1590	1479
5534	1479
5962	1479
7230	1479
4740	1479
5957	1479
6690	1479
5695	1479
4741	1479
1787	1479
6706	1479
5683	1479
6492	1479
6171	1479
1583	1479
6503	1479
7275	1479
3468	1479
5235	1479
3193	1479
2187	1479
1571	1479
1254	1479
5143	1479
3152	1479
2853	1478
6397	1478
5095	1478
3636	1478
6402	1478
6885	1478
5440	1478
5630	1478
4114	1478
2531	1478
4693	1478
6203	1478
3955	1478
996	1478
5783	1478
5700	1478
5493	1478
6660	1478
3573	1477
5194	1477
6841	1477
2986	1477
7103	1477
5685	1477
2957	1477
2900	1477
166	1477
6560	1477
4182	1477
6961	1477
6727	1477
4674	1477
6991	1477
4787	1477
7068	1477
3265	1477
816	1477
4248	1477
5268	1477
4464	1477
1831	1477
4694	1476
7198	1476
3543	1476
3786	1476
1940	1476
6593	1476
5089	1476
4851	1476
6230	1476
3545	1476
6867	1476
5787	1476
4729	1476
1633	1476
6356	1476
6701	1476
3175	1476
1943	1476
2239	1476
4120	1476
6899	1476
7056	1476
7234	1476
6964	1476
6918	1476
1332	1476
5657	1476
3729	1476
1165	1476
7058	1476
4378	1476
7112	1476
5992	1476
3172	1476
5025	1476
7206	1476
7217	1475
3037	1475
7166	1475
4412	1475
6742	1475
6335	1475
6350	1475
4645	1475
2545	1475
5863	1475
3661	1475
5968	1475
2424	1475
2064	1475
6883	1475
4295	1474
4726	1474
4383	1474
5002	1474
1942	1474
6302	1474
3466	1474
6413	1474
5256	1474
841	1474
5457	1474
6467	1474
453	1474
806	1474
7080	1474
5974	1474
5874	1474
6686	1474
2906	1474
5147	1474
3507	1474
3139	1474
3292	1474
4453	1474
4723	1474
3946	1474
6993	1473
6874	1473
3065	1473
884	1473
2347	1473
6519	1473
6101	1473
5849	1473
7067	1473
4567	1473
1359	1473
6400	1473
4171	1473
6820	1473
2262	1473
5167	1473
6585	1473
4528	1473
6524	1473
6570	1473
6468	1473
6436	1473
2513	1473
4978	1473
5719	1473
1137	1473
4913	1472
6968	1472
4815	1472
6238	1472
4746	1472
2450	1472
3090	1472
5633	1472
5350	1472
5077	1472
4637	1472
6186	1472
7174	1472
6130	1472
1398	1472
6730	1472
2984	1472
7319	1472
6537	1472
4008	1472
6499	1472
6039	1472
4056	1471
4579	1471
7135	1471
5538	1471
4555	1471
3373	1471
5994	1471
5665	1471
3843	1471
6648	1471
3939	1471
6369	1471
5728	1471
5777	1471
7216	1471
5746	1471
1811	1471
2912	1471
6896	1471
3398	1471
3266	1471
1585	1471
6129	1471
6606	1470
6889	1470
6640	1470
5932	1470
5558	1470
5610	1470
3387	1470
6952	1470
6515	1470
4301	1470
6662	1470
5482	1470
7097	1470
6083	1470
5660	1470
1737	1470
6855	1470
5894	1470
5310	1470
4466	1470
7158	1470
7006	1470
6223	1469
7018	1469
6712	1469
2929	1469
4217	1469
6920	1469
4232	1469
5853	1469
4796	1469
6969	1469
5129	1469
3451	1469
6099	1469
7295	1469
7055	1469
6847	1469
6871	1469
6184	1469
7326	1469
155	1469
6830	1469
5929	1469
6765	1469
5149	1469
4616	1469
7091	1468
2620	1468
6677	1468
3754	1468
7274	1468
6947	1468
2167	1468
6740	1468
6985	1468
3470	1468
5041	1468
3045	1468
6547	1468
4186	1468
6699	1468
5541	1468
5611	1468
3944	1468
4173	1468
4925	1467
6559	1467
6651	1467
7202	1467
7149	1467
5282	1467
6498	1467
6882	1467
2840	1467
4519	1467
1303	1467
6935	1467
4954	1467
7108	1467
7064	1467
442	1467
877	1467
7271	1467
6540	1467
6566	1467
4156	1467
4665	1467
2654	1467
6953	1467
6554	1467
5175	1467
4053	1466
3952	1466
3390	1466
6332	1466
6016	1466
2544	1466
4973	1466
6632	1466
6928	1466
6457	1466
5835	1466
6900	1466
6455	1466
3454	1466
545	1466
5213	1466
6536	1466
3453	1466
2480	1466
2525	1465
4933	1465
7240	1465
4229	1465
7165	1465
6718	1465
1751	1465
6367	1465
854	1465
6887	1465
5773	1465
3723	1465
6922	1465
3984	1465
1908	1465
714	1465
4940	1465
6962	1465
6220	1465
2603	1465
4568	1465
3122	1465
5178	1464
6989	1464
5999	1464
1446	1464
7317	1464
6738	1464
4763	1464
6167	1464
5503	1464
6013	1464
313	1464
537	1464
7059	1464
2961	1464
3785	1464
1452	1464
7195	1464
3257	1464
3297	1464
3568	1464
6127	1464
6870	1464
256	1464
7119	1463
5486	1463
5333	1463
4957	1463
4461	1463
781	1463
7094	1463
6955	1463
3201	1463
7308	1463
6103	1463
7152	1463
6806	1463
4238	1463
5996	1463
2905	1463
4364	1463
3878	1462
6315	1462
6360	1462
6120	1462
5737	1462
6565	1462
4702	1462
7185	1462
7258	1462
5477	1462
3017	1462
7220	1462
3457	1462
5019	1462
6822	1462
2506	1462
2511	1462
7060	1462
5483	1462
6979	1462
4817	1462
3571	1462
6416	1462
6219	1461
3815	1461
5648	1461
6346	1461
7244	1461
7065	1461
5834	1461
6444	1461
6433	1461
3243	1461
7010	1461
6666	1461
1922	1461
7256	1461
2259	1461
4924	1461
4707	1461
1719	1461
5947	1461
3120	1461
2039	1461
6485	1461
3976	1461
6741	1461
4955	1461
6144	1461
2265	1460
7086	1460
4547	1460
5071	1460
3298	1460
5141	1460
3361	1460
7066	1460
5063	1460
867	1460
6749	1460
6700	1460
6488	1460
7243	1460
7040	1460
7312	1460
5919	1460
6189	1460
7173	1460
1566	1460
2001	1460
6526	1460
2927	1460
6619	1459
7297	1459
7138	1459
4744	1459
5219	1459
6504	1459
4247	1459
1977	1459
6371	1459
5911	1459
253	1459
5589	1459
2027	1459
5595	1459
1027	1459
5639	1459
7201	1459
2538	1459
3748	1459
5123	1459
2721	1459
5316	1459
2915	1459
6674	1459
564	1459
1230	1459
7070	1459
2825	1459
3268	1458
6543	1458
5125	1458
6974	1458
6697	1458
4219	1458
5583	1458
7019	1458
6926	1458
6520	1458
4127	1458
4847	1458
6643	1458
7235	1458
2585	1458
6119	1458
7121	1458
7318	1458
6755	1458
5854	1458
6723	1458
2591	1458
7177	1458
5227	1458
1899	1458
4299	1457
6995	1457
6904	1457
5637	1457
7296	1457
6509	1457
7251	1457
6190	1457
6312	1457
7303	1457
6868	1457
6645	1457
6911	1457
6608	1457
7257	1457
3807	1457
1290	1457
5722	1457
6950	1456
5876	1456
5841	1456
7264	1456
6805	1456
4549	1456
5539	1456
3562	1456
5118	1456
6932	1456
766	1456
7122	1456
7301	1456
4762	1456
6872	1456
6644	1456
3062	1455
5047	1455
746	1455
6561	1455
6066	1455
3822	1455
7229	1455
6976	1455
2461	1455
6956	1455
3151	1455
4604	1455
1573	1455
6682	1455
5725	1455
2197	1455
4907	1455
6035	1455
7293	1455
4260	1455
5655	1455
4823	1455
7327	1455
3851	1455
6091	1454
5887	1454
6586	1454
4252	1454
6412	1454
4078	1454
6149	1454
6913	1454
7245	1454
6998	1454
7031	1454
3067	1454
4919	1454
7023	1454
6992	1454
5132	1454
4988	1454
6215	1454
3498	1454
6407	1454
4828	1454
6481	1454
7071	1454
7155	1453
6597	1453
7016	1453
5582	1453
5283	1453
6854	1453
6078	1453
7072	1453
4205	1453
5697	1453
7188	1453
3865	1453
6656	1453
7260	1453
3278	1453
6525	1453
4938	1453
7316	1453
4292	1453
5672	1453
1308	1452
6320	1452
5060	1452
7255	1452
5250	1452
5661	1452
2423	1452
5721	1452
1307	1452
5279	1452
7212	1452
7248	1452
7022	1452
4308	1452
4640	1451
5634	1451
5701	1451
6850	1451
5800	1451
171	1451
4068	1451
7025	1451
2966	1451
6453	1451
1994	1451
6604	1451
5003	1451
7038	1451
7041	1451
3091	1451
4132	1451
5625	1451
3204	1451
6157	1451
2083	1450
6809	1450
6507	1450
4854	1450
6725	1450
6865	1450
4024	1450
4335	1450
2404	1450
5918	1450
1502	1450
5454	1450
5689	1450
2261	1450
6041	1449
5133	1449
6743	1449
3133	1449
1700	1449
7315	1449
4683	1449
7193	1449
6915	1449
6053	1449
5491	1449
3650	1449
6927	1449
1168	1449
5122	1449
7035	1449
5312	1449
5412	1449
6506	1449
6116	1448
6575	1448
4193	1448
5438	1448
6963	1448
6473	1448
6624	1448
6916	1448
6866	1448
4715	1448
4161	1448
1374	1448
4281	1448
3240	1448
6014	1448
6683	1448
7272	1448
1474	1448
3465	1448
3651	1448
5103	1448
5776	1448
6684	1448
7085	1448
5636	1447
2470	1447
7002	1447
4083	1447
6892	1447
6187	1447
6408	1447
4240	1447
5058	1447
1944	1447
2945	1447
1799	1447
6670	1447
5166	1447
2950	1447
4360	1447
4821	1447
6834	1447
5275	1447
6309	1447
5130	1447
5747	1447
5549	1447
2429	1446
6388	1446
4804	1446
1350	1446
5756	1446
6944	1446
6917	1446
4767	1446
6059	1446
7104	1446
5785	1446
3728	1446
6838	1446
4614	1446
6382	1446
5326	1446
5005	1445
6986	1445
280	1445
2095	1445
6626	1445
1759	1445
7030	1445
6649	1445
3029	1445
3704	1445
5203	1445
2496	1445
5983	1445
6620	1445
1598	1445
7146	1444
6646	1444
4682	1444
5344	1444
4734	1444
6832	1444
6612	1444
7046	1444
6588	1444
6825	1444
7036	1444
3812	1444
6921	1444
1642	1443
5255	1443
1529	1443
6140	1443
7299	1443
5626	1443
3054	1443
6459	1443
7265	1443
6762	1443
6073	1443
6362	1443
6419	1443
7289	1443
4034	1443
1716	1443
7189	1443
6393	1443
5875	1443
6858	1442
6804	1442
3610	1442
6616	1442
6958	1442
3800	1442
7017	1442
6141	1442
2998	1442
2855	1442
6818	1442
7232	1442
7081	1442
4235	1441
1739	1441
4286	1441
6306	1441
7298	1441
7219	1441
6564	1441
6128	1441
2709	1441
2676	1441
6768	1441
1802	1441
6750	1441
7107	1441
6138	1441
5198	1441
5570	1441
1786	1441
7084	1441
4835	1441
2515	1440
4166	1440
2942	1440
335	1440
4730	1440
3300	1440
6807	1440
6304	1440
6685	1440
3684	1440
5201	1440
5559	1440
4641	1440
4958	1440
6463	1440
1649	1440
7237	1440
2529	1440
6877	1439
3131	1439
5699	1439
7254	1439
4311	1439
6817	1439
1555	1439
5727	1439
7088	1439
6732	1439
3104	1439
5241	1439
2874	1438
3575	1438
6487	1438
5214	1438
7192	1438
7101	1438
4768	1438
5667	1438
4780	1438
4386	1438
4455	1438
6859	1438
6535	1438
5926	1438
4792	1438
5512	1438
3743	1438
7224	1438
6860	1437
4773	1437
5714	1437
7218	1437
6846	1437
3367	1437
7115	1437
5921	1437
7221	1437
6161	1437
5471	1437
2048	1437
7277	1437
6910	1437
5408	1437
5215	1437
6819	1437
4480	1437
6897	1437
6642	1437
7076	1437
6545	1437
6383	1436
7001	1436
4225	1436
2988	1436
1896	1436
3181	1436
7175	1436
4371	1436
2408	1436
6214	1436
6513	1436
4023	1436
4051	1436
5557	1436
1915	1436
5621	1436
4348	1436
5616	1436
6840	1436
7269	1436
6000	1436
6736	1436
7223	1436
6079	1436
5478	1436
6931	1436
5662	1436
7093	1435
5877	1435
7124	1435
6494	1435
4110	1435
6319	1435
204	1435
6240	1435
3499	1435
6754	1435
299	1435
3821	1434
3988	1434
4856	1434
4015	1434
7294	1434
6435	1434
4137	1434
3876	1434
7029	1434
6094	1433
6902	1433
6912	1433
3680	1433
6982	1433
7118	1433
7197	1433
5765	1433
7151	1433
1803	1433
1512	1433
3617	1433
7126	1433
7266	1433
240	1433
4987	1433
938	1433
3291	1433
606	1433
6948	1432
5265	1432
6512	1432
2707	1432
6118	1432
6946	1432
7128	1432
1546	1432
7203	1432
6925	1432
7167	1432
5107	1432
3515	1432
6923	1431
7238	1431
692	1431
5290	1431
3323	1431
5329	1431
4757	1431
2068	1431
2847	1431
5445	1431
3794	1431
6379	1431
3344	1431
590	1431
6135	1431
3013	1431
5189	1431
6414	1431
7291	1431
5949	1431
7164	1431
6102	1430
6938	1430
4968	1430
7106	1430
5200	1430
7063	1430
3834	1430
4337	1430
6178	1430
2339	1430
7176	1430
6665	1430
6881	1430
3673	1430
6017	1429
468	1429
6581	1429
4838	1429
7191	1429
2161	1429
6894	1429
5821	1429
7102	1429
371	1429
6376	1429
6557	1429
6410	1429
5679	1429
6145	1428
5288	1428
4111	1428
4093	1428
6449	1428
6873	1428
128	1428
3739	1428
6693	1428
2136	1427
6940	1427
6327	1427
5186	1427
6638	1427
1677	1427
4139	1427
7227	1427
6814	1427
6943	1427
2246	1427
1828	1427
1601	1427
6584	1427
6864	1427
5963	1427
5049	1427
5744	1426
6181	1426
5870	1426
4203	1426
2844	1426
5607	1426
6764	1426
5906	1426
2819	1426
4583	1426
6965	1426
2725	1426
7168	1426
6691	1426
4701	1426
6568	1426
6601	1426
6983	1425
7077	1425
6399	1425
6692	1425
2234	1425
6826	1425
5474	1425
6610	1425
3632	1425
6748	1425
653	1425
6635	1424
6680	1424
4562	1424
81	1424
3981	1424
6884	1424
7180	1424
5726	1424
5340	1424
6531	1424
4179	1424
7083	1424
5247	1424
6937	1424
6654	1423
5619	1423
3845	1423
3302	1423
3756	1423
1576	1423
3393	1423
6977	1423
4681	1423
6005	1423
4802	1423
6156	1423
7184	1423
5033	1423
7143	1423
7205	1423
6676	1423
522	1422
7183	1422
1448	1422
6363	1422
6981	1422
7170	1422
7323	1422
6232	1422
7313	1422
3215	1422
5965	1422
5710	1422
7033	1422
3282	1422
7263	1422
7136	1421
7161	1421
4783	1421
6975	1421
3634	1421
5989	1421
5298	1421
3694	1421
6997	1421
7247	1421
4733	1421
6667	1421
6553	1421
6007	1421
5734	1421
6303	1421
3527	1421
6811	1421
7134	1421
7163	1421
7268	1420
6967	1420
2357	1420
5239	1420
3615	1420
7172	1420
5858	1420
7273	1420
7249	1420
3985	1420
7286	1419
3712	1419
1761	1419
6594	1419
1791	1419
7209	1419
5878	1419
6461	1419
5135	1419
7228	1419
6092	1419
6010	1419
6636	1419
5156	1419
6891	1419
5564	1419
1528	1419
6605	1419
7130	1418
7309	1418
7150	1418
6222	1418
7246	1418
7027	1418
6959	1418
6064	1418
6333	1417
5977	1417
7196	1417
1954	1417
6652	1417
3336	1417
4649	1417
6460	1417
6218	1417
6533	1417
6837	1417
7095	1416
6532	1416
5871	1416
7141	1416
7208	1416
7156	1416
2477	1416
2944	1415
2070	1415
6089	1415
1743	1415
4380	1415
7321	1415
7045	1415
6137	1415
5251	1415
4777	1414
4739	1414
6074	1414
2898	1414
3370	1414
6589	1414
7290	1414
4811	1414
5527	1413
5430	1413
6627	1413
4291	1413
7050	1413
3525	1413
6339	1413
2224	1413
2969	1412
5586	1412
5562	1412
6658	1412
5308	1412
6812	1412
7145	1411
7302	1411
1580	1411
7288	1411
5944	1411
5257	1411
7125	1411
4408	1410
4283	1410
5473	1410
7037	1410
5762	1410
4661	1410
6355	1410
7147	1409
7052	1409
6905	1409
4625	1409
7314	1409
7133	1409
6011	1409
3028	1409
6960	1409
6716	1409
4319	1408
6906	1408
6639	1408
3993	1408
5447	1408
7075	1408
6999	1408
1085	1408
2785	1408
937	1407
7113	1407
6180	1407
4042	1407
2926	1407
7210	1407
6529	1407
3295	1407
6631	1407
5490	1406
6954	1406
6580	1406
7009	1406
6430	1406
5556	1406
2673	1406
5706	1406
6045	1406
7324	1406
7171	1406
771	1406
6751	1406
5448	1405
6029	1405
585	1405
6583	1405
5869	1405
538	1405
4764	1404
7096	1404
5991	1404
5840	1404
7204	1404
5225	1404
6147	1404
7194	1404
919	1403
2441	1403
6057	1403
4827	1403
2897	1403
773	1403
6195	1402
87	1402
6600	1402
6213	1402
6372	1402
6711	1401
647	1401
7270	1401
6599	1401
6821	1400
4801	1400
5902	1400
3458	1400
6049	1400
7330	1400
5848	1400
3714	1399
6987	1399
1827	1399
6012	1399
7032	1398
5605	1398
6552	1398
5027	1398
6136	1398
6971	1397
6021	1397
6308	1396
3773	1396
5500	1396
539	1396
617	1395
6688	1395
6863	1395
6739	1395
4840	1395
7242	1395
5532	1394
5065	1394
352	1394
6592	1394
5197	1394
5705	1394
4390	1393
7215	1393
5888	1393
6096	1392
6893	1392
6563	1391
6862	1391
7311	1391
7057	1391
3340	1390
7024	1390
5954	1390
1554	1390
3049	1390
7000	1389
7073	1389
4816	1389
7020	1388
7042	1387
4902	1387
7259	1387
5732	1386
6301	1386
151	1386
6139	1386
6018	1385
5666	1385
6715	1384
7292	1384
3532	1384
5231	1384
6345	1384
2151	1383
4929	1383
6541	1382
4784	1381
6114	1380
6505	1380
3110	1379
6528	1379
6633	1378
5642	1378
6714	1378
6162	1377
6895	1375
5651	1374
5070	1374
4949	1373
4382	1373
322	1373
3237	1372
1973	1371
6428	1371
136	1370
3586	1369
6426	1365
6152	1365
5828	1363
4195	1361
6611	1360
6707	1353
5664	1348
6595	1346
Reply With Quote
  #14   Spotlight this post!  
Unread 08-29-2018, 09:21 PM
SoftwareBug2.0's Avatar
SoftwareBug2.0 SoftwareBug2.0 is offline
Registered User
AKA: Eric
FRC #1425 (Error Code Xero)
Team Role: Mentor
 
Join Date: Aug 2004
Rookie Year: 2004
Location: Tigard, Oregon
Posts: 552
SoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond reputeSoftwareBug2.0 has a reputation beyond repute
Re: paper: Miscellaneous Statistics Projects 2018

Quote:
Originally Posted by Caleb Sykes View Post
Here are the matches that had the biggest comeback or upset:
PNW District SunDome Event qf 2 2
Well, I guess a DQ after time expires is one way to get a big point swing.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 08:17 PM.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2018, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi