Real Week 2 update

[Boltman]

Originally Posted by plnyyanks

Another important thing to mention here is that this year's volunteer training very heavily emphasized only running the MatchMaker algorithm once and only once. And if you ran it a second time, you had to explain your reasoning on Skype. The reason for this policy is to avoid the exact kinds of bias we're discussing - because randomness is the fairest decider of them all.

I wonder which setting they are using?

Quality The algorithm allows the user to specify a desired quality. This simply determines the number of schedules that are generated and evaluated in the simulated annealing algorithm, as specified in the list below.

Fair: 100,000
Good: 750,000
Best: 5,000,000

If they are running only one perhaps they should choose "Best"

[Pat Fairbank]

Originally Posted by Boltman

I wonder which setting they are using?

Quality The algorithm allows the user to specify a desired quality. This simply determines the number of schedules that are generated and evaluated in the simulated annealing algorithm, as specified in the list below.

Fair: 100,000
Good: 750,000
Best: 5,000,000

If they are running only one perhaps they should choose best.

I'm reasonably certain they already run it on "best", having played around with the standalone program and been an official scorekeeper in the past. "Best" mode takes several minutes to run, while "Fair" only takes a few seconds. The schedule generation step in FMS also takes several minutes.

[Citrus Dad]

Originally Posted by plnyyanks

Another important thing to mention here is that this year's volunteer training very heavily emphasized only running the MatchMaker algorithm once and only once. And if you ran it a second time, you had to explain your reasoning on Skype. The reason for this policy is to avoid the exact kinds of bias we're discussing - because randomness is the fairest decider of them all.

First I disagree with the assertion that randomness is the fairest decider, especially when we never get to a full probability distribution which would allow schedules to balance out over a one or even several regionals. Second, these algorithms are in fact not completely random. They have inherent patterns that are difficult to erase.

I believe it would be simple add the bin constraint--I doubt that it would have to be run more than once. The only added step which is quite easy is to assign the teams to the separate bins.

As for leaving scheduling to the judgement calls of the officials, I think that's fraught with danger. For example, how should we have treated 5136 at CVR? They had made it to Einstein in 2014 yet they are only a 3rd year team. I think a transparent algorithm solves the problem the best.

[Citrus Dad]

Originally Posted by Alan Anderson

The programmer of the 2007 match algorithm believed the same thing. Whether or not it is correct doesn't matter much to the results of implementing that belief: with fewer than about 60 teams at an event, the number of times a given team plays with or against a specific other team skyrockets. This is exactly what you're trying to avoid, isn't it?



Unless you have an extraordinarily large pool of teams to draw from, "all of the other constraints" conspire to cluster teams into essentially the same groupings over and over again. What problem were you trying to solve again?

No, we're not trying to avoid playing with the same teams repeatedly. We're trying to avoid matches in which one alliance is over strong vs the other simply due to the nature of the composition. We should not have been paired with 254 or 1671 AT ALL at CVR, yet we played with both of them. Excluding such pairings is more imperative than playing with as many teams as possible.

Another option to use a sliding scale of point ratings for each team, and ensure that each alliance has a total that exceeds a minimum value. That would allow for more flexibility in alliance composition. Again an imperative to ensure that every alliance has a minimum level of competence. The flip side of having overly strong alliances is less important if each alliance surpasses a minimum.

I'm interested in what the problem was with the 2007 schedule? What criteria were they using?

BTW, I think it's interesting to see that it's only the NFL that uses strength of schedule. The other leagues have so many games (80+) that each team is able to play each other multiple times. (But I will note that the NBA changed its playoff qualification instead as a means of correcting a regional misbalance in schedule strength.) The NFL season is more akin to FRC with relatively few matches compared to the number of competitors.

[Boltman]

Originally Posted by Citrus Dad

No, we're not trying to avoid playing with the same teams repeatedly. We're trying to avoid matches in which one alliance is over strong vs the other simply due to the nature of the composition. We should not have been paired with 254 or 1671 AT ALL at CVR, yet we played with both of them. Excluding such pairings is more imperative than playing with as many teams as possible.

Another option to use a sliding scale of point ratings for each team, and ensure that each alliance has a total that exceeds a minimum value. That would allow for more flexibility in alliance composition. Again an imperative to ensure that every alliance has a minimum level of competence. The flip side of having overly strong alliances is less important if each alliance surpasses a minimum.

I'm interested in what the problem was with the 2007 schedule? What criteria were they using?

BTW, I think it's interesting to see that it's only the NFL that uses strength of schedule. The other leagues have so many games (80+) that each team is able to play each other multiple times. (But I will note that the NBA changed its playoff qualification instead as a means of correcting a regional misbalance in schedule strength.) The NFL season is more akin to FRC with relatively few matches compared to the number of competitors.

Thanks Citrus Dad for voicing from the top tier what I witnessed in my two years of analyzing. I think FRC can do better and that would possibly allow a new generation to break through to the necessary experience of a true national World Class competition. I have a feeling that is why its so hard for young teams to break through as captains..usually they get in as second ,third or backup because if multiple powerhouse qual pairings get skewed the randomness skewed its awfully hard to remain #1 in day 2 to have pick of the litter we would have picked 1678 not to discredit 254 or 973 or 1323 and many other worthy teams...just 1678 dominated by never losing and proved they could get 1 ball auto and scale. Plus pretty much un-blockable with arguably the fastest cycle times at CVR.. very close was 254 but no scale so (-2 HG equivalent)

I was happy to see in another regional a 5000# captain did break through not sure if that was powerhouse laced or not though..awesome to see.

All I know is going to worlds our first year is the main reason WHY we are so good now as a young third year team. It made an impression on us we will not forget we did not like the taste of finishing 90th or so there. The world championship experience is something every team should have as I believe it helps young teams to become that next generation. You see what it takes and witness first hand the best. I'm just glad CA offers every year a world class regional experience within driving distance.

[plnyyanks]

Originally Posted by Citrus Dad

The solution is in fact quite easy. It's a two-step scheduling algorithm that accounts for last season's performance. I believe that all of the professional sports leagues already do this. First, you sort the teams in bins A, B & C based on a ranking criterion--probably a system akin to the district point system. (Rookies = 0). Each match is scheduled by drawing a team from each bin for each alliance as an added step. All of the other constraints then come into place.

Originally Posted by Citrus Dad

First I disagree with the assertion that randomness is the fairest decider, especially when we never get to a full probability distribution which would allow schedules to balance out over a one or even several regionals. Second, these algorithms are in fact not completely random. They have inherent patterns that are difficult to erase.

I believe it would be simple add the bin constraint--I doubt that it would have to be run more than once. The only added step which is quite easy is to assign the teams to the separate bins.

As for leaving scheduling to the judgement calls of the officials, I think that's fraught with danger. For example, how should we have treated 5136 at CVR? They had made it to Einstein in 2014 yet they are only a 3rd year team. I think a transparent algorithm solves the problem the best.

I think this is a noble goal, but the devil will be in the implementation details. Done poorly, we'll be even worse off then we are today. People old enough will remember the 2007 "Algorithm of Doom", which selected teams from three buckets like you're proposing (although those buckets were populated by team age).

The biggest issue will be correctly populating the buckets - this thread discusses many of the same ideas. You'll also have to account for buckets that aren't of equal size (by whatever metric you choose), especially when the number of teams attending is not a multiple of 3. Would this cause more surrogates to be used, or something else?

Are you proposing using previous year's data, because that runs into trouble with fast team turnover and the potential for powerhouse teams to have down years, only to be paired with other powerhouses more often the following year. Plus, what do you do with rookies? They have no data to rely on yet, and there are always exceptional rookie teams, which would cause the same kind of inconsistency.

If you use current season data, what would you do for early events? You wouldn't have any data available for the early weeks, so you'd have to resort to a randomness-based algorithm again. This approach would only work for District Champs/World Champs, where all teams are already guaranteed to have played already.

Plus, Ed Law made a great point in the other thread I linked:

Originally Posted by Ed Law

I enjoyed the discussion with others on this topic but this is probably my last post on this subject. The reason is I have changed my position on this. I no longer think we should use strength in the algorithm. I was reminded that any rules no matter how well it was intended can have undesirable consequences. Think about the ranking points rule of 2010 that leads to 6 vs 0. If teams know that strength is being considered in the algorithm, the strong teams may start sandbagging during the season in hope of coming out stronger than their statistics show in the post season. This will destroy the game.
The only effect a tough schedule has on a team is they may not be alliance captains because they don't get enough wins. However if the team is indeed good, the OPR numbers and scouts will recognize that. Even if they rank low, they can still be number one pick. So it is not as bad as it seems.

So overall, I'll concede that random selection isn't perfect, but I'm not totally convinced a team strength based algorithm is better.

[Boltman]

Originally Posted by plnyyanks

I think this is a noble goal, but the devil will be in the implementation details. Done poorly, we'll be even worse off then we are today. People old enough will remember the 2007 "Algorithm of Doom", which selected teams from three buckets like you're proposing (although those buckets were populated by team age).

The biggest issue will be correctly populating the buckets - this thread discusses many of the same ideas. You'll also have to account for buckets that aren't of equal size (by whatever metric you choose), especially when the number of teams attending is not a multiple of 3. Would this cause more surrogates to be used, or something else?

Are you proposing using previous year's data, because that runs into trouble with fast team turnover and the potential for powerhouse teams to have down years, only to be paired with other powerhouses more often the following year. Plus, what do you do with rookies? They have no data to rely on yet, and there are always exceptional rookie teams, which would cause the same kind of inconsistency.

If you use current season data, what would you do for early events? You wouldn't have any data available for the early weeks, so you'd have to resort to a randomness-based algorithm again. This approach would only work for District Champs/World Champs, where all teams are already guaranteed to have played already.

Plus, Ed Law made a great point in the other thread I linked:


So overall, I'll concede that random selection isn't perfect, but I'm not totally convinced a team strength based algorithm is better.

I think you simply never allow repeats and treat world champions that got there "as captains or strong world class second bots" or those that won say 4 regionals in past 6 years slightly different from the remaining pool. I think you let teams like 5136 slide and remain in the greater non-powerhouse pool.

Those "powerhouse" teams should not be paired as a "dream alliance" in Quals at all and possibly face each other to provide more entertainment than just running up the score against three way younger teams scratching and clawing for the worlds only to likely get slaughtered (and drop 2 RP each )

Lets say an event such as CVR there are SIX teams that meet that criteria (with four wildcards not a stretch) so you put SIX fake teams in the algorithm as placeholders...you then analyze each fake entry if it look fair balance with other fake entries you then replace the real powerhouse teams randomly in...for the others you check other fake team place holders if you see a discrepancy shift a pool team in and make their games placeholders...rinse repeat. Until all six teams are more evenly sorted. Then randomly assign powerhouses to each fake entry.

-OR_

Re-run the algorithm until the fake placeholders look balanced is another way. Then enter the real teams again randomly for each fake placeholder.

Look at CVR.. #1 really took it to #5 in Finals ...no surprise to anyone there and #5 was very strong offensively but younger and not as many creds. I'm fairly certain we would have been a similar story as #2...that is how good #1 was.
It really can be like NE always getting to play Cleveland due to lack of young teams at #1 close but not quite at least in stacked regionals. In SD yes a rookie was #1 and got beat in QF SD only had 1 super class bot 399. Our alliance #8 took them out in 2 games because we were ranked low #19 due to missing RP's for breaches not credited. (Should have been rank 10 or 11 and captain) fine with us worked out not so fine for rookie #1 alliance (although they rightfully earned RAS)


Would treating powerhouses slightly different change the end result?
Who knows but it may have. Same with valid RP sorts.

FRC can do better IMO..lots of brains there figure it out.

You are never going to get truly random but you can try to get more balanced at the top tier dream team pairings and facing repeat low win probability having to face the same powerhouse team that beat you more than once especially when paired with another.

[Citrus Dad]

Originally Posted by plnyyanks

I think this is a noble goal, but the devil will be in the implementation details. Done poorly, we'll be even worse off then we are today. People old enough will remember the 2007 "Algorithm of Doom", which selected teams from three buckets like you're proposing (although those buckets were populated by team age).

The biggest issue will be correctly populating the buckets - this thread discusses many of the same ideas. You'll also have to account for buckets that aren't of equal size (by whatever metric you choose), especially when the number of teams attending is not a multiple of 3. Would this cause more surrogates to be used, or something else?

Are you proposing using previous year's data, because that runs into trouble with fast team turnover and the potential for powerhouse teams to have down years, only to be paired with other powerhouses more often the following year. Plus, what do you do with rookies? They have no data to rely on yet, and there are always exceptional rookie teams, which would cause the same kind of inconsistency.

If you use current season data, what would you do for early events? You wouldn't have any data available for the early weeks, so you'd have to resort to a randomness-based algorithm again. This approach would only work for District Champs/World Champs, where all teams are already guaranteed to have played already.

Plus, Ed Law made a great point in the other thread I linked:


So overall, I'll concede that random selection isn't perfect, but I'm not totally convinced a team strength based algorithm is better.

I've thought about many of these issues, the first is that it has to rely on past year data, not current so Ed's concern is not relevant. (The one exception might be for the Champs, but then teams can't get there by sandbagging.)

I want to take this conversation off line for those interested in addressing this. I have an idea for changing the scheduling process that would both incorporate strength factors AND make it easier. I'm particularly interested in what happened in 2007 as that was before my involvement. Please message me directly if you want to discuss this.

[Liam Fay]

Originally Posted by Boltman

SD only had 1 super class bot 399.

I wouldn't say that that's true at all. Just off the top of my head, 1266, 2486, 2102, 3255, and, though I may be biased, 2485 have all been to champs on multiple occasions.

Broader, though, it's not FIRST's job to make sure everyone gets to go to champs. The tech world that FRC prepares us for is cutthroat. Not everyone gets equal opportunity.

[Boltman]

Originally Posted by Liam Fay

I wouldn't say that that's true at all. Just off the top of my head, 1266, 2486, 2102, 3255, and, though I may be biased, 2485 have all been to champs on multiple occasions.

Broader, though, it's not FIRST's job to make sure everyone gets to go to champs. The tech world that FRC prepares us for is cutthroat. Not everyone gets equal opportunity.

Those are REALLY good perennial teams I agree but I put 399 above then based on their long consistent history. Starting to approach other CA giants.

Congrats to 2102, 3021 and 2486 for winning SD I'll be watching you in worlds 2485 made it too with a well deserved award. 5805 was super sleek awesome RAS. 2687 got the WC...awesome job San Diego teams. 3255 was very solid until they broke down at the wrong time they should be a favorite in their next event

Actually I'm glad Champs are hard makes you hungry every year.....