Continuing along on my journey of building a match scheduler. I decided to dive deeper into FIRST’s MatchRater Algorithm which is used to determine schedule quality. You can download MatchRater from the IdleLoop website to see how it works yourself.
The piece of MatchRater I am most interested in is seeing the way it “scores” schedules. You can find this score yourself with the “-x” command line parameter. Based on my investigation, there are 3 score components: duplicate partner penalties, duplicate opponent penalties, and repeat appearance penalties. It evaluates these penalties for every team pair at the competition, and summing all of the penalties together gives the final schedule score, where a higher number is worse.
For duplicate partners, the base penalty for a single duplication is 5, this penalty doubles for every additional pairing after the first 2. This is summarized in the below chart:
The left side represents the number of times that two teams are partners at an event. The top represents the number of times that the two teams are opponents at an event. The numbers in the middle are a representation of the penalty for that particular combination of partner and opponent appearances.
In a similar way, there is also a duplicate opponent penalty. The base penalty in this case is 1. Here is a chart for that penalty:
Finally, there is a duplicate appearance penalty. This penalty applies if two teams are in the same match at all, whether as partners or as opponents. It also has a base penalty of 1. Here is it’s chart:
With these three charts, we can sum up the penalties at each partner/opponent intersection to get the full penalty chart:
Thoughts on this chart:
I’m truly shocked to find that this is the scoring algorithm for schedules. I’m still wrapping my head around the implications.
To start, the manual is straight up lying about the priorities of the scheduler. From section 11.6.2 of the 2021 game manual:
Priority 2 according to the game manual is to minimize duplicate opponents, and priority 3 is to minimize duplicate partners, but that is blatantly not what the scoring algorithm values.
The total penalty for the 2-partner, 0-opponent case is 6, which is lower than the total penalty of 2 for the 0-partner, 2-opponent case. Not only that, but it is even lower than the 4 point penalty for the 0-partner, 3-opponent case!
Here’s a more detailed example outlining the preference for non-duplicated partners over non-duplicated opponents:
Hypothetically, say a team has 3 matches. The scheduler can either give them:
- 6 unique partners and 4 unique opponents
- 4 unique partners and 6 unique opponents
Let’s say that none of the partners appear as opponents for simplicity.
In case 1, two opponents appear in all 3 matches, one opponent appears twice, and one opponent appears once. All partners are unique and thus all appear once. The penalties are thus:
0-partner, 3-opponents: 2 at a penalty of 4 each
0-partner, 2-opponents: 1 at a penalty of 2
0-partner, 1-opponent: 1 at a penalty of 0
1-partner, 0-opponent: 6 at a penalty 0 each
Total penalty = 2 * 4 + 1 * 2 + 1 * 0 + 6 * 0 = 10
In case 2, three opponents appear twice and three appear once. Two partners appear twice and two partners appear once. The penalties are thus:
0-partner, 2-opponents: 3 at a penalty of 2 each
0-partner, 1-opponent: 3 at a penalty of 0
2-partner, 0-opponent: 2 at a penalty of 5 each
1-partner, 0-opponent: 2 at a penalty of 0 each
Total penalty = 3 * 2 + 3 * 0 + 2 * 5 + 2 * 0 = 16
So the scheduler very strongly prefers case 1, which clearly minimizes duplicate partners and not duplicate opponents. This preference exists in spite of the fact that I am forcing unique opponents down to 4 and 6 when, over the course of 3 matches, a team could have as many as 9 unique opponents. Comparing 9 unique opponents and 4 unique partners against 7 unique opponents and 6 unique partners provides even more of a penalty difference in favor of unique partners.
To be clear, I don’t mind that the scheduler prefers unique partners in a vacuum. Every schedule is going to have to make tradeoffs, but the tradeoffs the scheduler is incentivized to make are in direct opposition to what is stated in the game manual.
My next thought is that, intuitively, I feel like the cost function should prefer a more balanced number of partner/opponent appearances more strongly than duplicate opponents and no partner apperances. If two teams are going to encounter each other 3 times, I feel like 2 matches as an opponent and 1 match as a partner should be much more strongly preferred over 3 matches as an opponent and 0 matches as a partner. But the scoring algorithm only provides a slight incentive for this (3 point penalty vs 4 point penalty). Indeed, a very common complaint I’ve seen regarding schedules comes from the 3 (or 4) opponent, 0 partner instances. The fact that this happens is not some intrinsic undesirable property of FRC schedules, but rather a natural outcome of the fact that 3 partner 0 opponent schedules are not penalized very much relative to alternatives.
Finally, I think it’s bizarre that there is no associated penalty for the 0 partner, 0 opponent case. I don’t think it should be an enormous penalty, but I don’t see why the scoring algorithm shouldn’t nudge teams toward 1 total appearance instead of 0. Essentially, such a penalty would incentivize more unique team appearances. Currently, all the schedule does is dis-incentivize duplicate team appearances, which is a similar goal, but I think we might get more unique encounters with a slight 0 partner, 0 opponent penalty.
I’ll play the game and build schedules according to this algorithm. I’m noting though that I think this is a poor scoring structure for schedules. Regardless of my opinions, section 11.6.2 should definitely be changed to reflect the actual scoring algorithm used.