How is matchmaking done at competitions? (Kind of a rant)

That’s it, they don’t want a “group of death” scenario

And I would say, the 1st conversation is kind of silly until we first get point 2 addressed. If you try to add in another criterion to the current algorithm, you will be sacrificing quality in the other criteria.

3 Likes

Interesting. I just re-read the articles, and I couldn’t find the objective stated as clearly in the articles as you’ve stated it here. Thanks for providing it clearly here!

However, I must still be misunderstanding something, as I thought that the approach described in the third article on TBA (https://blog.thebluealliance.com/2019/02/18/schedule-strengths-3-of-3-generating-balanced-schedules/) requires a “before the event” ranking order of all the teams, from best to worst, using some sort of “strength” metric.

At least that’s what my take-away was from the sentence below:

I haven’t really thought about it why I feel this way, but I’ve been avoiding any match-scheduling solution which requires a “ranking” of the teams going in to the event, as I don’t think such rankings can be generated accurately, and there would be protests of “bias” on any such algorithms not ranking their team appropriately.

On a related thought, though, if the schedule truly is “balanced” so that all teams have the same relative strength of partners and opponents as all other teams, would it also be the case that if one were to invert the “seed order” in one of your match schedules, the schedule would still be just as balanced? If that’s the case, then one doesn’t actually need to rank the teams before the event, as the schedule is “balanced” even with an upside-down sort of teams.

I’d like to see schedules that are “balanced” for a random assignment of teams to a schedule, without requiring an “a priori” sort of teams to be used as seeding going into the event.

3 Likes

This is correct. I don’t see how you could possibly create schedules better “balanced” for team strength than random if you don’t come in with a sorted list of team strengths. That seems like a given to me, so if people are not comfortable with that (which I can understand), then these kind of schedules are a non-starter.

Oh certainly, I’m aware I’m just trading one problem for another :grinning: You’d at least get me on the side of defending the schedules though.

This property exists in my schedules

You would still need to “rank” or “seed” the teams somehow going into the event, it just doesn’t matter if you sort the list ascending or descending.

I don’t understand how this is different than the current system.

1 Like

Sounds like chicken vs. egg problem.

How do you balance quality of partners vs. quality of opponents without knowing the quality of the teams?

You certainly can’t rank rookie teams with perennial Einstein teams like 1678 and 2056.

This is the post about comparing matchmaker and cheesy arena schedules, irrespective of any strength of schedule balancing. Paper: Miscellaneous Statistics Projects 2018. This was posted by Calen in Post 51

One issue in 2007 was that age was taken as a proxy for team capability, which we all know is a very weak indicator. It wasn’t so much about facing strong vs weak rookie teams as it wasn’t really any better than the previous assignment method.

But was worse was that the assignment constraint led to teams being matched multiple times with the same teams. That’s a much worse outcome than the current scheduling method.

Unfortunately that’s not statistically possible with only 8 to 12 matches per competition. To even approach a statistically balanced outcome, we would need upwards of 30 matches and preferably 100 (which is why professional baseball, hockey and basketball schedules work reasonably well.)

If you accept that we can rank teams by capability/ability, which you apparently do by highlighting 1678 and 2056, then we have the ability to rank about 90% of the teams based on first, the previous year’s results, and then the updated results of the current year’s events. Caleb has developed an Elo ranking system that might work in this situation. But keep in mind that teams often change character from year to year, whether they are rookies or not. The ranking system cannot be precise or completely accurate for this reason. You then have to accept that having some type of ranking system, so long as it is demonstrably accurate in the broader sense, is better than no ranking system whatsoever.

As for rookies, who typically represent about 10% of the contestants in a year, we can either accept them as generally low ranked and that the exceptional ones are part of the normal variation, analyze where rookies have typically fallen among the rankings in previous years, or use another metric to identify the rookies more likely to be successful, such as a lower number in that vintage. For example I noted that a bunch of 5800 series teams in the West were particularly good in 2016 (and have continued to succeed). That year started with 5800.

This is a case of being willing to not make the perfect the enemy of the better.

Brian
I was typing in haste and did not make my point well. I was wondering if the initial seed i.e. first round matches were looked at by a human. If a real outlier came up, say 3 Hall of Fame teams vs three rookies, would said human say…hmmm, lets go with a different randomization that gets round one of the event off to a better start than a 108 to 6 drubbing! After that I concur, the model evens everything out and is as it should be for all.
TW

Have you ever noticed how no one ever posts saying, “We got great match pairings that resulted in us ranking much higher than we deserved. I think there is something wrong with the algorithm!”

The alliance selection process is the great equalizer for random alliance pairings… in the algorithm of death year, when we played against 1114 four times in one competition that meant that we had four chances to impress them with what a great alliance partner we would be.

The fact that we failed to make such an impression can hardly be blamed on the match scheduling algorithm or our alliance partners. :slight_smile:

It is human nature to feel slighted and blame others for our problems. But it is not the best part of human nature. By definition a random schedule will sometimes be slightly in your favour and sometimes slightly not. But if you’re not having competitive success at tournaments, perhaps there are more significant factors than an algorithm at play.

7 Likes

You’ve missed my posts where I’ve pointed out that we won the Curie Division in 2013 due to having a very favorable schedule. In fact it was so favorable that we set a record for “graciously declining” with 4. And I’ve pointed out how it was unfair to other teams. And last year, an alliance captain on our division field arrived there solely through the schedule. Its public knowledge that none of the other teams that would be in position to be alliance captains would accept their bid and their first pick had to be a team that had no choice but to accept.

I disagree that this balances out in the alliance selection process in two ways:

First, many highly capable teams are denied being alliance captains because of the teams that get there solely by being carried through its schedule. This is a lost opportunity for those teams, and too often they are overlooked by the undeserving alliance captains that are rarely prepared to develop a good draft list.

Second, teams that are beat down in the schedule more often are unable to fully display their capabilities because they have to fill roles in weak alliances to which they may not be well suited. No top team is scouting by watching only the matches that the team is playing in, and I almost never hear my drive coach come back talking about the teams that we played against (vs. the teams that we played with).

This is not really problem about blaming others for their misfortune. Clearly our team has been able to overcome most scheduling misfortunes (and every year since 2015 we have faced more difficult than average strength schedules in our divisions at Champs.) This is about wanting to see teams that worked hard to achieve what they can at competitions to do so without facing scheduling impediments. As I have pointed out, we do not play enough matches in the formal season, even over a student’s career, to statistically arrive at a balanced outcome where everything evens out. We will have to intervene to arrive at the more balanced outcomes.

I can say that my current team (1712) benefited from the 2007 algorithm in the Galileo division in 2007, to the tune of Ranking 6th in the division, picking (then rookies) 2056, and captaining an alliance to the semi-finals.
But they also weren’t my team at the time, and I certainly complained about my then team’s (116) schedule on Galileo that year for having 11 playoff opponents and only 1 playoff partner (although it’s 8 if you count that 116 played in each of 116’s matches).

I think you can get a more accurate estimation of rankings, as long as there is made public information about each team.

So if we had a way to gauge a veteran team’s likely change in expected performance, say like the amount of senior students that are gone, the amount of returning students and years experience, the amount of new students, key returning mentors with years experience as well as key leaving mentors with years experience.

Maybe also add in change in donations and machinery.

I agree team strength should be taken to effect when designing match schedules. I also agree that there should be a premade match schedule for n teams. n being any number in the interval [24,75] or so. There are still lots of questions to be answered, namely what to do about week 1 or 2 competitions with teams that haven’t competed yet.

Some points of info based on different things said in this thread. As a Regional FTA that also does off-season events, the MatchMaker utility is something I use offline as well (and have used it for things not robot competition based as well). It allows me to manually enter various parameters versus what I see when FMS runs it. The base utility is the same, but FMS feeds specific parameters based on the event size and field configuration. Since many off-seasons are pretty small, you have to override some of the defaults to make it work at all.

When you run it manually, you tell it how many teams, how many matches per, minimum matches between appearances, number per alliance, surrogates into what round, quality of schedules (how many to generate before starting to score them), optionally a team number list to use AFTER everything else is done, and most recently a mirroring of the alliance stations (to deal with the Steamworks field being a mirror and not symmetrical).

That second to last item is one of the most important ones. It doesn’t know ANYTHING about who is in a competition at all. It doesn’t care. It makes the schedule using numbers 1 to N. It might as well use color names or pictures of unicorns (my 7 year old would love this). Only after the best scored schedule of the 5 million it generates (using best quality) has been determined to the criteria it has been told to use, does it actually fill in a team number into each of the slots, randomly. Argue all you want about “random”, but by the time it gets to this part of the process, there isn’t a whole lot you can do to force a specific team(s) to be in a specific slot(s) of the schedule beforehand. It doesn’t care that 254, 1114, 1678, 2056 are all at the same as the rookie team that literally brought all of their kickoff stuff to the event, unopened, because they thought that was how it worked. They are treated equally. Any “bias” seen because you have examined 6 events worth of results is BECAUSE you only examined 6 events worth of results and not thousands.

When the FMS runs it, the FTA/Scorekeeper does not have ANY input into the parameters it will use. We hit a button that says Run MatchMaker and it then tells us how long the estimate is. When it spits out the results, the statistics can be looked at to see if there are any interesting outliers. If something looks really weird (why is one team paired up with only 6 unique other teams in a 12 match per team, 40 team competition when it should be 24?), we contact our support to also look and if they also determine it to be weird and needs correction, we then have to call Frank before going back and running it again. Since there is NOTHING else we can do with the schedule we now have (cannot edit teams, cannot edit match order, NOTHING), the next step is the click the Activate button, which pushes it out to the OFFICIAL public server at firstinspires.org and the various NON-OFFICIAL sites then pull it via the API (Spider, The Blue Alliance, etc.).

Again, we cannot edit the schedule at an event. It can either be published as created or recreated. Each time it is run, it is recorded as such and a rerun can only be granted by HQ themselves after serious discussions as to why you feel the need to. Having to do that once, due to a team really not showing up after loading in their robot the night before, is something I do not want to have to do again since I understand how much work many teams go through in (over)analyzing what their schedule is the literal second it hits the public space.

With all of the turnover you have within teams over the years, as well as turnover of just teams, making a schedule that just “doesn’t care” and literally just treats them as a number is, in my mind, the simplest solution. That is what we have now and really hope it doesn’t change. The MatchMaker program is freely available for your use. Run thousands of iterations and “statisfy” the results over time, you know…like what was done for its creation.

If you are at Arizona West in a couple of weeks and want to discuss, I will GLADLY sit down over a “soda” as this tool is one of the things I love to talk about for some reason… I’ll be the guy in the dark blue shirt/khakis at the field. You might need to break me out of an FRC Karaoke session though (this years playlist has me singing at the field… “Star Trekking Across the Universe”…)

11 Likes

Come on, at one match schedule a milli-second it would only take just under 2*10^45 years to check them all for a 40 team event. We would of course have to re-run it for each value of N

1 Like

I’m the driver on a rookie team and I think that for our first event, we got quite a hard match schedule. The event only had 26 teams, so playing against the same team more than once was kind of unavoidable. We played against 1305 3 times and 1325 twice. For alliance selection we were the third pick for alliance 7. We were mainly a defense robot, but we had a higher OPR than both of our alliance members. https://www.thebluealliance.com/team/7757

Easy solution take previous Regional winner captains/Rank 12 or higher and if early previous championship captains and highlight them on the schedule run it until an even distribution of each facing each other , same for rookies … simple. The others just fall wherever.

We faced our toughest quals lineup we’ve ever seen at Greater KC this weekend, facing #1 seed twice (and never with), and #2 seed twice (and once with). In fact, our first match was against #2 + #3, and our second match was #1 + #2. And it was 36 teams with 12 quals matches each.

To be clear, I’m not posting this to whine. We performed amazingly well even though we still ended up #23 of #36, owing not just to the lineup, but bad luck (6 out of 12 matches, we were a bot down in some way), and the fact we had no hab 3 climb (our own decision and our own fault). But I can say that our lineup wouldn’t have been fair to anyone and I wouldn’t want to see any other team face it either.

However, I’m 100% behind a static pre-generated set of quals matches with teams randomly slotted in. Especially if these are calibrated to be well balanced. I think it would work out best for everyone.

1 Like

The work of determining match separation, quality, etc. is 100% done by match maker. No volunteers can say “I want lower/higher quality matches this weekend.”

This disagrees with the Matchmaker White Paper, which says "The algorithm allows the user to specify a desired quality. " It may be that First has taken the decision away from the volunteer. If so, that’s not described by the whitepaper.

But, that leads to another question: why is quality ‘determined’ by matchmaker? Why is it not set to ‘best’ every time?

The schedule is determined by these inputs:

  • Match cycle times (time between starting match X and match X + 1)
  • Start/end times each day
  • Breaks (e.g. lunch)
  • Number of matches per team

It isn’t really the schedule that’s the problem; it’s the match assignments and their ordering. And, that does not depend on the first three of those bullets. It depends entirely on (a) the number of teams and (b) the number of matches. The set of {a,b} pairs is sufficiently small that “the” match ordering for {a,b} can be determined in advance with teams randomly assigned into the pre-determined schedule at the event. If, for whatever reason, a team withdraws, then just go to {a-1,b}. (Realistically, b would probably change also.)

There is evidence that the current scheme is not working as well as it could. At an event this past weekend, we were opposed to the same team 4 times, another 3 times and two others 2 times, but never in an ailiance with any of those teams. If “pairing uniformity” is a goal of the algorithm, it failed, at least in our case.

2 Likes