Suggestion to improve the alliance choosing program

[Matt H.]

It is incredibly hard to make any alliance sorting program based on team skill simply because teams don't preform based on any known pattern. Past performance is no indication of current success (mentors leave, students graduate etc.) and there are some rookie teams which absolutely shine. Also the more variables inputted into a sorting program the more likely teams are to be paired with/against each other again and again and again as there become fewer and fewer "fair" combinations.

With this years game I am against the alliance system completely as penalties could prevent a powerhouse team from winning making alliance partners a risk with many teams having negative average scores, but I can't think of a remedy so I'll have to live with it.

[Bongle]

I crunched numbers on this in 2006 based on 2 weeks of regionals, and here's an image that breaks down scores by 'average alliance number'. Average alliance number was the average of the 3 team numbers that made it up. Note that rookies this year were about 1700+, so a team with an average alliance number higher than that was probably all rookies.

There is definitely some correlation between team number and scores, but it is a fairly weak correlation, and more importantly, there is a LOT of variation in each group. There are rookies who can dominate regionals (2056 in 2007), and there are rookies who can barely get their robot to move. However, there are also older teams like this.

http://www.chiefdelphi.com/forums/at...1&d=1142127063

Edit: Going through all my old statistics threads is fun.

Here's another relevant one. Given two alliances, find their average alliance numbers (AAN1 and AAN2). The x-axis on this graph is the difference between opposing alliance's AANs. If a team like (1114, 1503, 1680) faced (25, 48, 71), then the difference would be something like 1200ish. This graph shows the win rate for the higher-numbered alliance.

Basically, it says that in 2006, if your AANs differed by 1200, then the team with the higher number had a 20% win rate.
http://www.chiefdelphi.com/forums/at...8&d=1142707377

[Rick TYler]

Originally Posted by Bongle

I crunched numbers on this in 2006 based on 2 weeks of regionals, and here's an image that breaks down scores by 'average alliance number'.

Time to go back and increase your data set size, add 2007 and 2008 data, and include a Z-axis with number of teams in each band. Good work!

[Kimberly]

You guys are overlooking something. There's another reason it's beneficial to not have rookie teams on the same alliance. As rookies, we're learning about all aspects of FIRST. We learn the most from experienced teams. It seems to me the mentoring aspect FIRST promotes throughout the build phase, would be appropriate for the competition phase too. On an alliance with 1 rookie team, there are 2 experienced teams to offer help, guidance, strategy, etc. I don't see a down side to this.

While it's certainly possible for a rookie team to outperform many experienced teams, I think it's still in the best interest of the organization overall for rookie teams to get the benefit of what more experienced teams can teach them during that first year. The more experienced alliance partners a rookie team has, the more information it receives on how to be even better next year.

[EricH]

Originally Posted by Kimberly

You guys are overlooking something. There's another reason it's beneficial to not have rookie teams on the same alliance. As rookies, we're learning about all aspects of FIRST. We learn the most from experienced teams. It seems to me the mentoring aspect FIRST promotes throughout the build phase, would be appropriate for the competition phase too. On an alliance with 1 rookie team, there are 2 experienced teams to offer help, guidance, strategy, etc. I don't see a down side to this.

While it's certainly possible for a rookie team to outperform many experienced teams, I think it's still in the best interest of the organization overall for rookie teams to get the benefit of what more experienced teams can teach them during that first year. The more experienced alliance partners a rookie team has, the more information it receives on how to be even better next year.

I don't see how that will help. Remember, FIRST is not about the competition or the robots; it's about the people.

Also, I can think of at least one veteran team off the top of my head that could use some on-field mentoring themselves. They aren't exactly in a position to give advice. You wouldn't know it to look at their number--and the number is what the algorithm uses.

Personally, I'd rather see the return of the design books.

[Dr Theta]

I am unsure as to how something like this would work for some of the younger regionals as well, I mean look at Minnesota this year. Over half of the field is rookie teams.

[Kimberly]

Ok, let me explain it this way: If a rookie team were on 10 alliances with 20 different experienced teams, that's 20 sets of data. The rookie team can decide for itself which advice is useful and which is not, but the more times the same advice is given, the more likely it is to be valid. More information is better than less.

I also am well aware it's not about the competition and winning, which is exactly why I'm suggesting the rookie teams be paired with 2 experienced teams during the competition. If I were promoting a better winning strategy, I'd suggest teams be seeded by individual performance, but I personally don't care about that, except to the extent of keeping track of our individual performance so we know how our design and strategy worked.

If the algorithm were changed to include the fewest pairings of rookie teams possible, and to balance the rookie distribution between the competing alliances, it wouldn't matter what percentage rookies were at the competition. It would only mean there wouldn't be alliances where experienced teams were competing against inexperienced.

[EricH]

Originally Posted by Kimberly

If the algorithm were changed to include the fewest pairings of rookie teams possible, and to balance the rookie distribution between the competing alliances, it wouldn't matter what percentage rookies were at the competition. It would only mean there wouldn't be alliances where experienced teams were competing against inexperienced.

I don't know how easy that would be to implement for next year. (It's too late for this year.) I think it won't be that easy.

Take a look at the 2007 match lists, if you can find any. (The Blue Alliance probably has them.) You will see almost exactly that situation. The hard part will be keeping the other teams from facing each other more than once or twice. Last year's algorithm was the most hated in FRC history. So you want the "third tier" to be made up only of rookies and only one other tier. That can't be easy to do. If you think it is, then I invite you to come up with an algorithm and submit it to FIRST for their use.

[XaulZan11]

Originally Posted by Matt H.

Past performance is no indication of current success (mentors leave, students graduate etc.)

I understand what you are saying, but if you talk to nearly every psycologist, past behavior is the best predictor of future behavior. If there was no correlation or indication, then we would expect teams like 71, 111, 233, 1114 to have a normal distribution of results (ie win 3 regionals in a year just as often as not getting picked for the eliminations in one year). As we know, however, these teams always are some of the top teams. I think you mean that the correlation is not strong enough to be a used. If so, I agree.

[Bongle]

Originally Posted by XaulZan11

I understand what you are saying, but if you talk to nearly every psycologist, past behavior is the best predictor of future behavior. If there was no correlation or indication, then we would expect teams like 71, 111, 233, 1114 to have a normal distribution of results (ie win 3 regionals in a year just as often as not getting picked for the eliminations in one year). As we know, however, these teams always are some of the top teams. I think you mean that the correlation is not strong enough to be a used. If so, I agree.

Again to go back to the stats I did a few years back, a team's seeding performance in one year is a very poor predictor of its performance in the year following.

Here's a graph:http://www.chiefdelphi.com/forums/at...6&d=1175831673

On the X axis is a team's seeding performance in 2005. Further left is better. On the Y axis is a team's seeding performance in 2006, lower is better. You'll see about the only thing you can predict is that teams who were top seeds in 2005 tended to not be dead last in 2006. Likewise, teams who did very poorly in 2006 tended to not win the following year (but some did). Past behavior predicting future behavior may work well in humans, but not so much in robotics teams.

The teams that do well year after year are very special cases. Out of the 1500ish active teams in FIRST, people can probably only name 50ish 'power houses' who win year after year after year and never hiccup.


On the main topic:
Keep in mind that a team's next-year performance will probably be modified FAR more by who they communicate in the pits with, rather than who they play with on the field. If you play 8 games, you're only on the field for 16-20 minutes, but you're at the regional in the proximity of other robotics teams (whether in the hotel, pits, stands, or fields) for 72 hours. The 71 hours and 40 minutes that you're not on the field are where your entire team can learn from vets, not just on-field.

[XaulZan11]

Originally Posted by Bongle

Again to go back to the stats I did a few years back, a team's seeding performance in one year is a very poor predictor of its performance in the year following.

Here's a graph:http://www.chiefdelphi.com/forums/at...6&d=1175831673

On the X axis is a team's seeding performance in 2005. Further left is better. On the Y axis is a team's seeding performance in 2006, lower is better. You'll see about the only thing you can predict is that teams who were top seeds in 2005 tended to not be dead last in 2006. Likewise, teams who did very poorly in 2006 tended to not win the following year (but some did). Past behavior predicting future behavior may work well in humans, but not so much in robotics teams.

The teams that do well year after year are very special cases. Out of the 1500ish active teams in FIRST, people can probably only name 50ish 'power houses' who win year after year after year and never hiccup.

I never said it was a fantastic predictor. I said it was the best. Predicting behavior in humans is very challenging.

The graph that you provide may very well prove that past preformance does predict future behavior. It appears the correlation coefficent would be around .2 or .3 and with a large sample size (around 1,000) I would not be surprised to see the correlation statistically significant. This means that the observed relationship is not due to random varation but because the two samples (results from 2005 and 2006) are in fact related.

Although it is not a perfect correlation or relationship (nearly all relationships in life are not perfect) doesn't mean there is none. It may not be a very strong relationship, but it appears there is one. I cannot think of another relationship (team number, funding...) that is a better predictor than past preformance.


To tie this back into the original topic, there is not perfect predictor for team preformance. Unless someone does a huge multiple regression study and finds a way to predict how teams are (I don't think there is one), the best way is to just randomly assign teams under the perameters (such as time inbetween matches) to ensure the most fair pairings.

[Bongle]

It does appear that lower-numbered teams had lower-numbered partners.

Using all the week 2-5 regional results, I found the average team number of each team's alliance partners. The results:

0-numbered teams played with avg team nbrs of 1091, against avg nbrs 1080
100-numbered teams played with avg team nbrs of 1093, against avg nbrs 1095
200-numbered teams played with avg team nbrs of 1073, against avg nbrs 1076
300-numbered teams played with avg team nbrs of 1083, against avg nbrs 1061
400-numbered teams played with avg team nbrs of 1158, against avg nbrs 1157
500-numbered teams played with avg team nbrs of 1147, against avg nbrs 1200
600-numbered teams played with avg team nbrs of 1264, against avg nbrs 1253
700-numbered teams played with avg team nbrs of 1269, against avg nbrs 1192
800-numbered teams played with avg team nbrs of 1268, against avg nbrs 1249
900-numbered teams played with avg team nbrs of 1351, against avg nbrs 1392
1000-numbered teams played with avg team nbrs of 1290, against avg nbrs 1314
1100-numbered teams played with avg team nbrs of 1382, against avg nbrs 1372
1200-numbered teams played with avg team nbrs of 1398, against avg nbrs 1357
1300-numbered teams played with avg team nbrs of 1399, against avg nbrs 1436
1400-numbered teams played with avg team nbrs of 1376, against avg nbrs 1403
1500-numbered teams played with avg team nbrs of 1394, against avg nbrs 1393
1600-numbered teams played with avg team nbrs of 1461, against avg nbrs 1452
1700-numbered teams played with avg team nbrs of 1446, against avg nbrs 1444
1800-numbered teams played with avg team nbrs of 1556, against avg nbrs 1533
1900-numbered teams played with avg team nbrs of 1687, against avg nbrs 1678
2000-numbered teams played with avg team nbrs of 1522, against avg nbrs 1528
2100-numbered teams played with avg team nbrs of 1563, against avg nbrs 1574
2200-numbered teams played with avg team nbrs of 1760, against avg nbrs 1730
2300-numbered teams played with avg team nbrs of 1757, against avg nbrs 1762
2400-numbered teams played with avg team nbrs of 1713, against avg nbrs 1735
2500-numbered teams played with avg team nbrs of 1793, against avg nbrs 1799
2600-numbered teams played with avg team nbrs of 1630, against avg nbrs 1630

I'm fairly sure that unless I have a bug, this indicates that higher-numbered teams indeed don't get to play with lower-numbered teams as often as randomness would suggest, which implies that the match-making algorithm is not blind to team numbers like I would think it should be.


Here's the summing code. I grouped things by hundreds so patterns would show up and it'd be easier to implement. m is a match object, which includes the team numbers of the red and blue alliances. m.red and m.blue are 3-element integer arrays of the alliance. If I have a big glaring bug, please point it out.

for(int x = 0;x < lstMatches.size();x++)
	{
		Match m = lstMatches[x];

		iTeamWithSum[m.blue[0] / 100] += m.blue[1] + m.blue[2];
		iTeamWithSum[m.blue[1] / 100] += m.blue[0] + m.blue[2];
		iTeamWithSum[m.blue[2] / 100] += m.blue[0] + m.blue[1];
		iTeamWithCount[m.blue[0] / 100]+=2;
		iTeamWithCount[m.blue[1] / 100]+=2;
		iTeamWithCount[m.blue[2] / 100]+=2;

		iTeamWithSum[m.red[0] / 100] += m.red[1] + m.red[2];
		iTeamWithSum[m.red[1] / 100] += m.red[0] + m.red[2];
		iTeamWithSum[m.red[2] / 100] += m.red[0] + m.red[1];
		iTeamWithCount[m.red[0] / 100]+=2;
		iTeamWithCount[m.red[1] / 100]+=2;
		iTeamWithCount[m.red[2] / 100]+=2;

	}
	for(int x = 0;x < iTeamsMax;x++)
	{
		cout<<(x*100)<<"-numbered teams played with avg team nbrs of "<<iTeamWithSum[x] / iTeamWithCount[x]<<endl;
	}

[XaulZan11]

Assuming those numbers Bongle provided are correct, that is a problem. I haven't completely gone through how the pairings are assigned, but it appears the team number (or the order of the teams) is a determining factor in the pairings (if not, we wouldn't have the relationship that Bongle provided). Would it be possible to randomly assign each team a number? Like say there are 50 teams at a regional, randomly assign each team a number from 1-50. Then, after the program assigns the pairings, replace the number (1-50) with the team number.

[GaryVoshol]

Using raw team numbers as integers does not create a fair test. Some regionals have very low team numbers, with only a few rookies (Detroit). Some have very high team numbers, mostly rookies (Hawaii, Minnesota, Oklahoma). The two can't be equated directly; they skew the distribution. You would have to figure out a way to assign an equivalent rank to each team number in each regional.

[SteveJanesch]

Originally Posted by Bongle

Here's the summing code. I grouped things by hundreds so patterns would show up and it'd be easier to implement. m is a match object, which includes the team numbers of the red and blue alliances. m.red and m.blue are 3-element integer arrays of the alliance. If I have a big glaring bug, please point it out.

Bongle,

I'm not sure, but I think your code includes the numbers of all three teams on each alliance. If you're calculating the average team number of only of the teams you're paired with, wouldn't you want to keep your own team number out of the average? If that's the case, it'll flatten out the "who you are paired with" average with respect to your own team number. It won't change your opponent's average team number, though. I could easily be reading the code wrong. I thought what was important was the average team number of who you were paired with, which should ideally be constant versus team number, not the whole alliance, which your own team number will influence.

- Steve