Open Challenge: Make a better FRC ranking algorithm

[KenWittlief]

Originally Posted by Lil' Lavery

I don't believe there is a FIRST team that doesn't already strive to get better and better each year though. ...

A system like this isn't "leveling the playing field", it's rigging it against the historically successful teams.


I don't particularly think that the current system is broken. ...

have you ever been on a team, or worked with a team that had six HS students and only one teacher for a mentor? A team that had nothing but a couple handtools, and ended up building their robot out of PVC pipes and plywood?

Do they have any chance against teams with $100k in funding, an huge corporate sponsor, and ten engineer mentors? Seriously? No. At best they end up trying not to be in the bottom 5 of the ranking at a regional.

We keep telling students that engineering is all about what happens between your ears, how well you following the design cycle / process, how well you think things through

but the competition is already rigged, its rigged in favor of the big veteran teams that have lots of resources

that doenst mean that little 7 person team didnt try just as hard as everyone else. They may have pulled 48 hour weekends, and worked all year to raise funds. They may have been excellent in their design process, but you can only do so much with PVC and plywood against teams with CNC machines, a full scale machine shop, and a team of mentors at your disposal.

Go back to my example of sailing. I may never be able to buy an F40 catamaran (40 foot hulls), but I do have a 16 foot hobie cat. If I can sail my boat to its absolute limit, then why shouldnt I be able to compete in an open class event, against the 40 foot cats. I will never be able to sail faster then they can, but that doenst mean I am not the best sailer.

Sailing commitees came up with a rating /handicap system that allows exactly that - open class competition, where each skipper has a chance to win an event if they are the best sailor, with the boat they have.

We already have open class competition, but the way things are now the little guy has no real chance.

So why cant we come up with a scoring / ranking system for events in which all teams have some hope of placing well? Is it too hard for us to figure out?

What causes far more upsets and hard feelings is, as was said before, the alliance algorithm.

Our team has seen both ends of the problem. We've had phenomenal matching with 90% of matches with at least one strong robot, placed high accordingly, but perhaps not deserved the position.

Inversly, we've had regionals where every match seems like a disadvantage- and often is...

this is another problem that a well devised handicap system would fix. If you are allied with a small team that historically has not done well, or with a small rookie team, they would have a higher handicap rating, that would tend to even out the match.

[Tom Bottiglieri]

Originally Posted by KenWittlief

have you ever been on a team, or worked with a team that had six HS students and only one teacher for a mentor? A team that had nothing but a couple handtools, and ended up building their robot out of PVC pipes and plywood?

Yes.

Do they have any chance against teams with $100k in funding, an huge corporate sponsor, and ten engineer mentors? Seriously? No. At best they end up trying not to be in the bottom 5 of the ranking at a regional.

If you call the #4 alliance captain at the NJ regional "in the bottom 5," then yes.

Maybe the big powerhouse teams are inspiration to everyone?! I know I sure was motivated to do better after seeing 45, 217, 111, and 25 in 2003 at nationals. Would that have happened with the communist ranking system? No.

[KenWittlief]

Originally Posted by Tom Bottiglieri

Yes.

If you call the #4 alliance captain at the NJ regional "in the bottom 5," then yes.
...

if a team accomplished that then they are truely exceptional.

If you average the rankings of all the very small teams with limited resources, I dont think that winning a top 8 seat will be anywhere close to typical.

Inspiration is a separate issue. We are discussing ways to possibly make the scoring and ranking system better.

For years we have all accepted that FIRST is not fair. Big teams have a clear advantage. Are we unwilling to even consider a way to level the playing field?

What is everyone afraid of?

[Tom Bottiglieri]

Originally Posted by KenWittlief

For years we have all accepted that FIRST is not fair. Big teams have a clear advantage. Are we unwilling to even consider a way to level the playing field?

What is everyone afraid of?

Its not so much that we dont want things to change or feel that it would be bad, we just dont see a point in it. Yes, some teams are successful from year to year, but the high school kids are constantly changing. How can you explain to new team members that they just lost a match on the book which they rightfully won? Why is right to "take from the rich and give to the poor"? Some teams are much better, but FIRST needs that. Even if there was a handicap, some teams would find ways to exploit it. So no, I don't think there should be a handicap, nor should we consider it.

[Cory]

Originally Posted by Tom Bottiglieri

That's life?
Yeah, sure it would be great if we could have some greater being rank and order teams in magnitude of "how good they really are," but when was the last time a national championship was won by someone who didn't deserve it? Remember, strategy and luck are part of the game just as robot design is.

The rankings don't have anything to do with this. The issue is number of matches

The real question here should be "How do we get teams more matches at nationals?" (Most regionals already get enough matches to have the cream of the crop at the top). There's absolutely nothing wrong with the ranking system. If it aint broke don't fix it. Another more worthwhile issue is improving the random match generator, cuz it's never worked so well.

[Tom Bottiglieri]

Originally Posted by Cory

The rankings don't have anything to do with this. The issue is number of matches

The real question here should be "How do we get teams more matches at nationals?" (Most regionals already get enough matches to have the cream of the crop at the top). There's absolutely nothing wrong with the ranking system. If it aint broke don't fix it. Another more worthwhile issue is improving the random match generator, cuz it's never worked so well.

I totally agree with both of those point. And no I'm not being a hypocrite!

[efoote868]

perhaps these arguements would be under a different category like "year to year" team rankings, and shouldn't make much of a difference to the "here and now", competition rankings. Please don't take this the wrong way, but I wouldn't want to see a completely uneven match in any of the qualifying rounds, because of a robot that isn't exceptional, put there because of its special circumstances.

Besides, the rating system does what it is supposed to do, and that is to seperate the robots from "best" to "worst". further more, if that doesn't do the trick, then the alliance selection does.

oh, high school students only stick around for 4ish years, so the "powerhouse teams" are really just the mentors

[Rick]

Originally Posted by Tom Bottiglieri

Maybe the big powerhouse teams are inspiration to everyone?! I know I sure was motivated to do better after seeing 45, 217, 111, and 25 in 2003 at nationals. Would that have happened with the communist ranking system? No.

Tom your the man.

But anyways...
The competition is the end of the journey. The journey is where the learning happens, and seeing others at competition inspire innovation for others and new thinking. Keep it simple, lets have some fun, and lets not make competition stressful with complicated ranking.

[Donut]

Originally Posted by efoote868

oh, high school students only stick around for 4ish years, so the "powerhouse teams" are really just the mentors

This would be the killer to any kind of handicap system, to me at least. If you're going to tell me I'm going to be penalized because last year's group of kids did really well (and since our team tends to have a rather large student and teacher turnover every year or 2, we wouldn't have any of the "same resources"), I would personally not want to compete that year.

We all know the competitions are not about winning it. There is, what, three awards based on competitions results? How many are there that aren't? I'm personally very happy to get ANY kind of award, so I don't see a reason for making the "victory" award any easier to get. If I win it yet I didn't have to perform nearly as well as *insert-really-good-team-here*, I wouldn't feel like I've earned it much.

[JaneYoung]

Originally Posted by efoote868

oh, high school students only stick around for 4ish years, so the "powerhouse teams" are really just the mentors

Apologies in advance for going off topic -

The powerhouse teams have many things going for them -
stability can be a key factor regarding location for build, mentors that commit their time and energy, and students who develop an organized system of training and being trained in an on-going manner. Off seasons are excellent tools for developing in all the areas the teams need to prepare for competition season. I do agree that the mentors on the powerhouse teams are important - they are incredible leaders. They are also wise ones in that they understand the process of developing new leadership, new growth, new ideas in order to continue to improve as teams competing in FIRST. Remarkable things can occur in 4 years - and with thought and preparation, beyond.

[MikeDubreuil]

If I were a game designer…

My goal would be to have great offensive matches. I would award winning teams. I also think teams should be rewarded for consistently scoring well despite some lost matches.

I would keep the Ranking Points system mostly intact:
2 points for a Win
1 point for a Tie
0 points for a Loss

Your RP score would be summed the entire day and divided by matches played. This is an important scoring metric because at the end of the day you must be able to beat your opponents.

You would receive an “RP Ranking” based solely on your RP score against the rest of the field. Ties would be allowed.

There would be a second ranking list of teams sorted by average match score. This is simply a running average of their average match score. Ties would be allowed.

Finally, I would average a team’s rank in the RP Ranking list with their rank in the Average Match Score list to create there Overall Rank.

Originally Posted by RP Rank

RP Rank		Team Name	RP Score
1 		Team M		  8
    2		 Team B		  6
    3		 Team D		  2
    4		 Team Z		  0

Originally Posted by Average Match Score Rank

MS Rank  Team Name  Match Score
   1	   Team M	 53
   2	   Team B	47
   3	   Team Z 	 45
   4	   Team D 	 20

Originally Posted by Overall Rank

MS Rank		  Team Name
  1		 Team M
     2		 Team B
     3		 Team Z
     4		 Team D

As you can see from the "Overall Rank" Team Z benefited most from my method. They scored high points but just couldn't seem to win matches. This probably had something to do with not so great alliance partners. Team D did not benefit at all. Their low match score points and decent RP points indicate that they were probably carried by their alliance partners.

I think this way is exciting because each match you're not just playing your opposing alliance you're against the whole regional.

EDIT: Tie breakers for computing overall rank are by priority:average match score, # of penalties, electronic coin flip.

[Daniel_LaFleur]

Originally Posted by KenWittlief

if a team accomplished that then they are truely exceptional.

If you average the rankings of all the very small teams with limited resources, I dont think that winning a top 8 seat will be anywhere close to typical.

Inspiration is a separate issue. We are discussing ways to possibly make the scoring and ranking system better.

For years we have all accepted that FIRST is not fair. Big teams have a clear advantage. Are we unwilling to even consider a way to level the playing field?

What is everyone afraid of?

Level the playing field??? That is nothing but a euphemism for telling someone that they are not good enough so we're going to give you some points.

I have been on a small, underfunded, under mentored team for the last 3 years. I am now switching to another small, underfunded team. Do I want your leveled playing field? NO! It would be an empty victory if I won by less than the "bonus" points I got because I wasn't "good enough".

FIRST already has a level playing field. There are no advantages on the court, except what you have built into your robot. And this is how it sould be.

What is needed is for the veteran teams to reach out to the newer teams and offer the support that the newer teams need (now, before the regionals). Most new teams don't know that there is support out there, or where to ask. THIS is where FIRST should work to 'level the playing field', not on the court.

I'm sorry Ken, but while I feel your heart is in the right place, your conclusion is wrong.

The above is, as always, IMHO.

[Nawaid Ladak]

hey guys, it's been a LONG time seince ive been on this website.

but yea, now to the algerathums

the point system should work like this.

as a ranking system

ie

Match 1
Team A: 12
Team B: 13
Team C: 9
Bonus Points: 10
Red Total Score: 44

Team D: 15
Team E: 18
Team F: 3
Bonus Points: 5
Blue Total Score: 41

(one robot for the blue, and two robots for the red made it up to the ramp in time)

Match 2

Team A: 33
Team C: 6
Team E: 12
Bonus Points: 10
Red Total Score: 61

Team B: 20
Team D: 3
Team F: 12
Bonus Points: 25
Blue Total Score: 60

(All three robots for blue made it up to the ramp, while two robots made it up to the ramp in time.

HERE WE GO.

Team A scores 33+12+5pts(ramp Match 1)+5pts(ramp Match 2)=55pts

Team A's alliance total score was 105 so...
55/105 x100= 52.381

Team A's Opponants total score was 101 so...
52.381/101 x100= 51.862

Total Rating: 51.862

Team B scores 13+20+5(ramp Match 1)+8.334(ramp Match 2)=46.334pts

Team B's alliance total score was 104 so...
46.334/104 x100= 44.552

Team B's Opponants average score was 102 so...
44.552/102 x100= 43.678

Total Ranking: 43.678

Team C scores 9+6=15pts

Team C's alliance total score was 105 so...
15/105 x100= 14.286

Team C's opponants total score was 101 so...
14.286/101 x100= 14.145

Total Ranking 14.145

Team D scores 15+3+8.334(ramp Match 2)=26.334pts

Team D's alliance total score was 101 so...
26.334/101 x100= 26.073

Team D's Opponents total score was 105 so...
26.073/105 x100= 24.831

Total Ranking 24.831

Team E scores 18+12+5(ramp, Match 2)=35

Team E's alliance total score was 102 so...
35/102 x100= 34.314

Team E's Opponants total score was 104 so...
34.314/104 x1000= 32.994

Total Ranking 32.994

Team F scores 3+12+5(ramp, Match 1)+8.334(ramp, Match 2)= 28.334

Team F's alliance total score was 101 so...
28.334/101 x100= 28.053

Team F's Opponents total score was 105 so...
28.053/105 x100= 26.717

Total Ranking: 26.717

Standings
1 Team A (2-0-0) 51.862
2 Team B (1-1-0) 43.678
3 Team E (1-1-0) 32.994
4 Team F (0-2-0) 26.717
5 Team D (0-2-0) 24.831
6 Team C (2-0-0) 14.145

How do you guys like that

[Donut]

Point based ranking systems are very dangerous to have. Any kind of system that is based off of how much you score typically promotes "running up the score" on your opponents, and with your system it would also promote inter-alliance competition (because you would be ranked higher if the rest of your alliance scored less). This is why most high school sports leave margin of victory out of ranking systems, because it promotes the teams winning 65-0 to run the score up to 100 instead of putting in their 3rd string and having some kind of mercy.

I'd hope this wouldn't be a problem in FIRST with GP and whatnot it supports, but I could still see this brewing into an issue.

[Cory]

Originally Posted by Donut

Point based ranking systems are very dangerous to have. Any kind of system that is based off of how much you score typically promotes "running up the score" on your opponents, and with your system it would also promote inter-alliance competition (because you would be ranked higher if the rest of your alliance scored less). This is why most high school sports leave margin of victory out of ranking systems, because it promotes the teams winning 65-0 to run the score up to 100 instead of putting in their 3rd string and having some kind of mercy.

I'd hope this wouldn't be a problem in FIRST with GP and whatnot it supports, but I could still see this brewing into an issue.

Not to mention the fact that it'd be impossible to keep track of, unless the scoring is not real-time, and the scoring items are color coded by team.