Open Challenge: Make a better FRC ranking algorithm

[groves]

I had the chance to see the 1189 (gearheads) Gearhead Crossing Scouting system a couple weeks ago at the FORD FIRST invitational. I thought it was a neat way to rank teams individually. The system has a fantasy robotics feel to it. I think it would be neat to be able to rank robots on the multi-national level all throughout the season. The major paradigm shift would be that defensive robots would tend to score low (because they do not score in the conventional method). Six or officials would be in charge of watching all of the matches each watching a robot. Filling out a performance report each match for your robot.

[KenWittlief]

Originally Posted by Ricksta121

The best rules in FIRST are rules that are black and white. What kind of commitee decides what is defensive and what is offensive? This is definatley a gray area. ...

If the rules were black and white the manual would be one page long.

I think it would be fairly easy to classify a robot. Using last years game as an example: if your robot has a mechanism that can launch the ball into the air, it is a centergoal shooter.

If it has a way to capture, hold or store balls, and dump them into the side goals, it is a dumper.

If all your robot can do at best if push balls around with a flat surface, with no captive edges on the 'plow', then it is purely defensive.

[Jack Jones]

After reading through some incredibly bad ideas - mine included - I think we should leave well enough alone.

[artdutra04]

I got it! How about an algorithim that allocates for cross space-time continuum flucuations as a function of seismic activity in China, over the squareroot of the temperature in Kelvin of the ambient temperature of the playing field plus two times the fourth devivitive of two times pi times one over x to the nth power, where n is the number of years that the team has existed, and x is the team number. To this, we subtract four times the third root of the loser's score while adding one fourth of the winner's score. To account for experience differences, we divide this by the combined age of the alliance drive team times minus the number of matches they have played. From all of this, we add this onto unix epoch and divide by the combined gracious professionalism quotient of the participating team, to get the new SUCK ranking score - Scoring that's Uberly Complicated Karma.


While discussing hypothetical changes to the rules is never a bad idea, let's not make everything more complicated than it needs be. Some things in life are best left plain and simple, like vanilla ice cream, and the FIRST scoring system should be one of them.

Life is not fair, and Dean always reminds us of that every chance he gets. Sure rookies and even veteran teams may get shafted occasionally by the random matches or the scoring algorithim, but that's life. Sometimes in life you will get the short end of the stick, no matter how much you try to prevent it. What better way is there to learn life lessons than in FIRST?

[Lil' Lavery]

Originally Posted by KenWittlief

If the rules were black and white the manual would be one page long.

I think it would be fairly easy to classify a robot. Using last years game as an example: if your robot has a mechanism that can launch the ball into the air, it is a centergoal shooter.

If it has a way to capture, hold or store balls, and dump them into the side goals, it is a dumper.

If all your robot can do at best if push balls around with a flat surface, with no captive edges on the 'plow', then it is purely defensive.

That isn't true. 116 HAD a shooter, yet we never used it. 56 used their shooter, yet they still typically would score as many or more points in the corner goals. 1139 had a shooter, yet only scored 6 (or maybe 9) total points in two matches on archimedes, as they spent almost all their time defending. 48 used their shooter for a majority of the Championship event, yet during the eliminations, they replaced it with a hunk of metal and become a defensive bot. Welcome to the fuzzy grey area.
And regarding an earlier point about rookies you made. While a majority of rookies are not extremely competative, there are several notable exceptions each year. 1731 built a fully functional camera-guided turret and shooter this year, finished as the #1 seed at the largest regional in America (VCU), and if 414 doesn't DQ their alliance in SF1-2, then they would have advanced AT LEAST to the VCU Finals. 1902 was the #1 seed at Lonestar, a finalist at UCF, and a finalist on Archimedes. 1816 won Midwest and was the #6 alliance captain on Archimedes. Etc.

[KenWittlief]

Originally Posted by Lil' Lavery

That isn't true. ....

The purpose of a handicap/rating system isnt to makes everyones final scores all come out exactly even

the purpose would be to take a step towards leveling the playing field.

On average rookie teams do not field very competitive robots. If a rookie team does expectionally well, they deserve to win.

If a team won a regional last year, we expect them to do even better this year, and be a championship contender.

If you put a shooter on your robot, and you do a poor job on the design, so it cannot score many points, then you will be one of the poorly designed robots in the shooter class, and your score will reflect that.

Having a handicap system will allow HS-only teams with little funding and no mentors to focus on a simple defensive robot, and at least have some chance of making it to the finals.

Look at it this way: you are seated in the top 8, and you need to pick two robots. You want one shooter and one defensive bot. With the handicap system you have more incentive to pick a good purely defensive bot, than to pick a team that tried to do everything and does nothing well, just to play defense. You would have more incentive to pick a rookie team, if that means your alliance has a better handicap rating.

Leveled playing field.

[Tom Bottiglieri]

Originally Posted by Cory

This may be true. But the reverse happens all the time. Teams who do end up in a good position who aren't good enough.

That's life?
Yeah, sure it would be great if we could have some greater being rank and order teams in magnitude of "how good they really are," but when was the last time a national championship was won by someone who didn't deserve it? Remember, strategy and luck are part of the game just as robot design is.

[Lil' Lavery]

Originally Posted by KenWittlief

The purpose of a handicap/rating system isnt to makes everyones final scores all come out exactly even

the purpose would be to take a step towards leveling the playing field.

On average rookie teams do not field very competitive robots. If a rookie team does expectionally well, they deserve to win.

And a veteran with an even better robot doesn't?

Originally Posted by KenWittlief

If a team won a regional last year, we expect them to do even better this year, and be a championship contender.

If you put a shooter on your robot, and you do a poor job on the design, so it cannot score many points, then you will be one of the poorly designed robots in the shooter class, and your score will reflect that.

But what about a team like 56 in 2006? They had a well above average shooter, but they decided to play as a "hybrid" and both shoot and dump. What "category" would they fall under? Or a team like 522 who had an excellent shooter, but played even better defense and ended up playing primarily defense? Because defense typically only requires a strong drivetrain, moderately sized frame, and a robust robot, it is often very easy to combine with any design, which creates problems with your system.

Originally Posted by KenWittlief

Having a handicap system will allow HS-only teams with little funding and no mentors to focus on a simple defensive robot, and at least have some chance of making it to the finals.

1139 made it to Einstein, that was the first award they ever won. 1731 made one of the best rookie bots ever in a barn. The playing field is plenty level. Many of the most creative and dangerous bots come out of these lower budget teams.

Originally Posted by KenWittlief

Look at it this way: you are seated in the top 8, and you need to pick two robots. You want one shooter and one defensive bot. With the handicap system you have more incentive to pick a good purely defensive bot, than to pick a team that tried to do everything and does nothing well, just to play defense. You would have more incentive to pick a rookie team, if that means your alliance has a better handicap rating.

Leveled playing field.

Or pick a robot that doesn't even move to get a ton of extra points....

[Rick]

Originally Posted by Lil' Lavery

EVERYTHING Lil' Lavery has been posting

Lil' lavery and tom, you guys are right on. I was in class and could not defend myself...

But think about it this way Ken. This might be too logical for some people.

Your on a rookie team that just defeated 71 or 111 or 469 or (other great team with a history of doing well). How great would you feel if you won that in a fair match? Pretty good huh. But what if you win since you get X amount of handicap. You would like the trophy, but wouldn't feel like you earned it.

You make a robot, you analyze the weaknesses, you make it better. Adding handicap to the mix would make it so innovation and change would stop. Teams with subpar robots just think "handicap will take care of our faulty X". You strive to be better every year. Goals are usually to improve on the knowledge from last year. What's next, everyone gets the same amount of trophies?

If your robot is not performing as well as other teams, instead of adding to scores, teams should realize they need to innovate and change, not default to a ranking system.

WIth handicap teams have no reason to improve since it would not matter. This is the basis of my argument.

[KenWittlief]

And a veteran with an even better robot doesn't?...

part of the idea of a handicap system is that you are playing against yourself, your teams past performance, just as much as you are playing against other teams.

If veteran team has done very well in the past, and they dont do even better this year, then they are sitting on their lauraels. Thats not what engineering is about.

If a team trys to design a robot that does everything, and they dont have the experience and resources to pull it off, then they have made a bad engineering decision.

Science and engineering - thats what we need to keep the focus on. If Im working alone as an engineer, and I decide Im going to put Microsoft out of business by writing a better operating system, then I have set my sites on an unreachable goal. No matter how good of an engineer I am I will never be able to take down Microsoft by myself.

But I might design a simpler product, or take on a smaller project, and create something that is pure genius in design and execution. Who would you give an engineering award to? A company that turns out a so-so operating system year after year

or an individual who creates something that is perfect in its function and form and design for its intended purpose?

I know we already have engineering design awards, and quality awards, but with some type of team handicap system we could open up the competition itself and make the contest more level for all the teams.

WIth handicap teams have no reason to improve since it would not matter.

maybe I need to create an example of how a handicap system would work. Your handicap would be based on your actual scores in last years events. The handicap would be based on the average of all the teams in your class (rookie, intermediate, veteran...) and the level you chose to play at (defensive, offensive, auton modes...). If you are a rookie team this year, your handicap would be based on the average of all the rookie teams last year.

a handicap would bring the average team in each class up to an equal footing for this year. That means, if you were average last year, and you are average this year, you will rank somewhere around 50% at an event. If you were below average your handicap would be higher, but if you play below average this year you will not win.

But if you were average last year, and you go home with a drive for improvement, and your performace this year is twice as good as before, you have a good shot at ranking high in the competition.

[Steve W]

What happened to the algorithms. I thought that this was going to be an idea thread not what it has become.

[Tom Bottiglieri]

Originally Posted by Steve W

What happened to the algorithms. I thought that this was going to be an idea thread not what it has become.

Let X be equal to how teams have always been ranked. Let Y be equal to their new ranking.

Y = X

[Heretic121]

Im gonna go soccer/footbal style...

Win's - Lose record = most important... win = 3 pts, 2(or 1) for a tie... 0 for a loss...

Ties are determined by the average points your that team has scored over thier qualifaction matches. Highest average = 1st place... if they happen to be tied... use either points scored against that alliance or head to head wins/loses...

easy... simple... not much change...

no more QP/RP stuff...

[Lil' Lavery]

Originally Posted by KenWittlief

part of the idea of a handicap system is that you are playing against yourself, your teams past performance, just as much as you are playing against other teams.

If veteran team has done very well in the past, and they dont do even better this year, then they are sitting on their lauraels. Thats not what engineering is about.

If a team trys to design a robot that does everything, and they dont have the experience and resources to pull it off, then they have made a bad engineering decision.

Science and engineering - thats what we need to keep the focus on. If Im working alone as an engineer, and I decide Im going to put Microsoft out of business by writing a better operating system, then I have set my sites on an unreachable goal. No matter how good of an engineer I am I will never be able to take down Microsoft by myself.

But I might design a simpler product, or take on a smaller project, and create something that is pure genius in design and execution. Who would you give an engineering award to? A company that turns out a so-so operating system year after year

or an individual who creates something that is perfect in its function and form and design for its intended purpose?

I know we already have engineering design awards, and quality awards, but with some type of team handicap system we could open up the competition itself and make the contest more level for all the teams.

I don't beleive there is a FIRST team that doesn't already strive to get better and better each year though. Sometimes you don't accomplish that, and that's part of life. It's quite hard to follow up a Regional win with a Division win, and a Division win with a Championship win, etc. That's why you see so few teams do it, which is why the few that can are so special. The fact is, you HAVE to be better to stay at the same level of success in FIRST without a system to force it. Because FIRST is growing, in order to win the same awards and get the same ranking etc, you have to beat more teams, which means being even better. A system like this isn't "leveling the playing field", it's rigging it against the historically successful teams.


I don't particularly think that the current system is broken. If I was going to change it, the only thing I'd do is add some more tie-breakers (I never like "coin flip" scenarios, as are always used as the eventual tie-breakers in every major sport and FIRST). Maybe Head-to-head record, or average points scored as additional tie-breakers after RP and Max Score. Although, I don't beleive that anyone has ever been tied after Max Score anyway...

[Andrew Blair]

Not to 'jack this thread, but the scoring system is quite fair, simple, and well practiced. What causes far more upsets and hard feelings is, as was said before, the alliance algorithm.

Our team has seen both ends of the problem. We've had phenomenal matching with 90% of matches with at least one strong robot, placed high accordingly, but perhaps not deserved the position.

Inversly, we've had regionals where every match seems like a disadvantage- and often is.

We've also been in one small regional where we played with 433 three matches in a row, and yet only played in matches- allied or against- with 3/4 of the teams there. If alliances are distributed evenly, consistently, a team will recieve a much more accurate ranking. There should be no "free rides" or teams who must serve as the primary machine every match.