Reducing Luck in FRC Competitions

Sorry I wasn’t clear. It’s always difficult to convey the direction with these “strength of schedule” metrics. Lower percentages indicate that the given team will have more trouble ranking better. Assuming my number has value, a team would tend to prefer higher percentages, as that would indicate their chances of getting a better rank have increased. So 67 did have the “worst”/“hardest”/“least favorable”/“best opponents/worst partners combo” schedule of any team in my list.

Also the percentages do their best to isolate a team’s schedule from that team’s performance/skill level. It’s kind of comparing that team’s current schedule to a bunch of random alternative schedules and finding out how good this one is relative to all the others. My TBA articles go more in depth explaining it.

8 Likes

This will come as a surprise to nobody in Michigan. And not to very many folks elsewhere.

10 Likes

This metric can also vary in usefulness across events. For example, here are the predicted average ranks I had at Milford before the schedule was released compared to the teams’ actual final results:
image

After the schedule was released, here were my predictions again compared to the actual results:
image

The R^2 value ticks up just a hair, but not really enough to be very noteworthy. This would indicate to me that the pre-match schedule strengths I calculated for this event probably didn’t matter particularly much.

Let’s compare that to another Michigan event that same week in Kingsford. Here are the predicted ranks versus actual ranks at Kingsford before the schedule was released.
image

This R^2 value was much lower than for Milford, indicating the seed values I used for teams were not as relevant here as they were in Milford. Here’s what it looks like after the Kingsford schedule was released though:
image
The R^2 is still drastically lower than that of Milford, but look how much it improved on the pre-schedule predictions. To me, this indicates that the team seeds are still worse than Milford’s, but that the schedule strength predictions were much better here, as my ranking predictions improved noticeably when they knew what the schedule would be.

Looking at these graphs, it is very strange to me that the predictions seem more confident generally before schedule release than after. I don’t know how to explain that, I feel like it should do the opposite. I wonder if that’s a bug or just some explanation I’m not thinking of right now.

1 Like

Could it be that any large differences between predicted strength and actual strength of a team get compounded? If a team’s predicted strength is way off (say it’s way too high), then that error gets used many times over in the SOS calculations and subsequent predictions. It’s like a copy of a copy, where the data just gets fuzzier and fuzzier.

I would be concerned that the fact that there are alliances would generate too much noise in a swiss tournament because if you group all of the “good” teams together they are also playing with each other which counters the fact that they are supposed to be playing against each other.

Have you/ could you simulate a swiss tournament and compare how the predicted rankings reflect elo rankings vs how a standard simulated predicted rankings reflect elo rankings?

Someone who isn’t busy working right now can you please thoroughly scout their match videos on TBA so we can lay this to rest. Or someone who was at the event who can share their actual scouting data…

1 Like

@tmichals, you keep mentioning two separate changes here:

a. Balancing individual matches so that each alliance has a more equal chance of winning.
b. Balancing schedule quality so that rankings more accurately reflect robot performance.

You seem to think that (a) also accomplishes (b). Do you understand that this is factually incorrect? Do you understand that these are actually direct opposites of each other? If not, I think you’re not critically thinking hard enough about what you’re saying.

(though the Swiss system that Caleb mentioned does seem to accomplish both…)

12 Likes

835 was also not picked for a playoff alliance (same as 7220) so what is the point here? Obviously the match schedule can affect win-loss records but unless your team is close to ranking in the top 8 the win loss record is irrelevant and how you preformed to scouts is what is important. Alliance selection is how the system allows teams who are ranked low but performed well to continue to playoffs. The system does not need to force an equal win-loss record on each team all it needs to do is allow each team to display their abilities and the teams who continue to playoffs are the teams that demonstrated abilities that the top 8 ranked teams desire for their alliance.

9 Likes

Expanding on this - ranking systems need to be designed around the boundary conditions.

Which team ranks 24/40 or 30/40 is far less important than ensuring that the right team ranks in 8 vs 13.

The fact that they ranked a few slots higher than you but both well outside alliance captain range is pretty much meaningless. And interestingly the fact that you both went unpicked really shows that the rankings (putting you both decidedly in the bottom half of teams at the event) seemed correct.

4 Likes

Really? Were meaningless? I guess the FRC should only be for the top teams… my mistake, you got me.

He didn’t say you were meaningless, a minor ranking difference between teams outside of alliance captain range is meaningless.

9 Likes

Ranking level means a lot to us and our kids.

I looked at a few matches just to to get a better take.

Your robot had a nice shooter, intake was okay. Cycle times were slow. Lining up to shoot took a long time and if defense was played it hurt you.

Reviewing match scores there were 4 matches where if your team would have climbed and all else remained the same your alliance would have won. You would now be 9 wins 3 losses.

That would have generated a minimum of 8 additional rank points 16 total. That would have moved you from being ranked 32nd to being ranked between 12th and 16th. If one of those would have been a balanced climb for a rank point you would have been ranked 11th or 12th at 19 rank points.

In my opinion your team team designed a nice robot that met two objectives of the game. It drove and it scored well in the upper goal.

From a strategic stand point (my opinion only) you missed the most important aspect of the game, climbing. Climbing in the last 20 seconds and scoring 25 points was equal to 8.3 inner shots or 12.5 outer shots. I did not review all matches, but I would be very surprised if you scored 25 points in teleop in any match, most teams did not.

My understanding is you are saying you had a poor draw for alliance partners that bad luck of the draw caused your team to be ranked 32nd despite the fact you had a good shooting robot.

My take is you didn’t build a climber. If you would have built one that worked, your team would have been ranked in the top 16.

Is being ranked in in the top 16 or being ranked 32nd a matter of luck?

20 Likes

My point was to give you guys an example that the current scheduling system does not work. The climbing was not a factor for our two robots. We want to answer only one question, does the current scheduling system work? If it did matches like I am showing you would not happen.

Reaching for something to be offended by…

The difference between those Ranks is meaningless for the purposes which the ranking system is designed to serve. Whether you ranked 24th or 40th is only really of value to you and does not assist in solving the problem the ranking system should be designed to solve.

17 Likes

The purpose of the ranking system is to define who the alliance captains are. Nothing more.

The purpose of the scheduling system is first, to make sure teams play an even amount of matches at a relatively even pace through the day. This is enforced before attempting to balance paring uniformity.

19 Likes

As someone here already stated, HQ really should come up with a single optimized schedule to use across all events of a given # of teams. Optimized the schedule on match turn around time and opponent/partner coverage. Then randomize the slots in that schedule that the teams go into.

15 Likes

I was hoping you would have noticed one of the matches included team 67, who got an easy win out of the deal. 164 to 78. Still I completely understand why most of you love the current system. Sure the top teams will have some tough matches(LOL), but having it this way also means they have at least a chance for some cake walks.

If you want to help make our ternaments something that everyone will want to watch, then you have to make it so it is worth watching. When an NFL game is 61 to 0 in the first half people will just turn it off.

1 Like

You’re mistaking understanding and accepting the system for liking it.

Understanding systems is a key part to advocating for change. I think you lack both an understanding of the systems and the reasons they exist.

You seem to have a solution in mind already without understanding the problem.

16 Likes