FiM DTE shows why the cargo RP is a huge problem

Anyone watching DTE would agree that the rankings at the top did not reflect the actual competitiveness of many teams. 5114 and 5460, two of the absolute best teams in the state, ranked below 12, while I don’t think anyone, even 4130, would disagree that they were not the best team on the field. I think this has a lot to do with the fact that FIRST did not change the cargo RP in the last team update; DTE has a Cargo RP% of 86.25% and a Hanger RP% of 80.62%. Because the vast majority of alliances are getting both bonus RPs, ranking becomes almost entirely dependent on win-loss, losing all of the “schedule-robust” ranking that bonus RPs have helped with in previous years.

8 Likes

All those threads complaining that the first seed wins too much are going to look like ancient history soon! The remaining DCMP’s and Houston are going to be really interesting. I predict we’ll see a lot more upsets.

21 Likes

wow, rankings aren’t always representative of robot performance, this is a shocking revelation

36 Likes

Exactly :joy:

6 Likes

I prefer this over 2020’s 14.70%/0.00% RP, but that might just be me :wink:

6 Likes

Of course. No one in those threads thought the #1 dominance would carry over to champs or worlds. The fields are deep enough that the rankings will reflect the randomness of the pairing schedule.
The observation on the high alliance dominance was limited to events like 40 or less robot district type events.

3 Likes

Bonus RP’s are an opportunity for FIRST to tune the rankling system over the course of the season so rankings better reflect robot performance with a limited number of matches.

I hope FIRST takes a more disciplined approach to tuning the ranking system over the course of the progress towards champs in future seasons.

-Mike

22 Likes

It is very weird that they aren’t doing this. I just don’t quite understand the hesitancy.

4 Likes

Here is Frank’s answer in response to changing the cargo threshold on FUN: https://youtu.be/joDtE-pPgJg?t=478

Frank’s transparency has been such a welcome change but would have been nice to hear a little more about the rationale to keep it the same.

2 Likes

Maybe there should be a different limit for events that have already self selected down to a more competitive set of teams.

If hangar RP is at 80+% should it be increased also? I expect there will be more resistance to this but it seems like the statistics at least for district championship support it.

3 Likes

I haven’t seen that much indication that FIRST HQ and the GDC actually want the best teams to seed at the top and win, especially at events like DCMP and CMP itself where CMP berths aren’t on the line. When extra RPs were introduced in 2016 (ignoring 2012 for… reasons…), I interpreted them as additional goals for alliances in imbalanced qual matches where the winner is not in doubt, rather than ways to ensure that the best robot seeds #1. In fact, in 2016, the best robot often did not seed #1.

Rather, I think the powers that be are fans of chaos, or at least controlled chaos. Why would they go with a serpentine draft if they just wanted the best robots to win all the time?

2 Likes

Man, 5460 had an atrocious schedule. According to my event simulator, they had a 64% chance of being in the top 8 before the schedule was released, and only a 16% chance of being in the top 8 after. 5114 went from a 93% chance of top 8 to a 84% chance. 4130 went from a 2% chance of seeding first to a 7% chance.

Summary
Team Schedule Strength (probability that team will seed higher with actual schedule than they would with a random schedule) 1 seed probability before schedule release top 8 probability before schedule release 1 seed probability after schedule release top 8 probability after schedule release 1 seed change due to schedule top 8 change due to schedule
3655 79.2% 0.3% 21.6% 3.1% 63.4% 2.8% 41.8%
74 76.5% 0.0% 1.0% 0.0% 4.2% 0.0% 3.2%
5660 71.6% 0.0% 0.8% 0.0% 1.6% 0.0% 0.8%
4130 70.8% 1.6% 40.1% 6.5% 69.8% 4.9% 29.7%
5525 69.1% 0.0% 0.4% 0.0% 1.2% 0.0% 0.8%
5505 68.9% 0.2% 10.3% 0.3% 19.3% 0.1% 9.0%
1718 68.1% 1.8% 46.4% 4.2% 68.8% 2.4% 22.4%
2405 66.8% 0.0% 0.8% 0.0% 1.3% 0.0% 0.5%
2054 65.4% 0.0% 17.4% 0.5% 35.1% 0.5% 17.7%
3538 63.8% 25.6% 94.6% 48.4% 98.0% 22.8% 3.4%
7211 63.2% 0.2% 13.2% 0.8% 26.0% 0.6% 12.8%
3618 63.0% 0.0% 0.0% 0.0% 0.1% 0.0% 0.1%
862 60.7% 0.0% 13.1% 0.1% 16.1% 0.1% 3.0%
245 58.9% 0.3% 15.0% 0.2% 19.3% -0.1% 4.3%
3534 58.1% 2.2% 46.0% 1.7% 57.3% -0.5% 11.3%
6642 57.8% 0.0% 0.0% 0.0% 0.2% 0.0% 0.2%
7191 50.3% 0.0% 0.1% 0.0% 0.1% 0.0% 0.0%
2834 49.3% 0.4% 20.4% 0.3% 15.5% -0.1% -4.9%
4384 48.4% 0.0% 0.5% 0.0% 0.0% 0.0% -0.5%
4392 47.3% 0.3% 20.3% 0.1% 13.2% -0.2% -7.1%
5150 46.1% 0.0% 0.0% 0.0% 0.0% 0.0% 0.0%
5980 45.6% 0.0% 0.4% 0.0% 0.0% 0.0% -0.4%
2611 45.6% 0.8% 25.2% 0.1% 18.6% -0.7% -6.6%
7244 43.0% 0.0% 0.5% 0.0% 0.0% 0.0% -0.5%
3452 42.8% 0.0% 1.7% 0.0% 0.2% 0.0% -1.5%
4362 42.7% 6.4% 69.4% 4.2% 59.3% -2.2% -10.1%
3322 42.7% 0.0% 10.2% 0.1% 5.3% 0.1% -4.9%
4779 42.3% 0.0% 0.6% 0.0% 0.2% 0.0% -0.4%
5114 41.5% 21.3% 92.5% 15.8% 84.3% -5.5% -8.2%
4237 40.6% 0.0% 0.6% 0.0% 0.2% 0.0% -0.4%
1684 39.3% 0.2% 12.1% 0.0% 4.8% -0.2% -7.3%
4967 37.4% 0.1% 5.2% 0.0% 0.7% -0.1% -4.5%
1502 37.0% 0.3% 18.7% 0.2% 8.1% -0.1% -10.6%
1189 33.1% 0.7% 24.5% 0.1% 9.5% -0.6% -15.0%
67 32.4% 32.2% 95.0% 13.0% 81.9% -19.2% -13.1%
240 29.8% 0.0% 0.6% 0.0% 0.0% 0.0% -0.6%
6088 29.2% 0.0% 1.7% 0.0% 0.1% 0.0% -1.6%
6120 29.2% 0.0% 0.1% 0.0% 0.0% 0.0% -0.1%
201 20.5% 0.3% 14.9% 0.0% 0.7% -0.3% -14.2%
5460 18.6% 4.8% 64.1% 0.3% 15.6% -4.5% -48.5%
15 Likes

On purpose or a typo… hmm… :wink:

1 Like

So, for rankings to best reflect team ability, what is the optimal overall RP success rate for FIRST to target? Intuitively I feel like it should be 50%, but not sure if there is a rigorous reason why that would be the case. I feel like 50% is probably best for the rankings overall, and something lower (maybe like 30%?) would be better for sorting just the top 8 teams.

In 2016 they essentially included a warning in the game manual from day 1 saying the tower RP could change. I assume the lack of inclusion of this same clause for 2022 was part of the reason FIRST decided to forego changing it this time around. Obviously they could change anything they wanted in a team update, but still.

7 Likes

I remember a similar thing in 2017 where they removed the free gear.

I thought the probability of achieving two bonus RP’s for any given alliance was supposed to be the same as the chance of winning a given match.

I was thinking one at ~80% and one at ~30% for the CMP level. The first would be achievable by most mid-tier teams at champs and serve as a differentiator at the district/regional level. The second would serve to separate the best teams at DCMP and CMP. I believe this is roughly in line with 2016.

A bit off-topic, but why was the third match played?
image

frc-events shows QF2-1 (Quarterfinal 2) as a win for the Red alliance, so I’m guessing it was replayed and TBA didn’t get the updated match score?

1 Like