Did we have a healthy ratio between luck and performance?

Competition Rules/Strategy
,
1

In this post (2022 Championship Match Schedules - #13 by Caleb_Sykes), Caleb Sykes did a great work analyzing the “schedule strength” (the likelihood that the given team will seed higher with this schedule than they would have with a random schedule.)

what struck me was the HUGE effect of the schedule on ranking this year (see table below)

image
image996×402 27.8 KB

Of course, all the winning teams did a great job! , they would not be there without building a great robot, driving it smoothly, using creative strategies, etc… nothing here comes to disrespect the great achievement/performance of all winning teams.
And still… seeing this table raised a concern -

All number one seeds had a schedule strength higher than 70 except for 1577, which also had a “positive” schedule of 62. (any number above 50 means that your schedule was favorable)
All top OPR robots (robots with the highest OPR) in all divisions that had a schedule strength below 60 are not #1 seed.

These two facts imply that:

  1. A high schedule strength was essential for finishing quals in the first place.
  2. No matter what was your OPR / performance - if your schedule strength was below 60 then you did not finish first place.

The implication of this is that this year - the schedule had a significantly higher effect on rankings than the teams’ performance.

I believe the main cause is the combination of only 10 qual matches for 75 teams, with a VERY low threshold of 20 balls/15 hangar for RP.
There is no way for a strong robot to recover from losing RP when getting the 2 bonus RPs is so easy for all alliances.

There are benefits in adding some “noise” to the pure performance rankings, by giving some strength to pure luck (e.g. schedule) - it creates amusement, tension, and uncertainty. It also gives more teams the opportunity to win -which is a good thing.
Results in real-life also depend somewhat on pure luck. Adding this to the competition results teaches everyone that some things are beyond their control (like luck), as in the real life.
All these are important factors for such a program, and I am happy there is some randomness and luck involved.

Still - my question is - did FIRST go too far this year?
Making it more of a lottery and less of a competition? (as the table implies)

At least for me, it was a real demoralizer (still recovering :slight_smile: ) - I never go to casinos and don’t have any interest in participating in lotteries, don’t like betting on luck … I am strongly motivated by fair competition. And believe (and teach) that working hard and diligently will bring results.
I am afraid that when the end result is so strongly connected to luck in schedule, it might create demoralization in future seasons.

Not to mention the educational message of being ranked based on luck…

thoughts?

29 Likes
2

Rapid React generally did an awesome job of sorting out the top teams during Quals at local events. That would have occurred more frequently at Champs with higher RP thresholds… I agree that FIRST should plan to scale the RP thresholds in future seasons to match the level of play going into Champs.

I think FIRST was correct in not doing so this season since they didn’t reserve the right to change the thresholds in the initial game manual.

23 Likes
3

“I’d rather be lucky than good.”

Luck is always going to have an effect. Besides 27 and 67, seems like the top teams still seeded in the top 8. Imagine if it was pre-2014 with less than 10 qual matches per team.

Regarding the idea of bumping the cargo rp requirement, what would it need to get bumped to? Someone ran the numbers with a requirement of 30 and it didn’t affect the overall rankings very much.

4

There’s a couple things that are interesting about this dialogue.

  1. This was obviously not the case at qualifying-level events (regionals/districts), so clearly a system that weighted # of scored balls and climb points was effective at giving accurate rankings - the #'s just needed to change.
  2. Before the invention of the “Bonus RP”, W/L/T was considered to be the “best” (balancing accuracy and not giving teams weird incentives) ranking system. And back in those days (2013-14) the fact that the rankings didn’t match which robots were best kind of felt like part of the overall challenge - how do you win the event even though the robot seeded first is maybe the 9th-12th best robot instead of the best.
  3. Even with Bonus RP’s, some recent games have been notoriously bad at ranking teams (see: 2017) due to other issues in game design. Compared to the 2017 game, 2022 rankings seem pretty good, even at the championship. I don’t think any of the Championship #1 seeds were any worse than something like 12th in the division.
8 Likes
5

for me this just highlights the need for good scouting on a team. At the championship level what pick you are doesnt matter as much as who your alliance is. None of these team didn’t have a chance to compete on eineinstien or at least a final in their division (could be wrong here). so in that way I think it’s perfectly fair. at lower levels is where I saw it make a much bigger difference. especially for bubble teams. I’ve always liked the design strategy of 3 rp is easy the 4th you need 2 high quality robots to achieve. Just wish that values would adjust with the level of play going on. Maybe have it tiered for weeks 1-5, districts and finally champs.

5 Likes
6

Yep, I ran the numbers with 30 cargo RP and it wasn’t a dramatic change. I actually ran all numbers between 15-40, and none of them did particularly well at sorting the teams by OPR (27-40 were in the same ballpark), which surprised me.

I came to the conclusion that there really wasn’t much you could do at champs to make schedules not matter as much. The simple fact that you were playing only 10 games in a 75 team field, and this field is historically the strongest field ever, just means that even the best teams will lose games.

36 Likes
7

It is a practical impossibility for every team to play with and against every other team, therefore scheduling will always have an impact on final ranking. It’s just a fact of competition in FRC. This effect is exacerbated at events like DCMPs and WCMPs with larger groups of elite teams.

9 Likes
8

Personally, I’d say don’t worry too much. This year’s competition seemed pretty well targeted at being a ‘softball’ to help teams ramp back into doing FIRST after a few years off. I strongly suspect the ceiling will be raised next year.

+1.

There’s a balance to be had for sure: Too much luck and you loose the ability to motivate students to do a good job. Too little luck and everyone but the top team knows they have no chance of winning, similarly demoralizing people from doing a good job.

IMO I want more games like this. A huge part of what drives our team’s pedagaogy forward is simply knowing, at the start of every year… there’s a chance we could win it all. I know from where my team is at, and trying to grow to, the luck/skill balance in 2022 was much better suited to us than 2020. But that’s gonna be different for different teams and their goals.

What I would like to see more of opportunities for wild and crazy gamebreaking strategies. Wallwalkers and jumpbots and whatnot.

9 Likes
9

yes, agree with you … as mentioned in the OP - there are many good things in adding “random noise” to the equation. and it is all about the right balance - which - I also agree with you that the “right” balance is team dependent

1 Like
10

My impression is that we really need more tiered styled RP’s. Like a not too hard one. A fairly difficult one. And a bloody hard one. I also think it should start that way. I don’t like changes in the middle of a season.

1 Like
11

Agreed on good scouting but I think the OP might be wishing the number 1 ranked team was the team with the highest OPR and then would have potentially picked OP’s team (number 2 in OPR).

2 Likes
12

I like this idea at first glance. An interesting experiment would be to see how a 5th RP for 6 cargo in auto would have affected things. Perhaps it would have made sense to do something like that as opposed to using auto cargo to reduce the cargo RP threshold?

13

or 5/6 cargo in auto unlocks the ability to earn RP#5 which is in the 30-40 teleop ball range. I.e. you must rock both Auto and Tele to earn RP 5.

Edited for clarity.

3 Likes
14

Since it’s not made clear in the post… where does the OPR number come from? Is that the team’s pre-champs OPR? Is it a full season (including champs) OPR? Is it the OPR just from the champs qualifying matches?

Because that really matters. It ultimately doesn’t matter how high a team’s OPR is before champs. Things change. Maybe their driver is home sick so someone new is driving the robot. Maybe something broke on the robot in their first match and the fix isn’t working as well as they want it to. On the other side, maybe a team with a lower OPR made a positive change and is doing better at champs. Whatever the reason, no matter what the schedule is, a teams performance at champs may not be the same as their performance earlier in the season, and that could throw off numbers like these - especially when you’re talking about 450 teams!

I’d love to see a graph that shows a team’s champs qualifying round OPR against their qualifying round rank. Such a graph could really help show the overall trend for that specific event.

4 Likes
15

This would be interesting. Instead of raising the cargo RP. Instead add a 3rd bonus RP.

It would be interesting to see how adding one for 40 cargo might impact the game. That is reaching 40 cargo would get 2 RPs.

2 Likes
16

yes - CMP quals OPR - in order to check the performance in the CMP itself and not historic OPRs

1 Like
17

There is also the fact that OPR is by no means a precise science that you can use to make claims about which robot was best.

7 Likes
18

I felt this game did a FAIRLY good job at ranking but luck in schedule definitely played a huge role. I found that this year that is was very difficult for a single robot to CARRY an alliance by themselves. Two pretty good robots would often beat one very good robot. Also if you happen to partner with two robots that both couldn’t climb OR score it made both of the bonus ranking points very difficult to impossible.

19

True, but neither is championship seeding. I believe that the OP is positing that there is a stronger correlation between OPR and robot quality than there is between Championship seeding and robot quality.

11 Likes
20

For what it’s worth, our mean score data looked like this for Turing’s top 8… this also has some flaws, but I think it’s interesting:

image

4 Likes