Standardized District Point Model

[Nathan Streeter]

So, based off a newsletter that went out to New England teams today, it's confirmed that FIRST is planning on standardizing the district system point model for 2014... so, what do you think?

First off, do you think it should be standardized? Should each region be able to choose how they qualify teams - in something akin to "States' Rights?" Or would it be silly for FIRST not to choose how awards and accomplishments are rewarded?

Second, what do you want to see in the point system? At least a dozen threads exist about exact point models... so while I'm not opposed to talking about details, I'd also like to talk about the "philosophy" and various approaches.

I'll start it off with some of my own thoughts...

While I do think regions certainly should be able to make some decisions about how they do things, I don't think FIRST can afford to allow every region to have it's own system. It could easily be a big source of contention for teams who just miss the cut in their own region (but would've made it under region __'s system)... and FIRST really does deserve to be able to decide how teams qualify for their events.

In terms of philosophy, I think the point system should do the best to advance the best teams. That means the best robots, the best teams, and those who bring a strong balance of each. I like that NE FIRST has tried to add emphasis to some of the other awards... however, I don't feel an award like Engineering Excellence should be equal to the Safety award or Team Spirit (both of which, while awarding good attributes, sometimes end up having to be awarded based on more superficial actions at events). That said, the Michigan/MAR system of assigning points based on alliance selection is critical to rewarding strong teams who have a tough qualifying schedule or elimination set.

I'm not a fan of the auto-bids. I'd like for everything to be assigned points... partly because of the scalability that Jim Zondag emphasized in his Q&A on the Michigan Districts... and partly because then it puts all the awards and accomplishments on the same level. FIRST can compare Chairmans' apples-to-apples to Regional Wins. Also, it gets rid of some of the little loopholes... With the proposed New England points system, winning Chairmans' at a district event autobids you to the DCMP... but it doesn't add to your point total to help you qualify for CMP.

While I'm not a huge fan of the auto-bids, I would like for teams' that win Chairmans', Engineering Inspiration, or Rookie All Star to be automatically eligible to send their judging team to the District Championships... that way the best teams for each award are being judged at DCMP.

Also, I think to help make the various judges' jobs easier, I'd like for teams to have to choose to be eligible for an award at an event. Judges for each award struggle to get much time with each team; however, if only the teams who are interested in winning a particular award apply for it, then the judges should be able to spend more time with each eligible team. It seems like judges have a nearly impossible job really determining the best team for an award with how little time they have... perhaps this could make it easier? Teams wouldn't need to be limited to a number of award applications at an event, or applications for a particular award at multiple events... but they could be if preferable. Personally, I'd rather teams not be limited to applying for an award at only one event... that way teams are less likely to "luck out" or get hung up by teams primarily applying for that award at one event or the other.

What do you think?

[Steven Donow]

In terms of moving towards a total remodel of the FRC Event/Championship/District/Regional/SuperRegional structure, this is definitely a step in the right direction towards unifying FIRST to a new era of event structure, and I definitely expected this to happen, especially with the addition of two district systems this year.

Originally Posted by Nathan Streeter

I'm not a fan of the auto-bids. I'd like for everything to be assigned points... partly because of the scalability that Jim Zondag emphasized in his Q&A on the Michigan Districts... and partly because then it puts all the awards and accomplishments on the same level. FIRST can compare Chairmans' apples-to-apples to Regional Wins. Also, it gets rid of some of the little loopholes... With the proposed New England points system, winning Chairmans' at a district event autobids you to the DCMP... but it doesn't add to your point total to help you qualify for CMP.

While I'm not a huge fan of the auto-bids, I would like for teams' that win Chairmans', Engineering Inspiration, or Rookie All Star to be automatically eligible to send their judging team to the District Championships... that way the best teams for each award are being judged at DCMP.

Did the NEFIRST email say that EI and RAS get autobids to DCMP? I'm not trying to sound condescending, but in MAR and (I assume) FiM, the RAS and EI teams both get to send judging teams to MSC/MAR Champs(in fact at MAR Champs this year, one of the EI winners, 321, was there judging-only, as was the RAS winner, 4637)

I'm gonna disagree with you on this though, and say that if FIRST wants to consider Chairman's the most important award, they should continue to grant teams that win it the same opportunities as teams with incredible robots. For some teams, if they have a strict administration, it might be hard to convince administrators to allow the trip to DCMP if there's no robot going-unfortunately, I'm sure this is true.

However, in two years of MAR, all teams that have won Chairman's have otherwise been in a pointswise position to make it to MAR Champs. I believe the same is generally true for FiM(one of them can confirm or deny; my memory of viewing spreadsheets tells me there have been one or two), but remember: generally, teams that win Chairman's usually have pretty good robots.

Now, I had started researching an analysis to see how the teams that won Chairman's in NE regionals this year and last year would have qualified based off their robot performance using these rankings, but then I realized that it contains the original NE point proposal. Has anyone applied the FiM/MAR ranking style to NE for last year?

[Mr V]

I believe it is a good thing that FIRST is implementing a standardized points system.

Frank said it pretty well in his blog post from the June 7th.

Our intention is to evolve the existing District system to allow for cross-District competition. We have begun work on a standardized points ranking system, and a standardized way in which teams will advance from District Championships to the FIRST Championship. As we grow, it’s important for us to maintain consistency between Districts. In addition to helping us maintain the quality of the FRC experience, this will give us the best chance of evolving the system to the point of allowing cross-District competition.

In addition to the reasons Frank mentioned this will make it easier for the move to districts across the US. It removes one more item for district organizers to plan for and worry about.

[Kims Robot]

Originally Posted by DevenStonow

Now, I had started researching an analysis to see how the teams that won Chairman's in NE regionals this year and last year would have qualified based off their robot performance using these rankings, but then I realized that it contains the original NE point proposal. Has anyone applied the FiM/MAR ranking style to NE for last year?

Lot of discussion in a bunch of different threads... the short answer is yes, I & others have looked at applying the FiM model to NE teams... (one of the models I looked at included applying the FiM points)

Originally Posted by Kims Robot

This depends on the model that is used. As it is possible/likely that FIRST will be standardizing the system across the districts, and its hard to tell what the final result will be. Using several points model simulators (ie the ones in this thread), of the 5 NE winners, only one was even close to the cutoff of the 53/60 line in one model I've seen (and easily made it in several others). Yes if you add 4 more, there is a slight possibility that the robots will not be as high in points, but in general, I think the District points wont produce as wide a variation as the CA performance at Worlds correlation.

Back to the general topic, I am all for the standardization... it's one less thing for the District Organizers to worry about, and it will get us much more quickly to a system that will allow for cross district play.

My only hope is that FIRST does (or involves the people that have done) countless hours of simulation on whatever model they come up with to help prove that compared to past years this new model will send the "right" teams to Championship based on whatever philosophy the group deams appropriate. (ie I want to know that someone "did the math")

And I agree with some of the previous suggestions that Im not a huge fan of the Auto-bids, but I do think CA & at least winning captain, possibly EI & RAS too should end up with high enough points that its almost no question they get into DCMP. Now the award winners shouldn't be slackers on the robot side. I wouldnt expect to see a CA robot that was 39th/40 and didnt make elims at both their district events have a robot at DCMP or even WCMP, but looking at the past data, CA teams seem to have relatively strong robots. I'm ok with EI or RAS teams only getting to "present" at DCMP... but if they then get to bring their robots to WCMP... they end up with a disadvantage at WCMP as they have had fewer plays than everyone else. So I think whatever model is adopted for Districts would likely have to be applied to WCMP as well... or maybe have a 5th Division that allows all these teams to play at WCMP, but doesn't qualify them for Einstein?? I dunno... plenty of ideas/discussion to kick around there.

At any rate, I'm excited to see what FIRST comes up with (and secretly happy to not have to deal with it at the NE level! ).

[Brandon Holley]

I definitely welcome a standardized point model- even though it may take away some of the flexibility and customization some people were looking forward to with a district model.

I believe it is important to maintain qualification and performance based on the same metrics wherever we can. As someone stated above- I could see a huge point of contention being slight differences district to district that cause teams to qualify/not qualify for whatever reason.

The task force (I assume there is one) that is currently trying to bake up a standardized points system does not have an easy job, and I do not envy them. I do feel however, this work will go a long way to 'reconnecting' the districts into the mainstream of FRC by tying them (loosely for now) to each other.


-Brando

[Rosiebotboss]

As a member of the NE FIRST point model team and without naming names (because it would be a diservice to them to have them personally under the scrutiny of this community and added pressure and possible inundation of PMs) I have seen the list and I am completely confident in the abilities of each and every committee member. They represent all 4 current districts, 2 districts "to be", WFAs, CCA reps as well as FIRST staff.

Just for a point of reference, we in NE, during our discussions, ran several iterations of our point model (based on 2010 Karthik scouting data) and the differences in outcomes were minimal. (The columns in the spreadsheet were labled FiM, MAR, NE 1, NE 2, NE 3 so we could see changes real time) In other words, the top 20 teams, regardless of algorithm, stayed top 20, the middle 20 stayed middle 20, the bottom 20 were the only ones effected. And that effect was 7 or 8 teams dropped out and 7 or 8 teams got in. There was movement within each sector, but the general rankings stayed pretty consistent.

Hang in there, the road may be rocky, but FIRST will be better coming out the other end.

[IKE]

Originally Posted by Rosiebotboss

As a member of the NE FIRST point model team and without naming names (because it would be a diservice to them to have them personally under the scrutiny of this community and added pressure and possible inundation of PMs) I have seen the list and I am completely confident in the abilities of each and every committee member. They represent all 4 current districts, 2 districts "to be", WFAs, CCA reps as well as FIRST staff.

Just for a point of reference, we in NE, during our discussions, ran several iterations of our point model (based on 2010 Karthik scouting data) and the differences in outcomes were minimal. (The columns in the spreadsheet were labled FiM, MAR, NE 1, NE 2, NE 3 so we could see changes real time) In other words, the top 20 teams, regardless of algorithm, stayed top 20, the middle 20 stayed middle 20, the bottom 20 were the only ones effected. And that effect was 7 or 8 teams dropped out and 7 or 8 teams got in. There was movement within each sector, but the general rankings stayed pretty consistent.

Hang in there, the road may be rocky, but FIRST will be better coming out the other end.

Underline emphasis is mine...

Having good values and measurable deltas at the cut points are crucial. Pay close attention to those areas. While who is #1 is important from an inspirational point, who is one above and one below your cut point is the biggest deal.

[Tristan Lall]

Originally Posted by Nathan Streeter

So, based off a newsletter that went out to New England teams today, it's confirmed that FIRST is planning on standardizing the district system point model for 2014... so, what do you think?

First off, do you think it should be standardized? Should each region be able to choose how they qualify teams - in something akin to "States' Rights?" Or would it be silly for FIRST not to choose how awards and accomplishments are rewarded?

One of the biggest problems with a real "states' rights" regime is that it gives states licence to do things out of self-interest, without regard for everyone else. It's not as big a deal when the scope of the state's power prevents them from significantly affecting anyone else—but realistically, that's a rare scenario in the real world.

On the other hand, one of the most convenient aspects of decentralized federalism is that each state can function as a (poorly-constrained) experimental group, to test the propositions embodied in their various laws. It's not perfect, but if there's genuine controversy over an issue, one way (but certainly not the only way) to solve it is by letting the chips fall where they may.

In our case, I'm not strongly opposed to regions having their own models, as long as there is an opportunity to discuss whether there will be far-reaching effects, and then an obligation to mitigate those effects unless consensus is reached with the other regions and FIRST.

Overall, I think I'd still prefer it if FIRST built the system at a national level—but I am concerned about their capacity to handle the implementation. (Though FIRST's current directorship does increase my confidence that they'll handle it well.)

Originally Posted by Nathan Streeter

Second, what do you want to see in the point system? At least a dozen threads exist about exact point models... so while I'm not opposed to talking about details, I'd also like to talk about the "philosophy" and various approaches.

It's worth clearly defining what each Championship (world, regional, etc.) is meant to be: Are they strictly tests of the best on-field performers? Or should they include the outstanding award-winners? Or the best overall teams by some combination of criteria? Some specific guidance from the top would be a nice addition to the conversation—or, if the FIRST leadership isn't certain, or if the community does not accept their reasoning, then maybe we need to settle that question before getting too invested in any particular point systems.