[FRC Blog] Two Championship Survey Results and Path Forward

[Siri]

http://www.usfirst.org/roboticsprogr...d-path-forward

As many of you know, after we announced that FIRST would be moving to having two Championships starting in 2017, we released a survey to FIRST Robotics Competition teams on this change.

7,355 individuals responded. This represents about 10% or so of the total number of mentors and students we have in FRC. About 75% of the 7,355 provided a team number, and those respondents were from 1,501 teams, about 52% of the total number of FRC teams we had in 2015.

One question we asked was just a simple ‘How do you feel about having two FIRST Championships starting in 2017?’ with an answer of 1 representing ‘Strongly Oppose’, an answer of 5 identified as “Neither Oppose nor Favor” and an answer of 10 identified as “Strongly Favor”. The average answer to this questions among all respondents was 4.45, somewhat below the 5 "Neither Oppose nor Favor" rating. Here is a graph of the full results:

We conducted additional analysis to better understand the rating score. FRC Teams that had never attended Championship had an average rating on this question of 5.85, while teams that had attended Championship had an average rating of 4.05. Mentors and Students had ratings relatively close to each other, at 4.54 and 4.30, respectively. When we determined average ratings from individual teams, then averaged those ratings (essentially giving every team one ‘vote’, and assuming the average rating of all team respondents was the one ‘vote’ for the team), the average rating was 5.2.*

We also asked teams what elements of the ‘Championship Experience’ were most important to them. There were 22 elements to choose from. Teams identified these as their top 10, in order from most to least important:

• Seeing and competing with the teams with the best robots in FRC
• The experience of attending a major, multi-day event with my team.
• Participating in a competition that identifies the best teams playing the game
• Seeing teams you have built relationships/partnerships with over the years
• Keeping attendance costs reasonable
• Participating in a very large scale event with tens of thousands of others
• Seeing and meeting international teams
• Participating in a competition that identifies the teams most deserving of awards, such as the Chairman's award
• Seeing and meeting top teams, like prior Chairman's Award winners (Hall of Fame teams)
• Having your matches in an impressive, large scale space

Taken as a whole, I think there is nothing within these survey results that is surprising. They do reinforce the idea that some within our community are strongly opposed to the two Championships concept, and that we should be using the elements identified by teams as most important to them as a guide to refining the concept to help ensure the best experience for all teams as we work through this significant change.

Our intent is to form committees, including representatives from the community outside FIRST HQ, to make recommendations to FIRST leadership in addressing the two key challenges listed below.

• Identifying what geographic regions will be assigned to which FIRST Championship as their ‘home’ Championship, including the way in which teams outside the United States would be handled
• Identifying a potential way in which teams may volunteer and be selected to attend their non-home Championship

You will hear more about these committees over the next few months. As we noted in the Championship informational session, the facts that there will be two Championships starting in 2017, and that all FIRST programs will be represented at each Championship, will not be changing, and so won’t be part of the discussions undertaken by these groups.

Additionally, FIRST HQ will be exploring the possibility of some culminating event to take place after the two Championships, at which we would bring together the top teams from each Championship in some final competition of the season. This idea is still in the early exploratory phase, and we will share additional details, including potential areas for community input, as appropriate.

Frank

*The analysis of average ratings from individual teams was completed by our Director of Research and Evaluation. She was the only one to see individual average team ratings, and after completing the analysis, deleted the team numbers from the data set.

[Steven Donow]

That's actually a surprisingly large amount of people filling out the survey (I've always wondered how many people fill out the surveys; I generally fill out most of the offseason ones and occasionally two or three of the weekly ones).

But srsly FIRST has made it clear since the town hall meeting that they have absolutely zero interest in listening to the community and doing anything other than two geographically split championship events, as opposed to two championship events divided in some other way*


*as much as I love these proposals, I just have so much hesitance over them because of teams possibly having to 'shuffle' between championships (ie. finalist at a week 1 event, then winning a week 6)

[Madison]

What is the perceived value in the idea of one team, one vote?

[BrendanB]

Originally Posted by Madison

What is the perceived value in the idea of one team, one vote?

Because it makes their case look better.

[PayneTrain]

Originally Posted by Madison

What is the perceived value in the idea of one team, one vote?

The survey providers' need for validation.

[Taylor]

Engineers love data.

Originally Posted by Madison

What is the perceived value in the idea of one team, one vote?

Originally Posted by BrendanB

Because it makes their case look better.

Originally Posted by PayneTrain

The survey providers' need for validation.

Raar. Too much data. By being exploratory, they're manipulating it!

[/s]

[Gregor]

The survey results were very surprisingly close to neutral, with the average response just under 5.

I was expecting a stronger bias.

[KeeganP]

Originally Posted by Madison

What is the perceived value in the idea of one team, one vote?

Team A1 has 60 members, and all of them voted for option 1 (strongly opposed).
Teams A2-61 then each have 10 or so members, but only 1 from each team votes. All 60 of those votes would have to be for option 10 (strongly support) in order to balance the overall score.

My assumption is that they are comparing the overall responses to the 1 vote-per-team responses just like we (in the US) have the House of Representatives (reps proportional to the state's (team's) population) and the Senate (2 reps per state (team)). I see a potential value in the separating it out, but not much.

[Holtzman]

Originally Posted by Gregor

The survey results were very surprisingly close to neutral, with the average response just under 5.

I was expecting a stronger bias.

What is a little misleading about the results is the survey scale. On a scale from 1-10 with 5 being neutral, we have 4 options that are negative, one that's neutral, and 5 options that are positive. This skews the data on the positive side. The average answer of 4.45 is misleading since the left and right sides of the data set have different weights.

Another way of looking at these results is that 55% oppose two championships, 12% are neutral, and only 33% favor two championships. To me, that’s is a much more powerful statement about how the community really feels.

[marshall]

Originally Posted by Holtzman

What is a little misleading about the results is the survey scale. On a scale from 1-10 with 5 being neutral, we have 4 options that are negative, one that's neutral, and 5 options that are positive. This skews the data on the positive side. The average answer of 4.45 is misleading since the left and right sides of the data set have different weights.

Another way of looking at these results is that 55% oppose two championships, 12% are neutral, and only 33% favor two championships. To me, that’s is a much more powerful statement about how the community really feels.

I was literally just saying this. The scale is really not right.

[wilsonmw04]

Originally Posted by Gregor

The survey results were very surprisingly close to neutral, with the average response just under 5.

I was expecting a stronger bias.

So what this data tells me is that a small number of teams really hate this idea and are very vocal about it. This seems to jive with what typically happens here on CD.

[Madison]

Originally Posted by wilsonmw04

So what this data tells me is that a small number of teams really hate this idea and are very vocal about it. This seems to jive with what typically happens here on CD.

You cannot possibly come to this conclusion without knowing who the respondents were.

[jaustinpage]

It is interesting that a lot of the analysis is team analysis; which is based off of an optional (identifying) field, e.g. team number. People in extreme camps are less likely to provide identifying information. This is likely going to skew all analysis done on data that excludes responses without the identifying information towards the perceived less dissenting answer, whatever that might be.

[EricH]

Originally Posted by Madison

You cannot possibly come to this conclusion without knowing who the respondents were.

No, I think he can, if you look at that statement in context. Gregor commented that he was expecting more bias (towards negative, I'm assuming). Wilson then concluded that there was a vocal minority.

I think that's reasonable: The vocal minority outweighs the silent majority in a lot of matters. This time, though, it's not exactly a minority--but it's close. I'd be thinking really carefully about my PR strategy if I was HQ--a good PR strategy can take a moderate opposition and take it to moderate advocacy given time, but a bad PR strategy can go the other way in a big hurry.


I think my spin detector went off, too, at one point. What I take away from this is: 1, this is going forwards regardless of community feeling, and 2, the overall community isn't exactly happy, but isn't actively opposed.


Y'all saw those committees, right? Boy do I pity those groups--I've got a feeling that more than anybody else (sorry, Frank and HQ), they're going to be the determiner of whether or not that survey result changes more towards strong approval.

[jman4747]

"Keeping attendance costs reasonable" being only 4th (Edit: sorry 5th)...

I think a lot of really good teams are likely larger than average and if more of them do oppose than I can see more people from each bothering to respond to voice a negative opinion rather than somewhat positive or neutral.
People tend to talk about things when they are very good or bad rather than in the middle. I you notice most or at least a lot of counter arguments supporting the move are merely pointing out that the switch seems more neutral than good or bad. People usually don't spend much time on something that they don't think will matter much.

Also dislike of the need to switch vs dislike of the decision. That is dislike of getting surgery vs dislike of someone taking something from you will pull the results negative though everyone in negative isn't mad at HQ.