[FRC Blog] Two Championship Survey Results and Path Forward

[efoote868]

Originally Posted by Steven Donow

I wish they gave actual analysis based off team number. Not even to show that,'this elite team felt this way!' but just to see how votes were split across team age, events attended, teams that have been to champs vs teams that haven't, etc... Then we'd be able to fully understand what the demographic reach that responded to the survey was.

What safeguards were in place to stop individuals from falsely identifying the wrong team? What about stop them from filling out multiple surveys? Prevent FIRST unaffiliated or unaffected from responding?

Why should we trust these results as anything more than a voluntary online survey?

[Drakxii]

Originally Posted by efoote868

What safeguards were in place to stop individuals from falsely identifying the wrong team? What about stop them from filling out multiple surveys? Prevent FIRST unaffiliated or unaffected from responding?

Why should we trust these results as anything more than a voluntary online survey?

Why do you suspect that these results do not match the views of FRC teams?

[efoote868]

Originally Posted by Drakxii

Why do you suspect that these results do not match the views of FRC teams?

Because individuals without a strong opinion do not invest their time in voluntary surveys.

[wilsonmw04]

Originally Posted by Gregor

The survey results were very surprisingly close to neutral, with the average response just under 5.

I was expecting a stronger bias.

So what this data tells me is that a small number of teams really hate this idea and are very vocal about it. This seems to jive with what typically happens here on CD.

[Madison]

Originally Posted by wilsonmw04

So what this data tells me is that a small number of teams really hate this idea and are very vocal about it. This seems to jive with what typically happens here on CD.

You cannot possibly come to this conclusion without knowing who the respondents were.

[EricH]

Originally Posted by Madison

You cannot possibly come to this conclusion without knowing who the respondents were.

No, I think he can, if you look at that statement in context. Gregor commented that he was expecting more bias (towards negative, I'm assuming). Wilson then concluded that there was a vocal minority.

I think that's reasonable: The vocal minority outweighs the silent majority in a lot of matters. This time, though, it's not exactly a minority--but it's close. I'd be thinking really carefully about my PR strategy if I was HQ--a good PR strategy can take a moderate opposition and take it to moderate advocacy given time, but a bad PR strategy can go the other way in a big hurry.


I think my spin detector went off, too, at one point. What I take away from this is: 1, this is going forwards regardless of community feeling, and 2, the overall community isn't exactly happy, but isn't actively opposed.


Y'all saw those committees, right? Boy do I pity those groups--I've got a feeling that more than anybody else (sorry, Frank and HQ), they're going to be the determiner of whether or not that survey result changes more towards strong approval.

[Green Potato]

From a statistics point of view, there is no way to control for the 2 big reasons why the results are NOT statistically on-par.

1. It's a voluntary internet poll. The results will always be skewed toward the extreme.
2. There was survey bias in the number of possible responses.

All in all, I feel that because of this, the results may be flawed, but there's one thing I can be certain of: the community responded negaitvely as a whole.

[Andrew Schreiber]

So, the big takeaway is that 50% of teams don't really care.

Well, at least that's not a surprise.

[jman4747]

"Keeping attendance costs reasonable" being only 4th (Edit: sorry 5th)...

I think a lot of really good teams are likely larger than average and if more of them do oppose than I can see more people from each bothering to respond to voice a negative opinion rather than somewhat positive or neutral.
People tend to talk about things when they are very good or bad rather than in the middle. I you notice most or at least a lot of counter arguments supporting the move are merely pointing out that the switch seems more neutral than good or bad. People usually don't spend much time on something that they don't think will matter much.

Also dislike of the need to switch vs dislike of the decision. That is dislike of getting surgery vs dislike of someone taking something from you will pull the results negative though everyone in negative isn't mad at HQ.

[evanperryg]

Originally Posted by BrendanB

Because it makes their case look better.

Originally Posted by PayneTrain

The survey providers' need for validation.

Originally Posted by wilsonmw04

So what this data tells me is that a small number of teams really hate this idea and are very vocal about it. This seems to jive with what typically happens here on CD.

Case in point. Angry people tend to be louder than happy people. However, Holtzman makes a very good point that we, and FIRST, should consider. The numbers show a larger negative trend than it might seem.
My problem isn't completely with the 2 championship idea. I think with the right refinements, it will work just fine. I have a problem with how FIRST is dividing the event, because, at least in my not-so-humble opinion, they are moving the wrong way. Here's a quote from Don Bossi, copied from the transcript of the 2 Champs informational session:
"At this event last year we announced that we found a way here within St. Louis
to increase that and try to get that back into the 20 percent [of all FRC teams qualifying for champs] range by going to 600 teams...
The story for FIRST® LEGO® League, FIRST® Tech Challenge, Junior FIRST® LEGO® League is
much worse. FIRST Tech Challenge has the capacity for about 3 percent of their teams at
Championship. FIRST LEGO League, it kills me when I talk to a FIRST LEGO League partner
for a country and I say, oh we can’t even send a team this year, we don’t have a slot this year."
He then discusses some thrown-out options for changing FRC. The fact that he doesn't go into detail about how changing the timing of FRC would improve qualification rates leads me to believe that the ideas he discusses wouldn't have fixed anything. As far as I'm concerned, his presentation of these ideas is little more than a way to strengthen his point through shocking the audience.
Rhetoric aside, the most viable solution to the problems experienced by FLL and FTC are most easily resolved by the one solution that was discussed for the least amount of time. Why not put FTC at one event, and FRC at another? I recognize that FIRST wants to keep their programs intertwined, going along with the "progression of programs," but by giving FTC their own world championship venue, the size and scale of the FRC venue, they will be able to boost qualification rates astronomically. An FTC/FLL championship would also permit FTC to gain its own public identity. When asked about student robotics competitions, I have never come across a non-FIRSTer who knew anything about FTC. Most of them will answer a question about student robotics with something related to "I saw one that plays basketball!" or "oh, the little lego robots, right?" Perhaps it's time FIRST allowed FTC to gain their own identity, and make their championship event into their championship event. Perhaps it would be more logical to expand FLL into a double championship format, as head-to-head competition is a small, even nonexistent part of their program. By splitting FLL, you avoid the problems with not deciding a single winner of a highly competitive program, and you offer more space for more FLL teams to qualify. By putting FRC at one event and FTC at another, you keep the 2 most competitive events together, while significantly increasing the qualification rates for FTC.

[Citrus Dad]

Originally Posted by evanperryg

Case in point. Angry people tend to be louder than happy people. However, Holtzman makes a very good point that we, and FIRST, should consider. The numbers show a larger negative trend than it might seem.
My problem isn't completely with the 2 championship idea. I think with the right refinements, it will work just fine. I have a problem with how FIRST is dividing the event, because, at least in my not-so-humble opinion, they are moving the wrong way. Here's a quote from Don Bossi, copied from the transcript of the 2 Champs informational session:
"At this event last year we announced that we found a way here within St. Louis
to increase that and try to get that back into the 20 percent [of all FRC teams qualifying for champs] range by going to 600 teams...
The story for FIRST® LEGO® League, FIRST® Tech Challenge, Junior FIRST® LEGO® League is
much worse. FIRST Tech Challenge has the capacity for about 3 percent of their teams at
Championship. FIRST LEGO League, it kills me when I talk to a FIRST LEGO League partner
for a country and I say, oh we can’t even send a team this year, we don’t have a slot this year."
He then discusses some thrown-out options for changing FRC. The fact that he doesn't go into detail about how changing the timing of FRC would improve qualification rates leads me to believe that the ideas he discusses wouldn't have fixed anything. As far as I'm concerned, his presentation of these ideas is little more than a way to strengthen his point through shocking the audience.
Rhetoric aside, the most viable solution to the problems experienced by FLL and FTC are most easily resolved by the one solution that was discussed for the least amount of time. Why not put FTC at one event, and FRC at another? I recognize that FIRST wants to keep their programs intertwined, going along with the "progression of programs," but by giving FTC their own world championship venue, the size and scale of the FRC venue, they will be able to boost qualification rates astronomically. An FTC/FLL championship would also permit FTC to gain its own public identity. When asked about student robotics competitions, I have never come across a non-FIRSTer who knew anything about FTC. Most of them will answer a question about student robotics with something related to "I saw one that plays basketball!" or "oh, the little lego robots, right?" Perhaps it's time FIRST allowed FTC to gain their own identity, and make their championship event into their championship event. Perhaps it would be more logical to expand FLL into a double championship format, as head-to-head competition is a small, even nonexistent part of their program. By splitting FLL, you avoid the problems with not deciding a single winner of a highly competitive program, and you offer more space for more FLL teams to qualify. By putting FRC at one event and FTC at another, you keep the 2 most competitive events together, while significantly increasing the qualification rates for FTC.

This is an interesting passage because it reveals an important motivation for FIRST HQ. I think they see FLL, not FRC, as their future. I suspect that LEGO has a huge voice behind FIRST--FLL must be an important market of LEGO. So raising the visibility of FLL becomes most important.

Unfortunately, this is like trying to use AYSO to promote interest in soccer. While AYSO participation has exploded, it has had little impact on the interest in soccer in the U.S. In fact such interest has only increased as the women's team became dominant at the World Cup and the men's team really became competitive in 1994.

[David Lame]

There's an awful lot of analysis going on. That makes sense because so many of us are engineers. We love to crunch numbers. A lot of that effort is wasted, though. The survey methods weren't designed to give precise answers, for all of the reasons so many others have already stated. This is a non-scientific poll, which is only good for getting a quick read on the general feelings of a non-uniform sample.

Taking a step back, though, the message is pretty clear. The big bars are on the left. The little bars are on the right. Generally speaking, the people who responded to this survey were pretty negative about the split. You don't need much mathematics to reach that conclusion.

Which brings up a couple of very obvious questions.

Do the survey results reflect opinion in general?
Why is the leadership pretending that somehow the survey results are neutral or only slightly negative?


From the discussion and analysis, though, I see a couple of other things.

One is that I find it interesting that there was a significant split between those who had never attended and those who had attended. That, to me, is meaningful.

The other thing that leaps out to me, mostly from the discussions, is....districts. Everyone ought to be doing them. Everywhere. I'm new here, but I can't imagine why anyone wouldn't.

I have to do a bit of interpretation of the numbers in order to reach this conclusion, but I think that those people who like the split may very well like it simply because it gives them another accessible, and significant, competition. A district championship would serve that purpose, much like it does in Michigan.

[Siri]

Originally Posted by David Lame

There's an awful lot of analysis going on. That makes sense because so many of us are engineers. We love to crunch numbers. A lot of that effort is wasted, though. ... Why is the leadership pretending that somehow the survey results are neutral or only slightly negative?

I think you've answered your own point here. A lot of the analysis that's happening now isn't because we think the data is so useful, it's to highlight the misleading nature of the results. This statement from the blog: "The average answer to this questions [sic] among all respondents was 4.45, somewhat below the 5 "Neither Oppose nor Favor" rating" is an insult to my intelligence as an engineer, and to your point seems to be the crux of that 'only sightly negative' spin that the leadership is putting on these results.

Originally Posted by David Lame

The other thing that leaps out to me, mostly from the discussions, is....districts. Everyone ought to be doing them. Everywhere. I'm new here, but I can't imagine why anyone wouldn't.

I have to do a bit of interpretation of the numbers in order to reach this conclusion, but I think that those people who like the split may very well like it simply because it gives them another accessible, and significant, competition. A district championship would serve that purpose, much like it does in Michigan.

Absolutely. This is the District Events --> Super Regionals --> Worlds argument that's been made for years (plus or minus the debate over another level of competition). Championsplit is basically that as a top-down attempt, wherein it limits the number of teams (to only 800) that can get an Large-but-less-than-Worlds Tier experience and also the number of teams (to zero) that can experience a Worlds event. They haven't fixed the core progression or scalability problem. However, standing up a District is unfortunately no small task and takes a certain density of teams and grassroots initiative among other things. There are huge discussions on CD and elsewhere about how to make it work in places that haven't yet--the devil's in the details.

[billylo]

My feedback for Frank and FIRST:

1. Students are the main stakeholders. If trade-offs are needed, articulate the pros and cons factually and consult them. (just like the patient/doctor relationship.)

2. Surveys are imperfect. But I can generally count on those who care to cast their ballots. There is nothing wrong with building a strategy based on the opinions of those who care.

[R2D2DOC]

Transitions are always difficult. The USA never quite converted to the metric system. . . .

Ultimately, if we saturate the season with districts (even that is contentious), then district champs would be the primary players at world champs. The remaining participants would be up for discussion: HOF, Rookie, Chairmans, etc. . . .

Would we than go back to a smaller single championship event?