Chief Delphi

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)
-   General Forum (http://www.chiefdelphi.com/forums/forumdisplay.php?f=16)
-   -   [FRC Blog] Two Championship Survey Results and Path Forward (http://www.chiefdelphi.com/forums/showthread.php?t=137251)

JB987 16-05-2015 11:11

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Can we stop suggesting that IRI would become a substitute for a Championship? Many years there are Einstein teams that are unable to make it to Indiana (just look at this summer's team list to see current examples) and often the drive teams are not the same anyways...

brrian27 16-05-2015 16:14

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Something used in business is the Net Promoter Score (NPS) based on the simple question to customers of whether they would recommend a product to others on a scale of 1-10 (actually 0-10, but we'll set the bottom at 1 as FIRST did). For the NPS, responses of 1-6 are "detractors," 7-8 are "passives," and 9-10 are "promoters." The NPS is calculated by the percentage of promoters minus the percentage of detractors.

This model doesn't perfectly apply to this survey question, since it is not a purely recommendation question, but we can view it as basically asking if you would recommend the championship split to FIRST. Despite the imperfect application, this model does remind us that people who vote 6-8 aren't as satisfied as we think.

Anyway, for this question the championship split has an NPS of -55, which is not pretty. An average company gets an NPS between of between 5 and 10. Here is a benchmark for NPS.

Here's more info about NPS. it's not a perfect application, but it's an interesting perspective.

gblake 16-05-2015 19:08

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by marshall (Post 1482494)
Are you serious? With the level of commitment and funding that these companies provide, I can't believe they weren't consulted or at least it had to have been considered. The total cost for FIRST has to go up considerably for hosting two of these events. They have to have a plan for fundraising for them to cover the cost of hosting. At least, I hope they do.

I suspect that the companies donating noticeable $ to FIRST will continue to let FIRST inspire students, without being the least bit interested in micromanaging how FIRST does it.

My very limited experience in that realm gives some weight to my opinion on the subject, but I'm not remotely close to an expert, and I never was a spokesman for any company.

waialua359 16-05-2015 22:38

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by JB987 (Post 1482504)
Can we stop suggesting that IRI would become a substitute for a Championship? Many years there are Einstein teams that are unable to make it to Indiana (just look at this summer's team list to see current examples) and often the drive teams are not the same anyways...

Joe, you bring up a good point.
As it stands, there is no easy way to find a solution to how the Blog suggests about bringing together both the North/South Champs to compete at a later date.
The cost would be enormous for teams to play another weekend at a neutral site, and the students/season are already maxed out.
No matter how creative the solution, it will not address what you are pointing out.

Ultimately, the season would have to be extended. Where? Before the New Year? It certainly cant afterwards. Once you hit May, we are talking about AP Exams, graduations, and other Academic Events that students are obligated to attend. Its not everyone, but enough that many teams would either miss events or certain students would miss them.
The point that FIRST is growing and yet wants to give the same % of students the Championship experience, will not find a happy medium to address having one set of Champions and everyone playing under the same roof. Too many pros AND cons. Is it really too late or impossible to find a venue 2020 and beyond that can hold 800 teams?

I wanted to add that it still doesnt sit well with me that in 2014, we had to choose between VEX and FRC Worlds because we do both programs. Even the VEX GDC said we had a good robot! We still wonder the what ifs had we played at VEX Worlds in 2014 for all of our teams that qualified. Many of our students had wished they could do both.
Our underclassmen are already a little bummed that they wont get to see teams from Canada and Michigan, assuming we would be in Houston in a couple of years.

who716 16-05-2015 23:26

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
this make me so angry! why even post the results that show the public disagrees with there choices and then try to manipulate and extrapolate the numbers to try and get it to support there decision.
I would much rather them say :this is how it is going to be, deal with it" instead of beating around the bush.

John 16-05-2015 23:34

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by Infinity2718 (Post 1482468)
I find both of these interesting, so I would like to expand on them:

1 vs 10 = 26 to 12 = 2.16:1 ratio
1-2 vs 9-10 = 37 to 16 = 2.31 ratio
1-3 vs 8-10 = 48 to 23 = 2.09 ratio
1-4 vs 7-10 = 55 to 29 = 1.90 ratio
1-5 vs 6-10 = 67 to 33 = 2.03 ratio

No matter how you slice the deck, for every one person approving the championsplit, there are two people opposing it.

This method isn't exactly fair either. In your first comparison, you are comparing the ratio of 25% of negative responses to 20% of positive. In your second it is 50% to 40%, and so on. In your last comparison, you count 5 (neutral) as negative.

What if we rescaled to a scale from 0 to 10? We can map the negative responses linearly to get the new responses. We replace 1 with 0, 2 with 1.25, 3 with 2.5, and 4 with 3.75. After performing the average based on this data, we get:

.26*0+.11*1.25+.11*2.5+.07*3.75+.12*5+.04*6+.06*7+ .07*8+.04*9+.12*10 = 4.06

This weights everything symmetrically. It keeps positive values positive, and negative values negative. However, I think it is still likely to be flawed, as someone who is approximately neutral might be more likely to lean towards the favorable side than negative simply because the positive side is larger. Someone who votes roughly neutrally based on the "center" of the scale may be unfairly counted as voting positively.

A better method might be to map the entire scale from 1-10 to 0-10. We replace 1 with 0, 2 with 1.111, 3 with 2.222, 4 with 3.333, 5 with 4.444, 6 with 5.555, 7 with 6.666, 8 with 7.777, and 9 with 8.888 (10 remains 10).

We now get:

.26*0+.11*1.111+.11*2.222+.07*3.333+.12*4.444+.04* 5.555+.06*6.666+.07*7.777+.04*8.888+.12*10 = 3.86

This scale is likely to be slightly biased towards negative, because it treats "neutral" according to the instructions as very slightly negative.

I think the true average, if the scale had been 0-10 instead of 1-10, would lie somewhere between these two numbers. In any case, they are closer to each other than they are to FIRST's number for the average (4.47).

There also are quite possibly some psychological effects that I have not accounted for. Do the numbers on the scale themselves affect how we vote? If given a poll, 1-5, and the average is 4, does this imply that if the same poll was conducted on a scale from 1-9, the average would be 7? This would be expected if people simply scaled their votes linearly (or at least, linearly on average) but that may not be the case.

If we remap the entire scale

PAR_WIG1350 17-05-2015 01:56

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by John (Post 1482591)
There also are quite possibly some psychological effects that I have not accounted for. Do the numbers on the scale themselves affect how we vote? If given a poll, 1-5, and the average is 4, does this imply that if the same poll was conducted on a scale from 1-9, the average would be 7? This would be expected if people simply scaled their votes linearly (or at least, linearly on average) but that may not be the case.

If we remap the entire scale

I have never found a study that confirms this, but I have heard it suggested that the widespread practice of using 75 as the 'center' of a 100 point grading scale in US schools has predisposed the people who attended those schools to center their rating on 75%, rather than on 50%. I feel that the most significant thing FIRST did correctly for this survey question was specifying a center, which I imagine would at least slightly help to fix that bias.

David Lame 17-05-2015 12:28

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
There's an awful lot of analysis going on. That makes sense because so many of us are engineers. We love to crunch numbers. A lot of that effort is wasted, though. The survey methods weren't designed to give precise answers, for all of the reasons so many others have already stated. This is a non-scientific poll, which is only good for getting a quick read on the general feelings of a non-uniform sample.

Taking a step back, though, the message is pretty clear. The big bars are on the left. The little bars are on the right. Generally speaking, the people who responded to this survey were pretty negative about the split. You don't need much mathematics to reach that conclusion.

Which brings up a couple of very obvious questions.

Do the survey results reflect opinion in general?
Why is the leadership pretending that somehow the survey results are neutral or only slightly negative?


From the discussion and analysis, though, I see a couple of other things.

One is that I find it interesting that there was a significant split between those who had never attended and those who had attended. That, to me, is meaningful.

The other thing that leaps out to me, mostly from the discussions, is....districts. Everyone ought to be doing them. Everywhere. I'm new here, but I can't imagine why anyone wouldn't.

I have to do a bit of interpretation of the numbers in order to reach this conclusion, but I think that those people who like the split may very well like it simply because it gives them another accessible, and significant, competition. A district championship would serve that purpose, much like it does in Michigan.

Monochron 17-05-2015 12:42

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by Holtzman (Post 1482290)
What is a little misleading about the results is the survey scale. On a scale from 1-10 with 5 being neutral, we have 4 options that are negative, one that's neutral, and 5 options that are positive. This skews the data on the positive side. The average answer of 4.45 is misleading since the left and right sides of the data set have different weights.

Another way of looking at these results is that 55% oppose two championships, 12% are neutral, and only 33% favor two championships. To me, that’s is a much more powerful statement about how the community really feels.

I have read all the posts on this thread, but I'm hoping that this point got a lot more attention than it did on the first page. The way FIRST collected data and they way they have presented it in this blog post is legitimately very misleading. The number of negative options versus the number of positive options really rubs me the wrong way.

Steven Donow 17-05-2015 12:46

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
I wish they gave actual analysis based off team number. Not even to show that,'this elite team felt this way!' but just to see how votes were split across team age, events attended, teams that have been to champs vs teams that haven't, etc... Then we'd be able to fully understand what the demographic reach that responded to the survey was.

Siri 17-05-2015 13:21

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by David Lame (Post 1482636)
There's an awful lot of analysis going on. That makes sense because so many of us are engineers. We love to crunch numbers. A lot of that effort is wasted, though. ... Why is the leadership pretending that somehow the survey results are neutral or only slightly negative?

I think you've answered your own point here. A lot of the analysis that's happening now isn't because we think the data is so useful, it's to highlight the misleading nature of the results. This statement from the blog: "The average answer to this questions [sic] among all respondents was 4.45, somewhat below the 5 "Neither Oppose nor Favor" rating" is an insult to my intelligence as an engineer, and to your point seems to be the crux of that 'only sightly negative' spin that the leadership is putting on these results.

Quote:

Originally Posted by David Lame (Post 1482636)
The other thing that leaps out to me, mostly from the discussions, is....districts. Everyone ought to be doing them. Everywhere. I'm new here, but I can't imagine why anyone wouldn't.

I have to do a bit of interpretation of the numbers in order to reach this conclusion, but I think that those people who like the split may very well like it simply because it gives them another accessible, and significant, competition. A district championship would serve that purpose, much like it does in Michigan.

Absolutely. This is the District Events --> Super Regionals --> Worlds argument that's been made for years (plus or minus the debate over another level of competition). Championsplit is basically that as a top-down attempt, wherein it limits the number of teams (to only 800) that can get an Large-but-less-than-Worlds Tier experience and also the number of teams (to zero) that can experience a Worlds event. They haven't fixed the core progression or scalability problem. However, standing up a District is unfortunately no small task and takes a certain density of teams and grassroots initiative among other things. There are huge discussions on CD and elsewhere about how to make it work in places that haven't yet--the devil's in the details.

efoote868 17-05-2015 13:33

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by Steven Donow (Post 1482638)
I wish they gave actual analysis based off team number. Not even to show that,'this elite team felt this way!' but just to see how votes were split across team age, events attended, teams that have been to champs vs teams that haven't, etc... Then we'd be able to fully understand what the demographic reach that responded to the survey was.

What safeguards were in place to stop individuals from falsely identifying the wrong team? What about stop them from filling out multiple surveys? Prevent FIRST unaffiliated or unaffected from responding?

Why should we trust these results as anything more than a voluntary online survey?

DanielleSisk 17-05-2015 14:51

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by Siri (Post 1482315)
More than a quarter of the respondents are as against this proposal as it is possible to register on this scale. That's closed to 2,000 people in itself (numerically 1912.3 people). 55% of respondents are against the proposal; that's more than 4045 people. CD is vocal, but even assuming the relationship between CD and the survey sample (which is a weird assumption when n=7355), opposition by definition is not the minority opinion. And despite the scale shift, the "strongly opposed" outnumbers all those who voted 10, 9, and 8 combined. More people voted for 1 or 2 than voted for anything above 5.



Does anyone know if there's a standard method of "centering" a scale like this? (The true center is at 5.5, the average of 1 and 10). I don't have a statistical method of turning 4 buckets into 5, but I think the worst-case scenario would be that everyone who voted 1 would've voted 0, and everyone in 2 took 1 (no one votes 4). This creates a new weighted average of 3.92, which represents the low end of possibility: thus the average is somewhere between 3.92 and 4.47 when centered about 5. Did I handle that correctly?

This is simple to do by reducing to number of bins instead of increasing them. Put all responses below the Neutral choice, the 55%, in one bin and call it say bin 4 (to keep our median at 5), then put the 12% which are Neutral into bin 5, and those above neutral, the 33%, into bin 6. Finding the average this way, (55*4+12*5+33*6)/100 = 4.78. Interesting... this shows that giving more options for a positive response gave their average, 4.45 (which should have been 4.47) a skew in the negative response direction. Now the same calculation cannot done for a '1 vote per team' because that data is unavailable.

Drakxii 17-05-2015 14:56

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by efoote868 (Post 1482645)
What safeguards were in place to stop individuals from falsely identifying the wrong team? What about stop them from filling out multiple surveys? Prevent FIRST unaffiliated or unaffected from responding?

Why should we trust these results as anything more than a voluntary online survey?

Why do you suspect that these results do not match the views of FRC teams?

efoote868 17-05-2015 15:24

Re: [FRC Blog] Two Championship Survey Results and Path Forward
 
Quote:

Originally Posted by Drakxii (Post 1482655)
Why do you suspect that these results do not match the views of FRC teams?

Because individuals without a strong opinion do not invest their time in voluntary surveys.


All times are GMT -5. The time now is 22:47.

Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi