How biased are CD polls

[Caleb Sykes]

Partially inspired by this (http://www.chiefdelphi.com/forums/showpost.php?p=1538575&postcount=30) post, I want to know how reflective CD polls are of the FRC community. I will use competitive advancement in the 2015 season as my metric because these results can easily compared against actual team advancement.

If you normally participate in polls, please participate in this one. If you do not normally participate in polls, please do not participate in this one.

[Caleb Sykes]

Using the 2834 scouting database, here is what I found for teams' first event in 2015:

N teams = 2872
N not in eliminations = 1564
N quarterfinalists = 687
N semifinalists = 313
N finalists = 155
N winners = 153

P not in eliminations = 54.5%
P quarterfinalists = 23.9%
P semifinalists = 10.9%
P finalists = 5.4%
P winners = 5.3%

Given, some of those teams do not exist from last year, and this doesn't account for rookie teams or non-FRC members of CD.

[maxnz]

I can say that my team, which has been around since 2009, joined chief Delphi this year. There's just me and a general team account.

(We have used it as a resource but have never posted anything.)

[MechEng83]

I answered "other" as my team was part of the week 1 Phyxed-Red-Cards alliance. We were technically eliminated in the semifinals, but actually shouldn't have been due to an errant red card...

[Dinokaiz2]

Wow, interesting. The results so far reflect what I've thought for a while: the average Chief Delphi poll-taker is from an above-average team.

This might put the low bar polls into perspective. Most of the teams voting are above-average. That, to some extent, helps justify the extra design challenge of the low bar.

[SenorZ]

We have a few students with CD accounts who check posts and "lurk". The only people to actually post are myself and our engineering mentor. Generally we see CD as a place where the hyper-involved share most of the material and the the soon-to-be-hyper-involved cut their teeth.

[Jon Stratis]

This is an interesting poll, and will probably show that a disproportionate number of teams here play in elims. But it's also just a single data point, looking at a 1-year view. If you take my team, for example... We did horrible last year. But prior to that, we pretty much always made it into elims. You'll have outliers like that - teams that normally do well but had an off year last year, or teams that made it to elims for the first time in team history last year. Something to keep in mind when looking at the results.

[Ginger Power]

This is an interesting poll, and will probably show that a disproportionate number of teams here play in elims. But it's also just a single data point, looking at a 1-year view. If you take my team, for example... We did horrible last year. But prior to that, we pretty much always made it into elims. You'll have outliers like that - teams that normally do well but had an off year last year, or teams that made it to elims for the first time in team history last year. Something to keep in mind when looking at the results.

+1

4607 fits this description to a T. Recycle Rush wasn't kind to us.

[OccamzRazor]

This is an interesting poll, and will probably show that a disproportionate number of teams here play in elims. But it's also just a single data point, looking at a 1-year view. If you take my team, for example... We did horrible last year. But prior to that, we pretty much always made it into elims. You'll have outliers like that - teams that normally do well but had an off year last year, or teams that made it to elims for the first time in team history last year. Something to keep in mind when looking at the results.

I would say the data here may not apply to all teams everywhere but when you look at just Championship only over the course of several years then the data stacks up better to what I would expect from CD polls. For instance, North Carolina is now in a District event model instead of a Regional event. This throws the metrics way off for areas like this because we have multiple district events and a state championship before the big show and therefore more winners.

[Keefe2471]

Thanks for creating this! The scouting data and combined with these poll results should be really interesting. So far the data here shows that I was wrong. It will be interesting to see how this poll develops as more people get home from school or work and contribute.

Assuming that the percentages stay the same, roughly 80% of Chief Delphi making eliminations (35% the finals?!) versus 50% of the population of FIRST in general making eliminations is pretty impressive.

[RoboChair]

If we assume that a fixed percentage of all FRC students and mentors are active on CD, having a larger team will disproportionately give them more votes.

On top of that, successful teams often have a greater percentage of CD members that are active then teams that are more the norm.

[Monochron]

If we assume that a fixed percentage of all FRC students and mentors are active on CD, having a larger team will disproportionately give them more votes.

I think that's justified. This poll is measuring how well all of the "voices" on CD do. So if there is a thread about the low bar with 30 people participating, but they are only members of 18 different teams, each of the participants get representation in polls such as this one.

[Bryce2471]

A couple more things that this poll does not account for:

Because this is not a public poll, the honesty of the individual voters cannot be verified.

The psychology of the Chief Delphi uses likely makes people who did poorly in their first event last year less likely to vote on this pole, and those who plan on doing a lot of things more likely to vote on the other polls about robot capabilities.

[alopex_rex]

I've always taken it for granted that Chief Delphi posters are from above-average teams. That's a good thing when you need technical help, but it also means you have to be careful when you see people talking about strategy. Teams whose goal is to play on Einstein think about strategy differently from teams whose goal is to be selected for an alliance at a district event.

[pntbll1313]

If we assume that a fixed percentage of all FRC students and mentors are active on CD, having a larger team will disproportionately give them more votes.

On top of that, successful teams often have a greater percentage of CD members that are active then teams that are more the norm.

That's the entire point of this poll. To show that any poll on CD is inherently biased because the best and large teams are disproportionately represented.

Just about every poll has at least 3 votes from 254. We'd need 9,000 responses to get that to even out the voices of all FRC teams. Based on Caleb's original post we should expect to see 54.5% of teams not making elims last year. As I'm writing this post there are over 200 responses and only 20.0% did not make playoffs. It's not a surprise that CD is biased, but it's cool to have this poll to refer back to years down the road when everyone get's bent out of shape from the responses of a different poll.

[evanperryg]

A couple more things that this poll does not account for:

*snip*

Also, keep in mind that this is only recording a team's first event. We got knocked out in quarters at our first event because of some really bad luck, then made it to finals at our second event, won chairman's and went to Einstein. The first event alone is hardly a metric for a team's overall performance in a given season.

[MikLast]

Would you count a backup team going to a final (without being in any other match in that events playoffs) count as being eliminated in the final? If not then it would be just quarterfinals for us.

[Koko Ed]

The CD community is just a small very vocal sampling of the overall FIRST community. Many are either too intimidated or think CD is just a colossal waste of time to participate. CD should not be a representative of the pulse of the FIRST community in any way, shape or form.

[Taylor]

I believe Chief Delphi-posting teams are

1. Better than the average team
2. Not as good as they think they are*

*obvious exemptions apply

[Caleb Sykes]

Would you count a backup team going to a final (without being in any other match in that events playoffs) count as being eliminated in the final? If not then it would be just quarterfinals for us.

I personally would count this. If you look at my numbers that I ran in my second post, you will see that there were two more finalists last year than winners. Your team is probably one of those, so I think you have every right to say you were a finalist.

[mjc49]

Also, keep in mind that this is only recording a team's first event. We got knocked out in quarters at our first event because of some really bad luck, then made it to finals at our second event, won chairman's and went to Einstein. The first event alone is hardly a metric for a team's overall performance in a given season.

I believe it is fair to ask for results in the first tournament since there are teams that only participate in one event a year. As others have mentioned, CD is just a subset of the overall FIRST community and the group tends to come from teams that do better than average. Probably better than average mentor and sponsor support as well.

[Caleb Sykes]

Does anyone have a good estimate for the number of 2016 rookies?

[Conor Ryan]

After 318 votes, I think the results are pretty accurate for what the CD user base is.

Average CD user probably is in the range of the 40th-80th percentile of performance.

Lots of hive mind here, and frequent repeated mistakes in design and strategy, as evidenced by the Karthik scared discussion. The highest performers buck the trend continuously.

[Citrus Dad]

If we assume that a fixed percentage of all FRC students and mentors are active on CD, having a larger team will disproportionately give them more votes.

On top of that, successful teams often have a greater percentage of CD members that are active then teams that are more the norm.

Devin, ditto! I was just going to say the same thing. We have at least a half dozen from our team posting on CD...

[pntbll1313]

Devin, ditto! I was just going to say the same thing. We have at least a half dozen from our team posting on CD...

That's exactly what we're trying to show with this poll. That poll's aren't a good representation of average teams because the biggest and elite teams have many members filling in the same poll skewing the results.

[alopex_rex]

When has any online poll been a good representation of anything?