Hi all. I’m just getting around to sharing it, but last year I finished off my AP Stats class with a project examining the correlation between the average income of an area and the success of their robotics team. I know that the project is far from perfect, but at the very least, I think it has some pretty interesting data and graphs. Let me know if you have questions and feel free to correct any mistakes I may have made.
I think that the average income of families with students on a robotics team in a county is not equivalent to the average income of families in that county. While our team strives to support underserved communities, there are a lot of barriers that prevent students from lower income households from participating in our robotics program.
I know you said you used county income because it was most readily available. However, as Ambrose said, this is too broad a population. My county has communities with some of the highest incomes in the state, and some communities among the lowest in the state. If your random team selector picked a team in my county, it might have picked a team from an area far from the county median income, by a factor of 3 or more.
There are some interdependencies and bounds that are kinda difficult here. Out of the gate I am not sure how you would get around a few of these problems, but considering you have obviously looked at your assumptions a fair bit I ask you the following.
Sample size. 10 teams per week is far from robust, but if you had to limit sample size for practical reasons we will move on.
Median score of qualifications, this does not have an underlying linear relationship (i.e. diminishing returns) and some years it is really out of wack (2017 bracketed gear scoring, 2018 time based scores)
Qualification score relies on other teams, yes those teams are more likely to be from a similar area but not everyone plays in their own county. Is there a better metric you may know of that has attempted to decouple the influence of other teams?
By randomly selecting teams you are less likely to run into the tails of your income distribution, which in turn limits the amount of power the data can provide. Is there a sampling method you know of that partially addressed this?
Is county median income (population in the thousands)representative of a team that has a population of 40?
Median income isn’t a normal distribution (it would be great if it was!) It more closely follows a gamma distribution, so you need to be careful (in a general sense) of applying anything that has an assumption of normal distributions
False discovery rate adjustment may also be a good idea, but you can play around with p values all day if you want.
Some counties may distribute resources to more teams so everyone gets some sort of experience rather than “stacking” resources on a single team. Just something to consider.
Interesting analysis re winning matches, but I don’t like calling this analysis team “success” and defining “success” as winning matches. I would call “success” something different and much more difficult to analyze with public data: things like student outcomes (college completion or ultimate incomes, or % of careers in science or technology fields) of students on the team being greater than the larger student population at that school. There’s of course a self-selecting bias even there (are students joining teams those already more likely to succeed), but is at least more aligned with the goals of the program.
SpencerS,
I want to open with some positive thoughts. This is a really neat question that you are looking at, and I think as you can see in your data, multiple variables that are influencing your measure. This is a really great example of how easy it can be to gather some data, but how tricky it can be to make a valid conclusion from said data.
As others have pointed out, there are many techniques and specific items you can look at to improve the validity of conclusions you might draw from the data.
And though I am sure your Stats class taught you this, it is always necessary to say:
“Correlation does not always equal causation”
Here is a link to some fun, highly correlated yet highly questionable relationships of pieces of data:
It is definitely worth commending that SpencerA took a “gut feel”, looked into it with data/time at their disposal and communicated the process and initial interpretations.
This is far more that may users on the Internet cough reddit cough do.
You’ve brought up some great points. Hopefully, my response can resolve some of the other questions in this thread as well.
Regarding sample size, I completely agree that 10 teams per week is not nearly enough, but as I was manually entering the data into the sheet, I had to limit it to a relatively small number if I wanted to have a sample from more than one week.
As I was only using data from the 2023 season, I tailored my data choices to this game in particular. It is still definitely imperfect though.
Finding a measure of success that is free of influence from other teams, as well as being easy to analyze and accurate was my largest struggle with this project. I made the decision not to use OPR or EPA as I felt that using a statistic that had already been calculated from a rather complex formula created by someone else was not true to the purpose of this project. I was willing to sacrifice accuracy to create a project that I could label as my own.
It’s been a while since I took stats, so at the moment, no, I do not know of a sampling method that addresses this.
The county median income refers to the median income of all households in the given area (most frequently county)
This was a critical oversight on my end; thanks for bringing it to my attention!
I thought about doing this, but once again it wasn’t practical for the time allotted.
I’ve never heard of this before. Had I known that this was more common practice, I definitely would have changed the project to something that better encapsulates each individual team.
Thanks to all in the thread for your insight so far. I figured there were some things I was missing, hence why I opened the project to criticism. I’m sure there are many people here who are also far more statistically capable than I am, so I wanted to get all of your thoughts on the topic at hand.
I agree. Unfortunately, I did not have the time nor the resources to investigate each team individually, so I made a lot of generalizations throughout the project.
This is the main problem I had with this project, as I previously referenced. I interpreted success as winning matches, as this was the most interesting and relatable to my peers who weren’t interested in robotics (to whom I would also be presenting).
Several people have brought this up but this game may not have been the best to use average score as a good indicator of “success”. A game with more linear scoring like 2022 where the game pieces are always worth the same value and there is no limit to how many can achieve said value would have been better.
I laughed when I saw that 59 was drawn as one of the teams in week 1. 59’s robot was rough in week 1 even they would admit that, but in week 3 Orlando they implemented a new scoring mechanism and had a whole competition of driving experience increasing their average score from 46.5 to 88.6. The average income of the area they were in did not change but because of the week they were assigned thanks to the random draw it where they are represented in the data is
I believe this is their dot on your histogram
Had they been selected after week 1 was filled thus be using their Orlando data this would be roughly where their dot is
I can make the same argument the other way with 86 by picking their second event where they have more practice as opposed to their first they are given credit for 101 average score when a week previous they averaged 85.8.
This also brings up that in general scores go up as the weeks go on as TBA clearly shows us.
A better metric may have been how these teams scores compared to the average score of a given week.
My other issue is using international teams in your data set. Any international team should have been ignored and moved on to another team. Including them in your data set created all 4 of your sub 40K a year teams, while also producing your 2 lowest average scores. There is a ton of reasons for this income disparity but it just created excess noise in the data by creating these outliers.
Yeah, practical considerations ( such as time) with exploratory stuff can get messy.
You would probably want a stratified random sample method, although you would need to decide how to split things up, bins of : percentiles/quantiles? Fixed interval?
In order to do this you would probably want a script that got the lat/lon of the team from TBA api and checked that against tabular census county data the lat lon would need to be converted to county too, so that would be a reverse geocode operation, (alternative may be to grab the county from the team address somehow). Selecting a proportional (or fixed) number of teams from each bin you can then run the stats.
The above indicates a script, which would mean sample size would be addressed for free.
As far as measuring team success there is nothing wrong with hypothesizing a few common sense metrics and trying them. Match scores, wins, OPR, ELO, district points awarded at events (you can calculate these for non distinct events), departures of scores from the weekly mean, etc. Problems come when you test EVERYTHING looking for a significant p value then hypothesizing after the fact (p-hacking)
Re: point 8, not sure how common it is for sure (this will happen naturally with regression towards the mean), but if there are multiple teams in an area I would suspect total local support to be higher vs a single team. However if there was a single team I would expect them to have a larger amount of resources than the multi team option. On the extreme end there is a group of teams in Michigan that are infamous for a few reasons, but match the “distributed” model.
This is also important because it helps decouple the partially random noise the progression of game scores creates.
I.e. a future world champion from a high income area may play week 1 and post an underlying robot contribution score of 100, a low resource team may come along for their first play week 4 and the game has evolved such that they post a underlying robot contribution score of 110. All this just because the community learned to play the game better through a myriad of factors.
I think this is really interesting data, but it does definitely need to be normalized based on week of competition. Using something like the ELO rating or score in the same week of competition would be more personally interesting, but may not pass muster for class.
I would suggest a take away here is the paucity of US teams in counties with sub-40K incomes (and there are hundreds of such counties).
FIRST was an incredible experience for my daughter, but she was going to be ok no matter what – we live in a prosperous “R1” University Town with among the best schools in the state. The real power of FIRST, IMO, is it can completely reshape the future of less advantaged kids (if we can get the program to them).
A worthy project. I’m not surprised at the results. I agree it should be looked at again with some of the suggestions above. I’m trying to sort out some knotty questions on school funding of teams. It would be interesting to know if teams from lower income geography also have lower per team school funding (likely), and what percentage of their budget comes from school systems. You could be absolute grant writing wizards and be well funded in a poor community, but this seems like a long shot. An easier thing to add on, and it would be relevant, is a comparison of community income and survival rate of newly launched teams past year three. For this please leave the international teams out entirely, but I think you’ll find some interesting numbers.
Great, super interesting. As said more resolution on the income of the school will help. I’m in Los Angeles county which includes a couple FRC teams in neighborhood with extremely different wealth. Beverly Hills vs. South Central LA are about as different as possible but all in LA county.
School level wealth data is often estimated with things such as if a school has Title 1 status. Other measures include Total Economically Disadvantaged (% of total), Free Lunch Program (% of total) , Reduced-Price Lunch Program (% of total). I’m not sure how to easily gather this data for thousands of teams but I could see it listed on https://www.usnews.com/education/best-high-schools
The U.S. News source is awesome for measuring the household income of students in schools. If I had known about it when I was gathering my data I definitely would have used i. If I get really bored I might re-run the study with better types of data (% of economically disadvantaged and others).
Here are multiple linear regression results including both income and competition week as independent variables. Week was modeled as a continuous variable.
The income coefficient changes slightly from the author’s univariate regression, but by a small amount. Both income and week are highly significant. Interaction between income and week was not significant. Given the small amount of data in this set, a categorical effect of USA vs non-USA team was also not significant. That doesn’t mean such an effect does not exist, but such an effect is not supported by this data.
To put things in perspective, a $10,000 increase in income results in a score increase of 3.4. Moving from one week to the next in the season results in an 8.0 score increase.
While many valid concerns have been raised about the data, one concern that can be answered is that controlling for competition week does not change the significance of the income effect in this dataset.
