Disclaimer: this is a thread looking for feedback on the formatting of a survey. If any of you want to use this thread as a soapbox to voice your opinions on the mentor involvement debate, please, please don’t. If you can’t help yourself, I ask that you look up some thread which is already a dumpster fire where the topic is already being discussed and post your opinions there instead.
I’m looking to conduct a survey about mentor involvement (similar to the one in Analysis of Student vs Mentor built Robots in FRC). I’m doing this purely out of personal interest, and I would share the results with the CD community. Since I want the results to be as useful to you guys as possible, before I conduct the survey, I’d appreciate any feedback on how the questions could be improved (e.g. expanded to be more useful, or worded / formatted to be more neutral). This is my current draft of the survey questions:
Are you currently an FRC student, alum, or mentor? Select all that apply.
How many years have you been participating in FIRST? (Text box)
How many years have you been participating in FRC? (Text box)
Please answer the following questions about a single team you have been on. If you would like to answer these questions again (in reference to a different team), you will have the option to do so at the end of the survey. If you would like to answer these questions about a team you are no longer on, let your answers be as they were when you were on the team (e.g. in regards to team age, competitiveness, etc.).
Roughly how many students does your team have? (Text box)
Roughly how many mentors does your team have? (Text box)
Roughly how many years old is your team? (Text box)
Compared to the teams you compete against, which of the following do you think best describes your team?
Select one: Low resource, Mid resource, High resource, I don’t know
Select one of the following for each: Not at all, Hardly at all, Somewhat, Very, I don’t know or I have no opinion
How do you think your team’s level of mentor involvement impacts your students’:
empowerment
learning
competitive success
Select one of the following for each: Strongly hurts, Somewhat hurts, Does not affect, Somewhat helps, Strongly helps, I don’t know or I have no opinion
Optional: Explain how and why you feel your team’s level of mentor involvement works or doesn’t work. (Text box)
Would you like to fill out the survey again for another team you’ve been on? (Yes or no — no ends the survey, yes leads to a repeat of questions 4-11).
For now, I’ve deliberately avoided asking for team numbers (and I would censor any team-identifying info when releasing the results), because I think maximizing anonymity is more important than any usefulness such info may provide. I’ve also tried to avoid things like “Scale of 1 to 10” because handling/interpreting those kinds of numbers can easily get screwy (see: [FRC Blog] Two Championship Survey Results and Path Forward - #9 by Holtzman).
This is much more neutral than the other survey. I’m not sure we’ll see any noticeable trends from this survey, but I’m sure all of CD will interpret them in an unbiased way and leave emotions out of all debates surrounding this. sooooo sure
You said this was purely for your own interest - what in particular are you interested in learning? As a mentor invested in continual improvement, I’m always interested in hearing other mentor’s mentorship philosophies and how they view their role and the value they bring to their team, and in hearing from students about where they wish their mentors would step forward or step back more. Your interests may be different from mine, but from reading your questions it’s not clear what you are trying to learn. Feels generic, I guess.
I think most people’s answers to questions 8 and 9 would be “it’s complicated”, and those questions might be better served by having an optional text box with them. For example, a mentor who does all the CAD themself, a mentor who sits with the CAD team all day every day to bounce ideas of each other and reinforce best practices, and a mentor who intensively teaches freshmen how to use OnShape and carry out small chunks of the CAD work while the student captain takes point on all the major design decisions, could all be described as “Very Directly Involved”. Same for the other subheadings of question 8.
Someone on CD (wish I remembered who) has described their approach to teaching a student a skill as progressing through four stages:
Mentor does, student watches
Mentor does, student helps
Student does, mentor helps
Student does, mentor watches
Perhaps framing the question in terms of where the team tends to land in that progression would be more useful than “how directly involved”. Some teams have a lot of 1 and rarely reach 4, some teams skip 1 and start at 2, etc.
Is this survey intended to be robot-centric? I only ask because a teams competitiveness on the field does not always reflect their competitiveness for awards. A team could be in perennial consideration for Chairman’s, yet not make it to the playoffs each year. Likewise, teams that regularly make it to the playoffs may never be in consideration for Chairman’s or EI. I would find it interesting to separate out robot build from awards prep involvement.
Have you considered also asking what the level of student involvement is in those areas? There’s a big difference between mentors Very Involved/students Hardly at All and mentors Very Involved/Students Very Involved.
I’d be curious to know if a participating team sees themselves as working towards an ideal student/mentor relationship or if they view themselves as having obtained it. Basically, how do teams perceive the student/mentor relationship on their own team in light of what FIRST describes it as or what they perceive it as. Further, do they feel that FIRST does enough to describe the relationship and setting goals for what it should look like or is it left undefined?
This is a weird question in that a team from the Midwest will have a wildly different competitive environment than a team from California and I think that will de-normalize the data. I don’t know how to improve it , but its one potential flaw I see.
Survey looks pretty good, although I agree that question 8 is an “it’s complicated”. I spend a lot of time reviewing designs and teaching machining stuff, but I try not to make anything directly for the team (although that’s slipped a bit this year). But to pretend that I’m not “very involved” with the design and manufacturing would be a mistake- I’m involved, but more through teaching than through doing.
Maybe have separate categories for “doing” and “teaching”. How much of what you design goes on the robot? How much time do you spend helping students design? And same for manufacturing.
Honestly, I’m interested in a bunch of things, which I agree makes the survey feel generic. Ideally, I want this survey to collect enough data s.t. it’d be useful to answer whatever questions you guys have (within reason).
Good catch. I’ll separate competitiveness between on-field and awards/culture.
That’s a really great point that I totally blanked on. I would probably try to account for that by adding a field for “differs strongly year to year” and then have a follow-up question for specifying the range + a text box for elaborating.
So that was intentional, the idea being that we should compare teams to whoever they interact with most as a way of normalizing for differences in average budget and competitiveness across regions. Is there a better way you think to handle that? I considered having users specify their region, but I worry that would lead to anonymity issues.
Good question. I’ll add fields that can help answer that.
On the “low resource, mid resource, high resource”, you may want to break this into a number of questions, such as “number/skill/interest of mentors/coaches”, “number/skill/interest of students”, “number/skill/interest of parents and community support”, “facilities”, “fabrication capability”, “money for parts and supplies”
Defining competitive success more clearly should also be reasonable - something like “we win most to all of our local events”, “we win some local events, and are usually alliance captains or first picks”, “we make eliminiations at most local events”, “we make eliminations sometimes”, “we qualify for eliminations occasionally, rarely, or never”.
This is good phrasing to prompt people to compare themselves to their own regions. Do you think it would be useful to ask respondents to quantify this in some way? Maybe asking them to provide the mathematical average of competitions they have gotten into the playoffs at? As well as made it to Finals or won?
I think that “mentor involvement” and “program success” are very soft subjects, so having the questions be more qualitative instead of quantitative makes sense.
I think it’s unreasonable to try to get to “Increasing mentor involvement X% causes a Y% change to average event performance, while also having a Z% change to student inspiration.” I think we can expect general trends at best, and these questions do a good job at getting to those without unnecessary specificity.
My feedback is similar to Marshall’s. We have goals of mentor / student balance. It would be better to ask about team goals and barriers to achieve goals. Actual results vary each year.
The best part of your survey are questions 9 and 10. Those are more open ended questions that don’t limit the topic to robot performance. I would swap out “competivive success” with “achieving team goals.”
Questions 7 and 8 seem to imply you are trying to correlate resources and mentor involvement to robot performance. Is that your intent?
I will definitely break down the different types of resources as you’ve suggested, because (as the question is written now) some users could feel conflicted on what to put if the subcategories you’ve described vary wildly.
I’m reluctant to start handling numbers or separating types of on-field success, mainly because I worry doing so would overcomplicate things for little benefit. I’m still open to more discussion on that though. Looking at the question more, I will at least widen the spectrum of responses (e.g. to “Well below average”, “Somewhat below average”, “Average”, “Somewhat above average”, “Well above average” and “I don’t know”).
Oh, whoops, that was not the intent; I just thought it would be neat to be able to separate the data based on competitiveness and resource level. I’m doing this more out of general curiosity than to test a single hypothesis. I’ll think more about how to not give that impression.
You may find it more effective to split some of this data up.
Such as: “the building if the robot” being turned into “the manufacturing of the robot” and “the assembly of the robot.”
I know lots of teams have school rules regarding what tools students can run. So while every single CNC part might have to be made by a mentor, I wouldn’t exactly classify that as “mentor built” if the students are the ones assembling everything.
And: “strategy during competition” could become “strategy during matches” and “strategy for a picklist.”
Those are two very different things IMO. At least on 649, the mentors have little to no say regarding the match strategy. However the students want as much input as possible when deciding who they want to pick (if applicable).
So much depends on the perspective of the individual. I think other than mentors with a first history, most individuals only have experience with one team to draw conclusions from about mentor involvement. This could affect your survey results. My kids have been involved with two different teams and I’ve mentored with both. If you ask kids on each team that only have one perspective and no comparison point, both would say the teams are student led. However, our perspective seeing two different teams operate, is much different.
The team we left (seeking more student engagement) has a 2:1 student to mentor ratio and the mentors are very hands on in all aspects of team function, robot design and build to the point that many students watch the process. At competition, there are usually more mentors than students in the pits and mentors constantly have their hands on the robot. Mentors lead all the meetings and have dominant input onto team decisions including design concepts and game strategy. The students however have been told this is normal and accepted. Their engagement at events is certainly mirrored during build season. Yet their students will tell me that they are student led as far as the First guidelines suggest.
The team we joined is extremely student led. We rarely have any mentors in the pits, our meetings are led and organized by the students and the few mentors we have engage with the students as teachers/educators. The student engagement is high and even though the students can get afield from their objectives, they are getting the hands on learning which is becoming more absent from the school system. We merely act as guideposts. For example, i as strategy mentor, will consult with the drive team 3-4 times a tournament and usually when they have a question for me. I will tell them to use their best judgement in strategy and consider their risk/reward equation with their decisions. When we select an elimination alliance, our students work out game strategy with the other drive teams, no adults.
I would even go so far as to say that our electrical mentor (term used loosely here) is one of the teams sophomores who designed the electrical layout, determined the parts to acquire, helped other students learn about wiring/components/power supply/etc. He simply knows more about the system and components than any adult in the team room.
Based on our experience with these teams, I believe perception is everything. All I’m saying is seek out multiple perspectives for a truly rounded point of view. I’ve used this experience to educate my kids that workplace culture can be critical to your job satisfaction and if you find yourself in an unsatisfying work culture, take the initiative to make a change.
And just to close, both teams are competitively successful and won multiple awards so you don’t have to sacrifice student experience for on field results.
All I’m saying is seek out multiple perspectives for a truly rounded point of view. I’ve used this experience to educate my kids that workplace culture can be critical to your job satisfaction and if you find yourself in an unsatisfying work culture, take the initiative to make a change.
Interesting to point out that the person who is putting this survey together was actually on two separate FRC teams herself. Not only that but the second team she ended up on (which is based at the USA’s first public residential high school with a STEM focused curriculum - the school predates FIRST’s founding by nearly a decade) has become a cultural amalgam of students from FRC, FTC, VEX, FLL, and other various STEM related programs throughout our state.
All of the Zebracorns these days have experience with the idea that different FRC teams have different cultures - it is something we talk about regularly, adapt to constantly, and work to better ourselves with.
My comment was more about potential respondents and taking into account a narrow point of view. I was a market researcher for 11 years doing data analysis for primary research. I think directing the questions to student satisfaction with adult involvement and opportunities gained/lost might enhance the results. Thanks for your thoughts Marshall.
