Teachers, assessment and FRC

[dtengineering]

I get what your prof is saying... FIRST and FRC participants, and VEX and VRC participants, and BotBall and BotBall participants, and High School Drag racing participants and Skills Canada participants and all sorts of programs would probably do just fine on standardized pre and post skills inventory tests and sure... it would be interesting to someone, somewhere, to measure that.

But I also get that what your prof is really saying is "We don't have a way to measure passion. We don't have a way to measure inspiration. We can't figure out what the long term impacts are because that will take at least 20 or 30 years and your paper is due Thursday. So let's measure this thing that I think we can measure, regardless of how relevant it is."

And I get that... I really do. I've jumped through the hoops, cited the studies and (amongst some other, more meaningful stuff) earned a couple of education degrees. It's not bad to say we can't measure passion... but we need to be honest about it... and not try and pass off some standardized test as an acceptable proxy.

I think most FRC coaches, mentors, volunteers and sponsors will agree with me when I say that we don't give up our evenings, weekends and (in many cases) a large part of our weekdays so that our students will score three percent higher on their next physics test. Yeah, it's great if they do, but taking away their TV and cramming facts into their head from a textbook will do that just fine, too.

We do it because we see that girl on the build team discover that she's not the only girl who likes to build things. That she's actually pretty normal to want to learn to weld. We do it to see the kid who's maybe not great academically, but awesome mechanically, discover that his talents require just as much smarts as those kids who cruise through math class without even trying. We do it to see that quiet grade 8 kid grow up to be the team captain four years later. We do it to share our passion.

And the only thing that can really measure that is when you look a kid in the eye, say "We've got a problem..." and you see them smiling, "Bring it on."

Good luck with the research... don't let the bastards grind you down!

Jason

[Ian Curtis]

To add to Jason's point, I think FIRST adds a lot to the intangible skill set of future engineers and scientists. FIRST gives you opportunities to make budgets, presentations, and persuade people of your opinion long in advance of your peers.

FIRST's successes won't be measured in terms of % on tomorrow's test, it will be measured by the leaps and bounds of progress wrought by FIRST's alumni.

It's no scientific study, but I've found a pretty impressive correlation in my own college experiences between competent people you can trust to get things done and kids who did FIRST.

Best of luck getting the actual data you need!

[JesseK]

The NWEA test sounds like extra fluff to add to the cost of FRC, IMO. The test simply misses the point: FRC isn't about increasing science literacy. Thus, I have to agree with the professor about your paper.

FRC is about increasing science interest with respect to other careers that do not involve large-scale problem solving. When have official FIRST mediums said otherwise?

IMO, the real measurable 'thing' isn't all that glamorous: we'll know FRC works when the 'professional' unemployment rate for technically-skill jobs isn't at a labor shortfall rate (about 3% unemployment) in a good economy. Currently it averages 4.4% (Table 625, 'Professional and related occupations').

Let me caveat further this with an opinion: this stat doesn't mean that the other jobs are less necessary -- it simply means that people in the U.S. are not following the trend of U.S. jobs to the 'more technical' industries and programs like FIRST are necessary. It means that people are not setting themselves up (via education) to succeed in whatever future industries/problems/situations arise. Many adults are returning to school, which is great. Yet the fact that an increasing % of young students are not moving into educational programs geared for these industries will simply cause more problems in the future.

[Jon Stratis]

FIRST. For Inspiration and Recognition of Science and Technology.

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

That says nothing about teaching them science literacy. Sure, you'll grow that as part of everything else, but the main point of FIRST is to get kids excited about science and technology. To get them passionate. To show them that there are role models other than the sports players, actors, and musicians that get all of our TV time. It's not about teaching kids physics and math. It's about guiding them towards wanting to learn those things on their own.

I'm sure everyone says this about their team, but I truly believe it's nowhere more apparent than on a team like the one I work with - an all girls team. With this team, you really see the growth of every student. They start out with practically no knowledge - It's not that they haven't taken physics or calculus yet, it's that they don't know basic concepts or skills like how to operate a drill. Without this program, they could go through school and might enjoy math or physics, but be completely unaware of the potential life paths those classes open up to them.

My High School didn't have FIRST. In fact, still don't. Looking back on it, I know it wasn't anything I did in High School that inspired me to become an engineer. If you were to go simply by what I did in High School, I'd be off somewhere trying to scrape up minimum wage playing the Saxophone. It was a single teacher back in 5th grade that set my path by offering an after school programming class. That's it - an after school activity that inspired me to become what I am today. That activity didn't increase my science literacy... but it inspired me to keep looking into programming on my own over the following 7 years, and eventually go to college for it. That's what FIRST is all about. I just wish I had it back when I was in school.

[KHall]

Thank-you for your comments.

Lets look at it this way. We at FIRST (specifically the engineers and teachers) claim/feel that what we are doing makes a difference. Testimonials are great, but they can only have so much credibility. I’ve read testimonials from people who claim to have been abducted by aliens and flown to Venus.

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

What we don’t have here is someone with a Ph.D. and a reputation that will get the others in the Ivory Towers to peer review the research and get it published in a respectable journal. That is the key piece. I’ve suggested Dr. Sullivan as a potential person of interest, but I’m sure there are plenty more. Who are they? Can we make a list and start having the FIRST brass approach these people?

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall

[Alan Anderson]

Originally Posted by KHall

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

How does the Brandeis study fall short?

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

Why do we care how FIRST impacts learning? It's not shop class. The goal is always stated using the term "inspire", not the term "teach". Check out the FIRST mission statement:

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

[KHall]

Thank-you Alan, you’re absolutely right. Our mission is to inspire. And if we accept that, then any discussion about learning is moot.

If I can bring Dr. Sullivan’s paper to your attention, and/or ask you to examine the AAAS and NRC standards I believe you’ll discover that FIRST is currently covering most of these standard’s goals. I’m just not sure that too many people realize how much FIRST is achieving.

From Dr. Sullivan’s 2007 research paper regarding AAAS Science standards:
“Robotics learning is strongly linked to these three goals of science literacy. First, robotics study requires utilization of four of the six thinking skills characteristic of scientifically literate people—namely, computation, estimation, manipulation, and observation. Second, students of robotics are engaged in science inquiry through both technological design and computer programming activities. Third, robotics teaches students about systems, one of the common themes in science education.”

So if you’re already achieving something, why not take credit for it?

Plus why would a mission statement prevent you from wanting to show your value to the educational community? Didn’t Dean say he wanted an FRC team in every high school? If you were a superintendent of a school system, wouldn’t you find verified results that align with accepted educational standards carry a lot of weight? I personally believe there are many such people out there who want to do what they know is best for their students. We need to give them what they need -- make it easier for them to adopt FIRST.

If nobody here sees the value of this, then I’ll drop it. I believe that we need to find ways to push more proven constructivist methods into the classrooms, and frankly, FIRST is low hanging fruit. The main idea here was assessment, and true to CD, we’ve drifted a bit.

FIRST is not considered a professionally administered education program because we do not currently perform formal assessments. Teachers are experts at assessment, and I am not. I simply asked if anyone was aware of the NWEA testing suggested by a professor who was trying to be helpful because he wanted to find a way to push FIRST (or at least its project oriented methods) into more schools. I still see no harm in asking about that.

KHall

[ajd]

Originally Posted by Alan Anderson

How does the Brandeis study fall short?
Why do we care how FIRST impacts learning? It's not shop class. The goal is always stated using the term "inspire", not the term "teach". Check out the FIRST mission statement:

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

Learning is not the primary goal of the program, but it's a really nice side-effect.

[Ian Curtis]

Originally Posted by KHall

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

I think part of the issue you'll have is the level of proof. Engineers and most scientists must be very sure of their data before we publish/build. When I took statistics, the professor mentioned that Psychologists get excited about r^2 values of .15 (for those you that haven't taken stats, that means 15% of the variation in the output can be explained by the input). If I was trying to characterize a dataset as an engineer, an r^2 value of .15 would get me laughed out of the room! On the other hand, if the Education Ivory Tower folks accept an r^2 of .15 (and it sounds like they do) then this works in your favor.

Also, I'm pretty sure the NWEA tests are not particularly difficult to administer. My mom is a Teacher's Aide/School Librarian for an elementary school and has been tasked with them many times because she has some computer savvy (i.e. knows how to use one). It's just a webpage that the kids click through, and the difficulty of the questions scale with how good they are at answering them.

[dtengineering]

Originally Posted by KHall

Thank-you for your comments...

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. ....

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall

I hope everyone agrees that you are on FIRST's side. And the side of VEX and Science Fairs and everyone else who gives a darn about making science, technology, engineering and our whole darn world more exciting and engaging for young people.

But education treads an interesting line between a science and an art. There are some things we can do and measure... certain classroom strategies and class sizes and compositions that work... and other things that we do are like music or painting. Is John Williams a better composer than Beethoven? Maybe in 200 years we'll have some perspective on that.

My background is in engineering. I love science. I love facts. I love the scientific method. But my profession is education, and my scientific opinion is that people... especially young people... are weird and wonderful beings that rarely conform to efficient experiments when one tries to measure the things that really matter.

Come up with a test that measures passion and inspiration, and your Ph.D awaits.

Jason

P.S. Williams>Beethoven, but my opinion is currently swayed by having watched Star Wars on Blu Ray twice tonight.

[KHall]

Sorry, no advanced degrees for that. Its been done.

Key search terms: “measures passion”

3,144 Results for Primo Central sorted by:relevance
Show only: Cited Articles(79) Peer-reviewed Journals(2,141)

(examples)

The Role of Passion for Teaching in Intrapersonal and Interpersonal Outcomes
Carbonneau, Noémie ; Vallerand, Robert J ; Fernet, Claude ; Guay, Frédéric
Journal of Educational Psychology, 2008, Vol. 100(4), p.977-987 [Peer Reviewed Journal]
less adaptive outcomes (e.g., shame and negative affect). In this study, 494 teachers completed measures of passion for teaching and various

Grit: perseverance and passion for long-term goals
Duckworth, Angela L ; Peterson, Christopher ; Matthews, Michael D ; Kelly, Dennis R
Journal of personality and social psychology, Jun, 2007, Vol.92(6), p.1087-101 [Peer Reviewed Journal]


Key search terms: “measures inspiration”

5,912 Results for Primo Central sorted by:relevance
Show only: Cited Articles(236) Peer-reviewed Journals(4,040)

(examples)

Inspiration as a Psychological Construct
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2003, Vol. 84(4), p.871-889 [Peer Reviewed Journal]
study of inspiration, both as a general construct and in specific content domains (e.g., religion, creativity, interpersonal relations). Inspiration is an experience with which we are all familiar.

Inspiration
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2004, Vol. 87(6), p.957-973 [Peer Reviewed Journal]

Abstract:
The authors examined the core characteristics, component processes, antecedents, and function of state inspiration. In Studies 1 and 2, inspiration was contrasted with baseline experience and activated positive affect (PA) using a vivid recall methodology. Results supported the tripartite conceptualization of inspiration. Confirmatory factor analysis indicated that inspiration may be decomposed into separate processes related to being inspired "by" and being inspired "to." Study 3 found that daily inspiration is triggered by illumination among individuals high in receptive engagement, whereas activated PA is triggered by reward salience among individuals high in approach temperament. Approach temperament was also implicated in being inspired "to." Inspiration and activated PA appear to serve different functions: transmission and acquisition, respectively. (PsycINFO Database Record (c) 2011 APA, all rights reserved) (from the journal abstract)

[dtengineering]

<chuckles>

That's the beauty of educational research, though... you can always explain why the "other guy/gal" did it wrong.

Jason