Teachers, assessment and FRC

[Ian Curtis]

Originally Posted by KHall

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

I think part of the issue you'll have is the level of proof. Engineers and most scientists must be very sure of their data before we publish/build. When I took statistics, the professor mentioned that Psychologists get excited about r^2 values of .15 (for those you that haven't taken stats, that means 15% of the variation in the output can be explained by the input). If I was trying to characterize a dataset as an engineer, an r^2 value of .15 would get me laughed out of the room! On the other hand, if the Education Ivory Tower folks accept an r^2 of .15 (and it sounds like they do) then this works in your favor.

Also, I'm pretty sure the NWEA tests are not particularly difficult to administer. My mom is a Teacher's Aide/School Librarian for an elementary school and has been tasked with them many times because she has some computer savvy (i.e. knows how to use one). It's just a webpage that the kids click through, and the difficulty of the questions scale with how good they are at answering them.

[ebarker]

Originally Posted by KHall

'until you can measure meaningful progress, you're just a bunch of people who play with robots'.

It is generally true that if you cannot measure what you are doing, you cannot manage what you are doing.

The purpose, mission, and vision of FIRST isn't to teach or improve scientific literacy, but to inspire. And as any teacher will tell you, an inspired student is more desirable than an resistant student. There are a million stories about how a student growing up was inhibited from STEM activities because the teachers, the community, and their peers viewed it as a negative activity. FIRST seeks to correct that. And yes, some of my strongest NEGATIVE memories as a child regarding STEM came from the teachers.

I am suggesting that your professor is measuring the INCORRECT thing !!!!

For whatever reason, I would suggest that your professor fails to recognize the linkage between an positive viewpoint toward STEM versus a negative viewpoint. It is that simple.

Just an hour ago I received an email from "Education Week" with this quote in an article: "The problem, then and now, is that the architects of futurist school reform are too invested in the status quo to tolerate, let alone foster, much subversion, criticism, or free thinking." Education Week

All too often the most resistance to change are those inside the school system.

Hundreds of thousands of students, 85,000+ volunteers and mentors, over 3,500 institutions and corporations outside the ivory tower that contribute $ 35M+ in direct costs and probably 3 times that in other costs, $ 14M in scholarships disagree with your professors assertion. This money is freely given and NOT taxes.

There are other perspectives and opinions about the value of FIRST and you don't need to have a graduate degree in education to recognize this, with all due respect.

[Tristan Lall]

This test?

It doesn't look like the MAP test is particularly relevant to robotics, apart from the physics. That's a couple steps removed from mechanical or electrical design, and definitely not a good proxy for "inspiration" (which probably has a lot to do with motivation and a personal sense of empowerment, and not just being conversant in the concepts of science).

Plus, it says the test is for grade 10 and below, and calibrated to state science curriculum standards. Neither do they seem to explain what their "two different scales" and "two scores" are. That doesn't sound too helpful at all.

Aside: Incidentally, any website that puts an "®" character in a URL (except for deliberate obfuscation) strikes me as being fundamentally out of touch with a technical audience....

[dtengineering]

Originally Posted by KHall

Thank-you for your comments...

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. ....

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall

I hope everyone agrees that you are on FIRST's side. And the side of VEX and Science Fairs and everyone else who gives a darn about making science, technology, engineering and our whole darn world more exciting and engaging for young people.

But education treads an interesting line between a science and an art. There are some things we can do and measure... certain classroom strategies and class sizes and compositions that work... and other things that we do are like music or painting. Is John Williams a better composer than Beethoven? Maybe in 200 years we'll have some perspective on that.

My background is in engineering. I love science. I love facts. I love the scientific method. But my profession is education, and my scientific opinion is that people... especially young people... are weird and wonderful beings that rarely conform to efficient experiments when one tries to measure the things that really matter.

Come up with a test that measures passion and inspiration, and your Ph.D awaits.

Jason

P.S. Williams>Beethoven, but my opinion is currently swayed by having watched Star Wars on Blu Ray twice tonight.

[KHall]

Sorry, no advanced degrees for that. Its been done.

Key search terms: “measures passion”

3,144 Results for Primo Central sorted by:relevance
Show only: Cited Articles(79) Peer-reviewed Journals(2,141)

(examples)

The Role of Passion for Teaching in Intrapersonal and Interpersonal Outcomes
Carbonneau, Noémie ; Vallerand, Robert J ; Fernet, Claude ; Guay, Frédéric
Journal of Educational Psychology, 2008, Vol. 100(4), p.977-987 [Peer Reviewed Journal]
less adaptive outcomes (e.g., shame and negative affect). In this study, 494 teachers completed measures of passion for teaching and various

Grit: perseverance and passion for long-term goals
Duckworth, Angela L ; Peterson, Christopher ; Matthews, Michael D ; Kelly, Dennis R
Journal of personality and social psychology, Jun, 2007, Vol.92(6), p.1087-101 [Peer Reviewed Journal]


Key search terms: “measures inspiration”

5,912 Results for Primo Central sorted by:relevance
Show only: Cited Articles(236) Peer-reviewed Journals(4,040)

(examples)

Inspiration as a Psychological Construct
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2003, Vol. 84(4), p.871-889 [Peer Reviewed Journal]
study of inspiration, both as a general construct and in specific content domains (e.g., religion, creativity, interpersonal relations). Inspiration is an experience with which we are all familiar.

Inspiration
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2004, Vol. 87(6), p.957-973 [Peer Reviewed Journal]

Abstract:
The authors examined the core characteristics, component processes, antecedents, and function of state inspiration. In Studies 1 and 2, inspiration was contrasted with baseline experience and activated positive affect (PA) using a vivid recall methodology. Results supported the tripartite conceptualization of inspiration. Confirmatory factor analysis indicated that inspiration may be decomposed into separate processes related to being inspired "by" and being inspired "to." Study 3 found that daily inspiration is triggered by illumination among individuals high in receptive engagement, whereas activated PA is triggered by reward salience among individuals high in approach temperament. Approach temperament was also implicated in being inspired "to." Inspiration and activated PA appear to serve different functions: transmission and acquisition, respectively. (PsycINFO Database Record (c) 2011 APA, all rights reserved) (from the journal abstract)

[dtengineering]

<chuckles>

That's the beauty of educational research, though... you can always explain why the "other guy/gal" did it wrong.

Jason

[JuliaGreen]

Something to think about for the action research - Is there a relationship between participation in FIRST and students taking "higher level" science and math courses? Look at the grades earned by students in and out of FIRST. If you have AP or ACT subtest scores available - look at the scores from students who do and do not participate in FIRST. Action research doesn't require you to be a psychometrician, but you'll likely find some interesting trends.

Constructivism is messy and loud and doesn't flow in a linear fashion. But it works.

Julia - NBCT EA Science

[Taylor]

I'm sure you've already looked at it, but why not use the information FIRST provides? There are direct links between college preparation and acceptance, conducted by independent firms, that show FIRST helps with STEM-related careers.

[ajd]

Originally Posted by Alan Anderson

How does the Brandeis study fall short?
Why do we care how FIRST impacts learning? It's not shop class. The goal is always stated using the term "inspire", not the term "teach". Check out the FIRST mission statement:

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

Learning is not the primary goal of the program, but it's a really nice side-effect.