Teachers, assessment and FRC
 

This is a big hello to all the experienced teachers out there. I’m taking my final science methods class before student teaching this spring. Passed the Praxis II Physics test last Nov.

My current methods class requires a short research paper. I wanted to do something about robotics as inquiry learning, but found that general research on the subject was somewhat thin. There are a couple of excellent papers by Dr. Sullivan, Department of Teacher Education and Curriculum Studies, University of Massachusetts, Amherst on her web site: http://people.umass.edu/florence/papers.htm She really nails the connections between robotics and the AAAS & NRC science standards in her 2007 paper and discusses some rather practical aspects of inquiry based learning techniques in robotics in her 2009 paper. Good stuff.

But my professor was unimpressed. He wanted to know if FIRST had ever done a NWEA “growth in science literacy test”. He described it as a pre- and post- test that measures the growth of science literacy. I’ve found their web site, http://www.nwea.org but honestly it does not mean a lot to me. I did get this much, apparently its not a test that can be administered by just anyone, you have to go to one of their sites to take the test. That and they have a database to compare the results of a given student to students with similar demographics.

Its too late to save my research paper, but I would still like to know if anyone is doing this kind of assessment, and if we have some sort of measure acceptable to the Ivory Tower folks that FIRST does improve science literacy skills? If so, please let me know. Not out to prove anything to anyone, just want to learn more about these things from those of you who know.

Thank-you,
Keith Hall

Re: Teachers, assessment and FRC
 

I can't say I've ever heard of the nwea.org test. The only assessments that I can think of that I have administered to check for prior knowledge, literacy, or whatever you want to call prior learning have been either the Mechanics Baseline Test or the Force Concept Inventory and that when was when I was teaching a second year mechanics class that we have since phased out in favor of AP - B and or C.

I'm not even sure how to quantify what a student learns in FIRST in the sense of a curriculum, especially if you are not limiting it to just FRC.

Re: Teachers, assessment and FRC
 

I completely forgot about something that might help you: http://modeling.asu.edu/

While not FIRST related it can provide you some detailed information about inquiry, specifically physical science.

Re: Teachers, assessment and FRC
 

Well, what you say supports my professor's position. He said it nicely but the gist was 'until you can measure meaningful progress, you're just a bunch of people who play with robots'.

That's probably the Ivory Tower way of looking at FIRST. He did admit that project-based and open-ended, inquiry based methods have been proven to work. And that FIRST makes those approaches possible, but -- we've never directly measured and documented that FIRST’s programs are proven to work in a scientifically valid study. (That I could find!)

The NWEA test he suggested may be one possible way to establish that connection. My professor is not belittling FIRST. In fact, I believe he wants someone to do this so that FIRST can gain more credibility in his circles. Remember it’s the Ivory Tower folks who are writing the next round of standards, making methods recommendation, etc. I think he really wants FIRST to be a scientifically proven program so he can use some or all of it.

We’ve seen major progress towards the dream this year. The state of Minnesota officially recognized FIRST FRC as a sport. Wil i am has done us a tremendous favor by spreading the word. Great things are happening. If we are going to change the culture we need valid proof that what we are doing is more than just playing with robots. If we had proof that FIRST programs (for example) increase science literacy skills 30% faster or higher than the average student, we’d have another gem in our toolboxes.

But as I mentioned before, when I went to look for the research that verifies that FIRST is proven to work, I found nothing. So I propose we beg, badger or cajole someone like Dr. Sullivan (a known robotics in education fan and respected researcher) to help us do exactly that. NWEA might be the answer or at least a starting point that could force the Ivory Tower to take notice of FIRST. We need some quantifiable scientific proof that our programs result in learning.

How do we get that ball rolling?

Keith Hall

Re: Teachers, assessment and FRC
 

I'll agree that this is an area that FIRST HQ has not been incredibly focused on over the years, and would probably benefit by having more substantial quantifiable data to show exactly what we are doing to these students.

Re: Teachers, assessment and FRC
 

Have you read the Brandeis study? http://usfirst.org/aboutus/impact

Re: Teachers, assessment and FRC
 

Yes, thank-you I've seen that study. As near as I can tell, its an attitude assessment, not a learning assessment. What people say they prefer and what you can measure about what they have learned are different things.

Re: Teachers, assessment and FRC
 

I think this is an interesting thread, have you tried talking to your senior mentor in the area or FIRST headquarters about pursuing this idea?

Re: Teachers, assessment and FRC
 

I get what your prof is saying... FIRST and FRC participants, and VEX and VRC participants, and BotBall and BotBall participants, and High School Drag racing participants and Skills Canada participants and all sorts of programs would probably do just fine on standardized pre and post skills inventory tests and sure... it would be interesting to someone, somewhere, to measure that.

But I also get that what your prof is really saying is "We don't have a way to measure passion. We don't have a way to measure inspiration. We can't figure out what the long term impacts are because that will take at least 20 or 30 years and your paper is due Thursday. So let's measure this thing that I think we can measure, regardless of how relevant it is."

And I get that... I really do. I've jumped through the hoops, cited the studies and (amongst some other, more meaningful stuff) earned a couple of education degrees. It's not bad to say we can't measure passion... but we need to be honest about it... and not try and pass off some standardized test as an acceptable proxy.

I think most FRC coaches, mentors, volunteers and sponsors will agree with me when I say that we don't give up our evenings, weekends and (in many cases) a large part of our weekdays so that our students will score three percent higher on their next physics test. Yeah, it's great if they do, but taking away their TV and cramming facts into their head from a textbook will do that just fine, too.

We do it because we see that girl on the build team discover that she's not the only girl who likes to build things. That she's actually pretty normal to want to learn to weld. We do it to see the kid who's maybe not great academically, but awesome mechanically, discover that his talents require just as much smarts as those kids who cruise through math class without even trying. We do it to see that quiet grade 8 kid grow up to be the team captain four years later. We do it to share our passion.

And the only thing that can really measure that is when you look a kid in the eye, say "We've got a problem..." and you see them smiling, "Bring it on."

Good luck with the research... don't let the bastards grind you down!

Jason

Re: Teachers, assessment and FRC
 

To add to Jason's point, I think FIRST adds a lot to the intangible skill set of future engineers and scientists. FIRST gives you opportunities to make budgets, presentations, and persuade people of your opinion long in advance of your peers.

FIRST's successes won't be measured in terms of % on tomorrow's test, it will be measured by the leaps and bounds of progress wrought by FIRST's alumni.

It's no scientific study, but I've found a pretty impressive correlation in my own college experiences between competent people you can trust to get things done and kids who did FIRST.

Best of luck getting the actual data you need! :cool:

Re: Teachers, assessment and FRC
 

The NWEA test sounds like extra fluff to add to the cost of FRC, IMO. The test simply misses the point: FRC isn't about increasing science literacy. Thus, I have to agree with the professor about your paper.

FRC is about increasing science interest with respect to other careers that do not involve large-scale problem solving. When have official FIRST mediums said otherwise?

IMO, the real measurable 'thing' isn't all that glamorous: we'll know FRC works when the 'professional' unemployment rate for technically-skill jobs isn't at a labor shortfall rate (about 3% unemployment) in a good economy. Currently it averages 4.4% (Table 625, 'Professional and related occupations').

Let me caveat further this with an opinion: this stat doesn't mean that the other jobs are less necessary -- it simply means that people in the U.S. are not following the trend of U.S. jobs to the 'more technical' industries and programs like FIRST are necessary. It means that people are not setting themselves up (via education) to succeed in whatever future industries/problems/situations arise. Many adults are returning to school, which is great. Yet the fact that an increasing % of young students are not moving into educational programs geared for these industries will simply cause more problems in the future.

Re: Teachers, assessment and FRC
 

Thank-you for your comments.

Lets look at it this way. We at FIRST (specifically the engineers and teachers) claim/feel that what we are doing makes a difference. Testimonials are great, but they can only have so much credibility. I’ve read testimonials from people who claim to have been abducted by aliens and flown to Venus.

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

What we don’t have here is someone with a Ph.D. and a reputation that will get the others in the Ivory Towers to peer review the research and get it published in a respectable journal. That is the key piece. I’ve suggested Dr. Sullivan as a potential person of interest, but I’m sure there are plenty more. Who are they? Can we make a list and start having the FIRST brass approach these people?

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall

Re: Teachers, assessment and FRC
 

FIRST. For Inspiration and Recognition of Science and Technology.

That says nothing about teaching them science literacy. Sure, you'll grow that as part of everything else, but the main point of FIRST is to get kids excited about science and technology. To get them passionate. To show them that there are role models other than the sports players, actors, and musicians that get all of our TV time. It's not about teaching kids physics and math. It's about guiding them towards wanting to learn those things on their own.

I'm sure everyone says this about their team, but I truly believe it's nowhere more apparent than on a team like the one I work with - an all girls team. With this team, you really see the growth of every student. They start out with practically no knowledge - It's not that they haven't taken physics or calculus yet, it's that they don't know basic concepts or skills like how to operate a drill. Without this program, they could go through school and might enjoy math or physics, but be completely unaware of the potential life paths those classes open up to them.

My High School didn't have FIRST. In fact, still don't. Looking back on it, I know it wasn't anything I did in High School that inspired me to become an engineer. If you were to go simply by what I did in High School, I'd be off somewhere trying to scrape up minimum wage playing the Saxophone. It was a single teacher back in 5th grade that set my path by offering an after school programming class. That's it - an after school activity that inspired me to become what I am today. That activity didn't increase my science literacy... but it inspired me to keep looking into programming on my own over the following 7 years, and eventually go to college for it. That's what FIRST is all about. I just wish I had it back when I was in school.

Re: Teachers, assessment and FRC
 

How does the Brandeis study fall short?

Why do we care how FIRST impacts learning? It's not shop class. The goal is always stated using the term "inspire", not the term "teach". Check out the FIRST mission statement:

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

Re: Teachers, assessment and FRC
 

Thank-you Alan, you’re absolutely right. Our mission is to inspire. And if we accept that, then any discussion about learning is moot.

If I can bring Dr. Sullivan’s paper to your attention, and/or ask you to examine the AAAS and NRC standards I believe you’ll discover that FIRST is currently covering most of these standard’s goals. I’m just not sure that too many people realize how much FIRST is achieving.

From Dr. Sullivan’s 2007 research paper regarding AAAS Science standards:
“Robotics learning is strongly linked to these three goals of science literacy. First, robotics study requires utilization of four of the six thinking skills characteristic of scientifically literate people—namely, computation, estimation, manipulation, and observation. Second, students of robotics are engaged in science inquiry through both technological design and computer programming activities. Third, robotics teaches students about systems, one of the common themes in science education.”

So if you’re already achieving something, why not take credit for it?

Plus why would a mission statement prevent you from wanting to show your value to the educational community? Didn’t Dean say he wanted an FRC team in every high school? If you were a superintendent of a school system, wouldn’t you find verified results that align with accepted educational standards carry a lot of weight? I personally believe there are many such people out there who want to do what they know is best for their students. We need to give them what they need -- make it easier for them to adopt FIRST.

If nobody here sees the value of this, then I’ll drop it. I believe that we need to find ways to push more proven constructivist methods into the classrooms, and frankly, FIRST is low hanging fruit. The main idea here was assessment, and true to CD, we’ve drifted a bit.

FIRST is not considered a professionally administered education program because we do not currently perform formal assessments. Teachers are experts at assessment, and I am not. I simply asked if anyone was aware of the NWEA testing suggested by a professor who was trying to be helpful because he wanted to find a way to push FIRST (or at least its project oriented methods) into more schools. I still see no harm in asking about that.

KHall

Re: Teachers, assessment and FRC
 

I think part of the issue you'll have is the level of proof. Engineers and most scientists must be very sure of their data before we publish/build. When I took statistics, the professor mentioned that Psychologists get excited about r^2 values of .15 (for those you that haven't taken stats, that means 15% of the variation in the output can be explained by the input). If I was trying to characterize a dataset as an engineer, an r^2 value of .15 would get me laughed out of the room! On the other hand, if the Education Ivory Tower folks accept an r^2 of .15 (and it sounds like they do) then this works in your favor.

Also, I'm pretty sure the NWEA tests are not particularly difficult to administer. My mom is a Teacher's Aide/School Librarian for an elementary school and has been tasked with them many times because she has some computer savvy (i.e. knows how to use one). It's just a webpage that the kids click through, and the difficulty of the questions scale with how good they are at answering them.

Re: Teachers, assessment and FRC
 

It is generally true that if you cannot measure what you are doing, you cannot manage what you are doing.

The purpose, mission, and vision of FIRST isn't to teach or improve scientific literacy, but to inspire. And as any teacher will tell you, an inspired student is more desirable than an resistant student. There are a million stories about how a student growing up was inhibited from STEM activities because the teachers, the community, and their peers viewed it as a negative activity. FIRST seeks to correct that. And yes, some of my strongest NEGATIVE memories as a child regarding STEM came from the teachers.

I am suggesting that your professor is measuring the INCORRECT thing !!!!

For whatever reason, I would suggest that your professor fails to recognize the linkage between an positive viewpoint toward STEM versus a negative viewpoint. It is that simple.

Just an hour ago I received an email from "Education Week" with this quote in an article: "The problem, then and now, is that the architects of futurist school reform are too invested in the status quo to tolerate, let alone foster, much subversion, criticism, or free thinking." Education Week

All too often the most resistance to change are those inside the school system.

Hundreds of thousands of students, 85,000+ volunteers and mentors, over 3,500 institutions and corporations outside the ivory tower that contribute $ 35M+ in direct costs and probably 3 times that in other costs, $ 14M in scholarships disagree with your professors assertion. This money is freely given and NOT taxes.

There are other perspectives and opinions about the value of FIRST and you don't need to have a graduate degree in education to recognize this, with all due respect.

Re: Teachers, assessment and FRC
 

This test?

It doesn't look like the MAP test is particularly relevant to robotics, apart from the physics. That's a couple steps removed from mechanical or electrical design, and definitely not a good proxy for "inspiration" (which probably has a lot to do with motivation and a personal sense of empowerment, and not just being conversant in the concepts of science).

Plus, it says the test is for grade 10 and below, and calibrated to state science curriculum standards. Neither do they seem to explain what their "two different scales" and "two scores" are. That doesn't sound too helpful at all.

Aside: Incidentally, any website that puts an "®" character in a URL (except for deliberate obfuscation) strikes me as being fundamentally out of touch with a technical audience....

Re: Teachers, assessment and FRC
 

I hope everyone agrees that you are on FIRST's side. And the side of VEX and Science Fairs and everyone else who gives a darn about making science, technology, engineering and our whole darn world more exciting and engaging for young people.

But education treads an interesting line between a science and an art. There are some things we can do and measure... certain classroom strategies and class sizes and compositions that work... and other things that we do are like music or painting. Is John Williams a better composer than Beethoven? Maybe in 200 years we'll have some perspective on that.

My background is in engineering. I love science. I love facts. I love the scientific method. But my profession is education, and my scientific opinion is that people... especially young people... are weird and wonderful beings that rarely conform to efficient experiments when one tries to measure the things that really matter.

Come up with a test that measures passion and inspiration, and your Ph.D awaits.

Jason

P.S. Williams>Beethoven, but my opinion is currently swayed by having watched Star Wars on Blu Ray twice tonight.

Re: Teachers, assessment and FRC
 

Sorry, no advanced degrees for that. Its been done.

Key search terms: “measures passion”

3,144 Results for Primo Central sorted by:relevance
Show only: Cited Articles(79) Peer-reviewed Journals(2,141)

(examples)

The Role of Passion for Teaching in Intrapersonal and Interpersonal Outcomes
Carbonneau, Noémie ; Vallerand, Robert J ; Fernet, Claude ; Guay, Frédéric
Journal of Educational Psychology, 2008, Vol. 100(4), p.977-987 [Peer Reviewed Journal]
less adaptive outcomes (e.g., shame and negative affect). In this study, 494 teachers completed measures of passion for teaching and various

Grit: perseverance and passion for long-term goals
Duckworth, Angela L ; Peterson, Christopher ; Matthews, Michael D ; Kelly, Dennis R
Journal of personality and social psychology, Jun, 2007, Vol.92(6), p.1087-101 [Peer Reviewed Journal]


Key search terms: “measures inspiration”

5,912 Results for Primo Central sorted by:relevance
Show only: Cited Articles(236) Peer-reviewed Journals(4,040)

(examples)

Inspiration as a Psychological Construct
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2003, Vol. 84(4), p.871-889 [Peer Reviewed Journal]
study of inspiration, both as a general construct and in specific content domains (e.g., religion, creativity, interpersonal relations). Inspiration is an experience with which we are all familiar.

Inspiration
Thrash, Todd M ; Elliot, Andrew J
Journal of Personality and Social Psychology, 2004, Vol. 87(6), p.957-973 [Peer Reviewed Journal]

Abstract:
The authors examined the core characteristics, component processes, antecedents, and function of state inspiration. In Studies 1 and 2, inspiration was contrasted with baseline experience and activated positive affect (PA) using a vivid recall methodology. Results supported the tripartite conceptualization of inspiration. Confirmatory factor analysis indicated that inspiration may be decomposed into separate processes related to being inspired "by" and being inspired "to." Study 3 found that daily inspiration is triggered by illumination among individuals high in receptive engagement, whereas activated PA is triggered by reward salience among individuals high in approach temperament. Approach temperament was also implicated in being inspired "to." Inspiration and activated PA appear to serve different functions: transmission and acquisition, respectively. (PsycINFO Database Record (c) 2011 APA, all rights reserved) (from the journal abstract)

Re: Teachers, assessment and FRC
 

<chuckles>

That's the beauty of educational research, though... you can always explain why the "other guy/gal" did it wrong.

Jason

Re: Teachers, assessment and FRC
 

Something to think about for the action research - Is there a relationship between participation in FIRST and students taking "higher level" science and math courses? Look at the grades earned by students in and out of FIRST. If you have AP or ACT subtest scores available - look at the scores from students who do and do not participate in FIRST. Action research doesn't require you to be a psychometrician, but you'll likely find some interesting trends.

Constructivism is messy and loud and doesn't flow in a linear fashion. But it works.

Julia - NBCT EA Science

Re: Teachers, assessment and FRC
 

I'm sure you've already looked at it, but why not use the information FIRST provides? There are direct links between college preparation and acceptance, conducted by independent firms, that show FIRST helps with STEM-related careers.

Re: Teachers, assessment and FRC
 

Learning is not the primary goal of the program, but it's a really nice side-effect.