Teachers, assessment and FRC

[KHall]

This is a big hello to all the experienced teachers out there. I’m taking my final science methods class before student teaching this spring. Passed the Praxis II Physics test last Nov.

My current methods class requires a short research paper. I wanted to do something about robotics as inquiry learning, but found that general research on the subject was somewhat thin. There are a couple of excellent papers by Dr. Sullivan, Department of Teacher Education and Curriculum Studies, University of Massachusetts, Amherst on her web site: http://people.umass.edu/florence/papers.htm She really nails the connections between robotics and the AAAS & NRC science standards in her 2007 paper and discusses some rather practical aspects of inquiry based learning techniques in robotics in her 2009 paper. Good stuff.

But my professor was unimpressed. He wanted to know if FIRST had ever done a NWEA “growth in science literacy test”. He described it as a pre- and post- test that measures the growth of science literacy. I’ve found their web site, http://www.nwea.org but honestly it does not mean a lot to me. I did get this much, apparently its not a test that can be administered by just anyone, you have to go to one of their sites to take the test. That and they have a database to compare the results of a given student to students with similar demographics.

Its too late to save my research paper, but I would still like to know if anyone is doing this kind of assessment, and if we have some sort of measure acceptable to the Ivory Tower folks that FIRST does improve science literacy skills? If so, please let me know. Not out to prove anything to anyone, just want to learn more about these things from those of you who know.

Thank-you,
Keith Hall

[Phyrxes]

I can't say I've ever heard of the nwea.org test. The only assessments that I can think of that I have administered to check for prior knowledge, literacy, or whatever you want to call prior learning have been either the Mechanics Baseline Test or the Force Concept Inventory and that when was when I was teaching a second year mechanics class that we have since phased out in favor of AP - B and or C.

I'm not even sure how to quantify what a student learns in FIRST in the sense of a curriculum, especially if you are not limiting it to just FRC.

[Phyrxes]

I completely forgot about something that might help you: http://modeling.asu.edu/

While not FIRST related it can provide you some detailed information about inquiry, specifically physical science.

[KHall]

Well, what you say supports my professor's position. He said it nicely but the gist was 'until you can measure meaningful progress, you're just a bunch of people who play with robots'.

That's probably the Ivory Tower way of looking at FIRST. He did admit that project-based and open-ended, inquiry based methods have been proven to work. And that FIRST makes those approaches possible, but -- we've never directly measured and documented that FIRST’s programs are proven to work in a scientifically valid study. (That I could find!)

The NWEA test he suggested may be one possible way to establish that connection. My professor is not belittling FIRST. In fact, I believe he wants someone to do this so that FIRST can gain more credibility in his circles. Remember it’s the Ivory Tower folks who are writing the next round of standards, making methods recommendation, etc. I think he really wants FIRST to be a scientifically proven program so he can use some or all of it.

We’ve seen major progress towards the dream this year. The state of Minnesota officially recognized FIRST FRC as a sport. Wil i am has done us a tremendous favor by spreading the word. Great things are happening. If we are going to change the culture we need valid proof that what we are doing is more than just playing with robots. If we had proof that FIRST programs (for example) increase science literacy skills 30% faster or higher than the average student, we’d have another gem in our toolboxes.

But as I mentioned before, when I went to look for the research that verifies that FIRST is proven to work, I found nothing. So I propose we beg, badger or cajole someone like Dr. Sullivan (a known robotics in education fan and respected researcher) to help us do exactly that. NWEA might be the answer or at least a starting point that could force the Ivory Tower to take notice of FIRST. We need some quantifiable scientific proof that our programs result in learning.

How do we get that ball rolling?

Keith Hall

[sanddrag]

I'll agree that this is an area that FIRST HQ has not been incredibly focused on over the years, and would probably benefit by having more substantial quantifiable data to show exactly what we are doing to these students.

[ebarker]

Originally Posted by KHall

'until you can measure meaningful progress, you're just a bunch of people who play with robots'.

It is generally true that if you cannot measure what you are doing, you cannot manage what you are doing.

The purpose, mission, and vision of FIRST isn't to teach or improve scientific literacy, but to inspire. And as any teacher will tell you, an inspired student is more desirable than an resistant student. There are a million stories about how a student growing up was inhibited from STEM activities because the teachers, the community, and their peers viewed it as a negative activity. FIRST seeks to correct that. And yes, some of my strongest NEGATIVE memories as a child regarding STEM came from the teachers.

I am suggesting that your professor is measuring the INCORRECT thing !!!!

For whatever reason, I would suggest that your professor fails to recognize the linkage between an positive viewpoint toward STEM versus a negative viewpoint. It is that simple.

Just an hour ago I received an email from "Education Week" with this quote in an article: "The problem, then and now, is that the architects of futurist school reform are too invested in the status quo to tolerate, let alone foster, much subversion, criticism, or free thinking." Education Week

All too often the most resistance to change are those inside the school system.

Hundreds of thousands of students, 85,000+ volunteers and mentors, over 3,500 institutions and corporations outside the ivory tower that contribute $ 35M+ in direct costs and probably 3 times that in other costs, $ 14M in scholarships disagree with your professors assertion. This money is freely given and NOT taxes.

There are other perspectives and opinions about the value of FIRST and you don't need to have a graduate degree in education to recognize this, with all due respect.

[Tristan Lall]

This test?

It doesn't look like the MAP test is particularly relevant to robotics, apart from the physics. That's a couple steps removed from mechanical or electrical design, and definitely not a good proxy for "inspiration" (which probably has a lot to do with motivation and a personal sense of empowerment, and not just being conversant in the concepts of science).

Plus, it says the test is for grade 10 and below, and calibrated to state science curriculum standards. Neither do they seem to explain what their "two different scales" and "two scores" are. That doesn't sound too helpful at all.

Aside: Incidentally, any website that puts an "®" character in a URL (except for deliberate obfuscation) strikes me as being fundamentally out of touch with a technical audience....

[Joe Ross]

Have you read the Brandeis study? http://usfirst.org/aboutus/impact

[KHall]

Yes, thank-you I've seen that study. As near as I can tell, its an attitude assessment, not a learning assessment. What people say they prefer and what you can measure about what they have learned are different things.

[Renee Becker-Blau]

I think this is an interesting thread, have you tried talking to your senior mentor in the area or FIRST headquarters about pursuing this idea?

[dtengineering]

I get what your prof is saying... FIRST and FRC participants, and VEX and VRC participants, and BotBall and BotBall participants, and High School Drag racing participants and Skills Canada participants and all sorts of programs would probably do just fine on standardized pre and post skills inventory tests and sure... it would be interesting to someone, somewhere, to measure that.

But I also get that what your prof is really saying is "We don't have a way to measure passion. We don't have a way to measure inspiration. We can't figure out what the long term impacts are because that will take at least 20 or 30 years and your paper is due Thursday. So let's measure this thing that I think we can measure, regardless of how relevant it is."

And I get that... I really do. I've jumped through the hoops, cited the studies and (amongst some other, more meaningful stuff) earned a couple of education degrees. It's not bad to say we can't measure passion... but we need to be honest about it... and not try and pass off some standardized test as an acceptable proxy.

I think most FRC coaches, mentors, volunteers and sponsors will agree with me when I say that we don't give up our evenings, weekends and (in many cases) a large part of our weekdays so that our students will score three percent higher on their next physics test. Yeah, it's great if they do, but taking away their TV and cramming facts into their head from a textbook will do that just fine, too.

We do it because we see that girl on the build team discover that she's not the only girl who likes to build things. That she's actually pretty normal to want to learn to weld. We do it to see the kid who's maybe not great academically, but awesome mechanically, discover that his talents require just as much smarts as those kids who cruise through math class without even trying. We do it to see that quiet grade 8 kid grow up to be the team captain four years later. We do it to share our passion.

And the only thing that can really measure that is when you look a kid in the eye, say "We've got a problem..." and you see them smiling, "Bring it on."

Good luck with the research... don't let the bastards grind you down!

Jason

[Ian Curtis]

To add to Jason's point, I think FIRST adds a lot to the intangible skill set of future engineers and scientists. FIRST gives you opportunities to make budgets, presentations, and persuade people of your opinion long in advance of your peers.

FIRST's successes won't be measured in terms of % on tomorrow's test, it will be measured by the leaps and bounds of progress wrought by FIRST's alumni.

It's no scientific study, but I've found a pretty impressive correlation in my own college experiences between competent people you can trust to get things done and kids who did FIRST.

Best of luck getting the actual data you need!

[JesseK]

The NWEA test sounds like extra fluff to add to the cost of FRC, IMO. The test simply misses the point: FRC isn't about increasing science literacy. Thus, I have to agree with the professor about your paper.

FRC is about increasing science interest with respect to other careers that do not involve large-scale problem solving. When have official FIRST mediums said otherwise?

IMO, the real measurable 'thing' isn't all that glamorous: we'll know FRC works when the 'professional' unemployment rate for technically-skill jobs isn't at a labor shortfall rate (about 3% unemployment) in a good economy. Currently it averages 4.4% (Table 625, 'Professional and related occupations').

Let me caveat further this with an opinion: this stat doesn't mean that the other jobs are less necessary -- it simply means that people in the U.S. are not following the trend of U.S. jobs to the 'more technical' industries and programs like FIRST are necessary. It means that people are not setting themselves up (via education) to succeed in whatever future industries/problems/situations arise. Many adults are returning to school, which is great. Yet the fact that an increasing % of young students are not moving into educational programs geared for these industries will simply cause more problems in the future.

[Jon Stratis]

FIRST. For Inspiration and Recognition of Science and Technology.

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

That says nothing about teaching them science literacy. Sure, you'll grow that as part of everything else, but the main point of FIRST is to get kids excited about science and technology. To get them passionate. To show them that there are role models other than the sports players, actors, and musicians that get all of our TV time. It's not about teaching kids physics and math. It's about guiding them towards wanting to learn those things on their own.

I'm sure everyone says this about their team, but I truly believe it's nowhere more apparent than on a team like the one I work with - an all girls team. With this team, you really see the growth of every student. They start out with practically no knowledge - It's not that they haven't taken physics or calculus yet, it's that they don't know basic concepts or skills like how to operate a drill. Without this program, they could go through school and might enjoy math or physics, but be completely unaware of the potential life paths those classes open up to them.

My High School didn't have FIRST. In fact, still don't. Looking back on it, I know it wasn't anything I did in High School that inspired me to become an engineer. If you were to go simply by what I did in High School, I'd be off somewhere trying to scrape up minimum wage playing the Saxophone. It was a single teacher back in 5th grade that set my path by offering an after school programming class. That's it - an after school activity that inspired me to become what I am today. That activity didn't increase my science literacy... but it inspired me to keep looking into programming on my own over the following 7 years, and eventually go to college for it. That's what FIRST is all about. I just wish I had it back when I was in school.

[KHall]

Thank-you for your comments.

Lets look at it this way. We at FIRST (specifically the engineers and teachers) claim/feel that what we are doing makes a difference. Testimonials are great, but they can only have so much credibility. I’ve read testimonials from people who claim to have been abducted by aliens and flown to Venus.

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

What we don’t have here is someone with a Ph.D. and a reputation that will get the others in the Ivory Towers to peer review the research and get it published in a respectable journal. That is the key piece. I’ve suggested Dr. Sullivan as a potential person of interest, but I’m sure there are plenty more. Who are they? Can we make a list and start having the FIRST brass approach these people?

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall