Teachers, assessment and FRC
 

This is a big hello to all the experienced teachers out there. I’m taking my final science methods class before student teaching this spring. Passed the Praxis II Physics test last Nov.

My current methods class requires a short research paper. I wanted to do something about robotics as inquiry learning, but found that general research on the subject was somewhat thin. There are a couple of excellent papers by Dr. Sullivan, Department of Teacher Education and Curriculum Studies, University of Massachusetts, Amherst on her web site: http://people.umass.edu/florence/papers.htm She really nails the connections between robotics and the AAAS & NRC science standards in her 2007 paper and discusses some rather practical aspects of inquiry based learning techniques in robotics in her 2009 paper. Good stuff.

But my professor was unimpressed. He wanted to know if FIRST had ever done a NWEA “growth in science literacy test”. He described it as a pre- and post- test that measures the growth of science literacy. I’ve found their web site, http://www.nwea.org but honestly it does not mean a lot to me. I did get this much, apparently its not a test that can be administered by just anyone, you have to go to one of their sites to take the test. That and they have a database to compare the results of a given student to students with similar demographics.

Its too late to save my research paper, but I would still like to know if anyone is doing this kind of assessment, and if we have some sort of measure acceptable to the Ivory Tower folks that FIRST does improve science literacy skills? If so, please let me know. Not out to prove anything to anyone, just want to learn more about these things from those of you who know.

Thank-you,
Keith Hall

Re: Teachers, assessment and FRC
 

I can't say I've ever heard of the nwea.org test. The only assessments that I can think of that I have administered to check for prior knowledge, literacy, or whatever you want to call prior learning have been either the Mechanics Baseline Test or the Force Concept Inventory and that when was when I was teaching a second year mechanics class that we have since phased out in favor of AP - B and or C.

I'm not even sure how to quantify what a student learns in FIRST in the sense of a curriculum, especially if you are not limiting it to just FRC.

Re: Teachers, assessment and FRC
 

I completely forgot about something that might help you: http://modeling.asu.edu/

While not FIRST related it can provide you some detailed information about inquiry, specifically physical science.

Re: Teachers, assessment and FRC
 

Well, what you say supports my professor's position. He said it nicely but the gist was 'until you can measure meaningful progress, you're just a bunch of people who play with robots'.

That's probably the Ivory Tower way of looking at FIRST. He did admit that project-based and open-ended, inquiry based methods have been proven to work. And that FIRST makes those approaches possible, but -- we've never directly measured and documented that FIRST’s programs are proven to work in a scientifically valid study. (That I could find!)

The NWEA test he suggested may be one possible way to establish that connection. My professor is not belittling FIRST. In fact, I believe he wants someone to do this so that FIRST can gain more credibility in his circles. Remember it’s the Ivory Tower folks who are writing the next round of standards, making methods recommendation, etc. I think he really wants FIRST to be a scientifically proven program so he can use some or all of it.

We’ve seen major progress towards the dream this year. The state of Minnesota officially recognized FIRST FRC as a sport. Wil i am has done us a tremendous favor by spreading the word. Great things are happening. If we are going to change the culture we need valid proof that what we are doing is more than just playing with robots. If we had proof that FIRST programs (for example) increase science literacy skills 30% faster or higher than the average student, we’d have another gem in our toolboxes.

But as I mentioned before, when I went to look for the research that verifies that FIRST is proven to work, I found nothing. So I propose we beg, badger or cajole someone like Dr. Sullivan (a known robotics in education fan and respected researcher) to help us do exactly that. NWEA might be the answer or at least a starting point that could force the Ivory Tower to take notice of FIRST. We need some quantifiable scientific proof that our programs result in learning.

How do we get that ball rolling?

Keith Hall

Re: Teachers, assessment and FRC
 

I'll agree that this is an area that FIRST HQ has not been incredibly focused on over the years, and would probably benefit by having more substantial quantifiable data to show exactly what we are doing to these students.

Re: Teachers, assessment and FRC
 

Have you read the Brandeis study? http://usfirst.org/aboutus/impact

Re: Teachers, assessment and FRC
 

Yes, thank-you I've seen that study. As near as I can tell, its an attitude assessment, not a learning assessment. What people say they prefer and what you can measure about what they have learned are different things.

Re: Teachers, assessment and FRC
 

I think this is an interesting thread, have you tried talking to your senior mentor in the area or FIRST headquarters about pursuing this idea?

Re: Teachers, assessment and FRC
 

I get what your prof is saying... FIRST and FRC participants, and VEX and VRC participants, and BotBall and BotBall participants, and High School Drag racing participants and Skills Canada participants and all sorts of programs would probably do just fine on standardized pre and post skills inventory tests and sure... it would be interesting to someone, somewhere, to measure that.

But I also get that what your prof is really saying is "We don't have a way to measure passion. We don't have a way to measure inspiration. We can't figure out what the long term impacts are because that will take at least 20 or 30 years and your paper is due Thursday. So let's measure this thing that I think we can measure, regardless of how relevant it is."

And I get that... I really do. I've jumped through the hoops, cited the studies and (amongst some other, more meaningful stuff) earned a couple of education degrees. It's not bad to say we can't measure passion... but we need to be honest about it... and not try and pass off some standardized test as an acceptable proxy.

I think most FRC coaches, mentors, volunteers and sponsors will agree with me when I say that we don't give up our evenings, weekends and (in many cases) a large part of our weekdays so that our students will score three percent higher on their next physics test. Yeah, it's great if they do, but taking away their TV and cramming facts into their head from a textbook will do that just fine, too.

We do it because we see that girl on the build team discover that she's not the only girl who likes to build things. That she's actually pretty normal to want to learn to weld. We do it to see the kid who's maybe not great academically, but awesome mechanically, discover that his talents require just as much smarts as those kids who cruise through math class without even trying. We do it to see that quiet grade 8 kid grow up to be the team captain four years later. We do it to share our passion.

And the only thing that can really measure that is when you look a kid in the eye, say "We've got a problem..." and you see them smiling, "Bring it on."

Good luck with the research... don't let the bastards grind you down!

Jason

Re: Teachers, assessment and FRC
 

To add to Jason's point, I think FIRST adds a lot to the intangible skill set of future engineers and scientists. FIRST gives you opportunities to make budgets, presentations, and persuade people of your opinion long in advance of your peers.

FIRST's successes won't be measured in terms of % on tomorrow's test, it will be measured by the leaps and bounds of progress wrought by FIRST's alumni.

It's no scientific study, but I've found a pretty impressive correlation in my own college experiences between competent people you can trust to get things done and kids who did FIRST.

Best of luck getting the actual data you need! :cool:

Re: Teachers, assessment and FRC
 

The NWEA test sounds like extra fluff to add to the cost of FRC, IMO. The test simply misses the point: FRC isn't about increasing science literacy. Thus, I have to agree with the professor about your paper.

FRC is about increasing science interest with respect to other careers that do not involve large-scale problem solving. When have official FIRST mediums said otherwise?

IMO, the real measurable 'thing' isn't all that glamorous: we'll know FRC works when the 'professional' unemployment rate for technically-skill jobs isn't at a labor shortfall rate (about 3% unemployment) in a good economy. Currently it averages 4.4% (Table 625, 'Professional and related occupations').

Let me caveat further this with an opinion: this stat doesn't mean that the other jobs are less necessary -- it simply means that people in the U.S. are not following the trend of U.S. jobs to the 'more technical' industries and programs like FIRST are necessary. It means that people are not setting themselves up (via education) to succeed in whatever future industries/problems/situations arise. Many adults are returning to school, which is great. Yet the fact that an increasing % of young students are not moving into educational programs geared for these industries will simply cause more problems in the future.

Re: Teachers, assessment and FRC
 

Thank-you for your comments.

Lets look at it this way. We at FIRST (specifically the engineers and teachers) claim/feel that what we are doing makes a difference. Testimonials are great, but they can only have so much credibility. I’ve read testimonials from people who claim to have been abducted by aliens and flown to Venus.

So here is the skinny: If you believe something to be true, prove it. Create a hypothesis, design an experiment to test that hypothesis, collect data and analyze the results to see if it supports or denies your hypothesis. That is the essence of science. Everything else is arm waving.

The next step would be to have the results peer reviewed and published in a reputable journal. This is where I believe we fall short. We can understand the basics of designing a non-biased test that will yield statistically significant results. We can separate and measure quantitative and qualitative variables. Psychologists measures attitudes and feeling such as passion and initiative all the time, its been done, published and accepted. Teachers measure learning with much the same results.

What we don’t have here is someone with a Ph.D. and a reputation that will get the others in the Ivory Towers to peer review the research and get it published in a respectable journal. That is the key piece. I’ve suggested Dr. Sullivan as a potential person of interest, but I’m sure there are plenty more. Who are they? Can we make a list and start having the FIRST brass approach these people?

Please understand that I’m on FIRST’s side here. All the claims in this thread are valid in my opinion, I’ve seen FIRST work miracles too. But I don’t have the credentials that matter, so nobody really cares what I think.

How do we find someone like Dr. Sullivan and get that person motivated to do a study about just how FIRST impacts learning?

KHall

Re: Teachers, assessment and FRC
 

FIRST. For Inspiration and Recognition of Science and Technology.

That says nothing about teaching them science literacy. Sure, you'll grow that as part of everything else, but the main point of FIRST is to get kids excited about science and technology. To get them passionate. To show them that there are role models other than the sports players, actors, and musicians that get all of our TV time. It's not about teaching kids physics and math. It's about guiding them towards wanting to learn those things on their own.

I'm sure everyone says this about their team, but I truly believe it's nowhere more apparent than on a team like the one I work with - an all girls team. With this team, you really see the growth of every student. They start out with practically no knowledge - It's not that they haven't taken physics or calculus yet, it's that they don't know basic concepts or skills like how to operate a drill. Without this program, they could go through school and might enjoy math or physics, but be completely unaware of the potential life paths those classes open up to them.

My High School didn't have FIRST. In fact, still don't. Looking back on it, I know it wasn't anything I did in High School that inspired me to become an engineer. If you were to go simply by what I did in High School, I'd be off somewhere trying to scrape up minimum wage playing the Saxophone. It was a single teacher back in 5th grade that set my path by offering an after school programming class. That's it - an after school activity that inspired me to become what I am today. That activity didn't increase my science literacy... but it inspired me to keep looking into programming on my own over the following 7 years, and eventually go to college for it. That's what FIRST is all about. I just wish I had it back when I was in school.

Re: Teachers, assessment and FRC
 

How does the Brandeis study fall short?

Why do we care how FIRST impacts learning? It's not shop class. The goal is always stated using the term "inspire", not the term "teach". Check out the FIRST mission statement:

Our mission is to inspire young people to be science and technology leaders, by engaging them in exciting mentor-based programs that build science, engineering and technology skills, that inspire innovation, and that foster well-rounded life capabilities including self-confidence, communication, and leadership.

Re: Teachers, assessment and FRC
 

Thank-you Alan, you’re absolutely right. Our mission is to inspire. And if we accept that, then any discussion about learning is moot.

If I can bring Dr. Sullivan’s paper to your attention, and/or ask you to examine the AAAS and NRC standards I believe you’ll discover that FIRST is currently covering most of these standard’s goals. I’m just not sure that too many people realize how much FIRST is achieving.

From Dr. Sullivan’s 2007 research paper regarding AAAS Science standards:
“Robotics learning is strongly linked to these three goals of science literacy. First, robotics study requires utilization of four of the six thinking skills characteristic of scientifically literate people—namely, computation, estimation, manipulation, and observation. Second, students of robotics are engaged in science inquiry through both technological design and computer programming activities. Third, robotics teaches students about systems, one of the common themes in science education.”

So if you’re already achieving something, why not take credit for it?

Plus why would a mission statement prevent you from wanting to show your value to the educational community? Didn’t Dean say he wanted an FRC team in every high school? If you were a superintendent of a school system, wouldn’t you find verified results that align with accepted educational standards carry a lot of weight? I personally believe there are many such people out there who want to do what they know is best for their students. We need to give them what they need -- make it easier for them to adopt FIRST.

If nobody here sees the value of this, then I’ll drop it. I believe that we need to find ways to push more proven constructivist methods into the classrooms, and frankly, FIRST is low hanging fruit. The main idea here was assessment, and true to CD, we’ve drifted a bit.

FIRST is not considered a professionally administered education program because we do not currently perform formal assessments. Teachers are experts at assessment, and I am not. I simply asked if anyone was aware of the NWEA testing suggested by a professor who was trying to be helpful because he wanted to find a way to push FIRST (or at least its project oriented methods) into more schools. I still see no harm in asking about that.

KHall