Learning by Making Rockets & Robots

[Ian Curtis]

Originally Posted by Siri

I feel like this is one of the big misunderstandings with these types of tests (sorry, I don't meant to pick on you). It's not ~95% = A = 5, ~85% = B = 4. Yes, A =~ 5 and B =~ 4, but the percentages are (in the case of APs) based on the grades of students in college comparability studies. Depending on the test, your composite score could be 66% and still get a 5.

This is not to say that the scoring setting is necessarily done correctly in all standardized tests.

Perhaps I am not smart enough to understand the College Board, but it seems like if you only need to get 50% of the test right to be as good as a college student getting an A you are losing a lot of resolution you could be using to separate students.

To expand on this a bit, I always tell my students about how the company I work for hires. We're a fairly large medical device company... but we contrary to popular assumption, we don't hire a lot of biomedical engineers. It's easier to hire a good mechanical, electrical, or computer engineer and teach him/her the biology they need for the job, than it is to teach a biomedical engineer the in-depth knowledge of a particular engineering discipline they need.

Absolutely. It seems to me that in most cases what you learn in school really just lays a broad base anyways. For example the project I spent 3 months working on when I was an intern got covered by the professor for about 2 minutes in my flight mechanics class.

[Siri]

Originally Posted by Ian Curtis

Perhaps I am not smart enough to understand the College Board, but it seems like if you only need to get 50% of the test right to be as good as a college student getting an A you are losing a lot of resolution you could be using to separate students

I won't speak for College Board, but I found a good description of their score setting process here. Perhaps it's because resolution can still be useful, even if it's not fed directly into the score?

[Chris Hibner]

Originally Posted by Ian Curtis

Perhaps I am not smart enough to understand the College Board, but it seems like if you only need to get 50% of the test right to be as good as a college student getting an A you are losing a lot of resolution you could be using to separate students.

This is a misconception a lot of people have about testing.

If a large percentage of the population is getting 90% and multiple 100% socres, then THAT is where you are losing a lot of resolution. Saturation of data (in this case, 100% scores) means you're really not separating those at the top.

Many people believe the ideal test of subject knowledge should have the median score around 50%. Practically no one should get 100%. There should be a wide range of difficulty to the questions, from fairly easy to extremely difficult. By constructing a test this way, they hope to achieve the "characteristic of discrimination" in the test - which means the results should be able to separate the test takers along the entire spectrum of knowledge and understanding - not have everyone lumped into a tight band.

[Andrew Schreiber]

Originally Posted by Chris Hibner

This is a misconception a lot of people have about testing.

If a large percentage of the population is getting 90% and multiple 100% socres, then THAT is where you are losing a lot of resolution. Saturation of data (in this case, 100% scores) means you're really not separating those at the top.

Many people believe the ideal test of subject knowledge should have the median score around 50%. Practically no one should get 100%. There should be a wide range of difficulty to the questions, from fairly easy to extremely difficult. By constructing a test this way, they hope to achieve the "characteristic of discrimination" in the test - which means the results should be able to separate the test takers along the entire spectrum of knowledge and understanding - not have everyone lumped into a tight band.

So, what you are saying is that tests should not be used to evaluate absolute understanding but understanding relative to a peer group?

[Jon Stratis]

I would say that depends on the point of a test. A test designed to evaluate how well someone knows a subject (aka most of the tests we take in school) should be designed to evaluate absolute understanding. One used to evaluate the understanding of the population as a whole in order to better direct resources to problem areas should be designed to evaluate understanding compared to a peer group.

Honestly, I've always wondered how much I really got out of my Honors Calc 3 class freshman year of college (trust me, after that I didn't take any more honors classes!). I just don't feel that a 65% in the class, considered an A after the professor curved the grades, really indicates a deep understanding of the subject matter. Of course, the professor couldn't just fail the entire class. That would be as good as admitting that he doesn't know how to teach the subject. Before that class, I had never scored less than 95% in any math class, going all the way back to 1st grade!

[Nemo]

Originally Posted by Andrew Schreiber

So, what you are saying is that tests should not be used to evaluate absolute understanding but understanding relative to a peer group?

Google norm referenced vs. criterion referenced and you can find when each type would be more useful.

Tests that evaluate relative to peers are useful if you're simply looking for the best people you can bring to your university or company.

[Siri]

Originally Posted by Jon Stratis

Honestly, I've always wondered how much I really got out of my Honors Calc 3 class freshman year of college (trust me, after that I didn't take any more honors classes!). I just don't feel that a 65% in the class, considered an A after the professor curved the grades, really indicates a deep understanding of the subject matter. Of course, the professor couldn't just fail the entire class. That would be as good as admitting that he doesn't know how to teach the subject. Before that class, I had never scored less than 95% in any math class, going all the way back to 1st grade!

Obviously I wasn't in your class, but I've had professors that deliberately treat tests the way Chris mentioned above. I once had a prof hand me a 300/400-level* chemistry exam (this was in CHEM 102) and say 'just do it'. I got an A with something like 33%. But it truly (trust me) challenged us, and the thought we took with it manifested in our answers clearly, even if we didn't understand most of the content. Different people "lightbulbed" at different places/questions, and it really showed each of us what was ahead and showed him just how deeply each student understood the knowledge we did have.

Even for heavily curved classes in which my professor didn't indicate intended acceptable-knowledge test score, I have to feel somewhat confident that my A meant something since I definitely understood the knowledge therein well enough to apply it in many subsequent courses and projects.

*In this case that means a good chemistry senior should score around 95%.

[Nemo]

Three classes are studying the same subject, but they have different teachers who give different tests.

Average Test Scores:
Class A: 20%
Class B: 50%
Class C: 80%

Question 1: Which group of students knows its subject the best?

Question 2: Which test was the most cleverly designed?

Answer: we don't know. Any combination of answers to the two questions is possible: it all depends on the questions asked on the tests. The class averages tell us very little.

[Chris Hibner]

Originally Posted by Andrew Schreiber

So, what you are saying is that tests should not be used to evaluate absolute understanding but understanding relative to a peer group?

Not at all (at least not necessarily). If the test is to cover something you just learned and the test is only on material you just learned, then perhaps 90% should be a typical score for a good student. If the test is to determine your absolute knowledge of the entire subject, then the average should be closer to 50% for a good student. Let's face it - subjects are very broad and you only get a glimpse of it in your studies.

Here's an example:

Let's say I'm hiring for an advanced development position in signal processing. In order to determine knowledge, we will use a written test.

Let's say everyone that applies gets 90%+ on the test and there are a lot of 100% scores. Do I really believe that there are that many people that know 90% of all signal processing knowledge. There's no way that is true. Time to write a new test.

If you're trying to determine absolute knowledge of a subject, then ideally on that test only the elite PhD's who are at the top of the field should get 100% (if they're really, really good). PhD's who are marginal should probably score 85%, showing they know maybe 85% of the field of knowledge. People with master's degrees you would expect to score in the 65 - 85% range. Good bachelor's degree candidates would be expected to get maybe 50%.

The point is, just because the bachelor's candidate scores 50%, that doesn't make him an idiot or a bad candidate for a lot of jobs. It just means he doesn't have PhD level knowledge, but that's okay - we expect that. But if the bachelor's candidate came in and scored 85%, then you know he may be brilliant. You would have never found that with the test where everyone scores 95% or better.

If the goal is to determine if your knowledge is "good enough", then the test where many candidates score 90% is appropriate. If your goal is to determine the absolute level of knowledge of a subject, then the having everyone score 90% isn't realistic because very few people know 90% about any subject (unless the subject is very small and limited).

[Ian Curtis]

Originally Posted by Chris Hibner

Not at all (at least not necessarily). If the test is to cover something you just learned and the test is only on material you just learned, then perhaps 90% should be a typical score for a good student. If the test is to determine your absolute knowledge of the entire subject, then the average should be closer to 50% for a good student. Let's face it - subjects are very broad and you only get a glimpse of it in your studies.

Here's an example:

Let's say I'm hiring for an advanced development position in signal processing. In order to determine knowledge, we will use a written test.

Let's say everyone that applies gets 90%+ on the test and there are a lot of 100% scores. Do I really believe that there are that many people that know 90% of all signal processing knowledge. There's no way that is true. Time to write a new test.

If you're trying to determine absolute knowledge of a subject, then ideally on that test only the elite PhD's who are at the top of the field should get 100% (if they're really, really good). PhD's who are marginal should probably score 85%, showing they know maybe 85% of the field of knowledge. People with master's degrees you would expect to score in the 65 - 85% range. Good bachelor's degree candidates would be expected to get maybe 50%.

The point is, just because the bachelor's candidate scores 50%, that doesn't make him an idiot or a bad candidate for a lot of jobs. It just means he doesn't have PhD level knowledge, but that's okay - we expect that. But if the bachelor's candidate came in and scored 85%, then you know he may be brilliant. You would have never found that with the test where everyone scores 95% or better.

If the goal is to determine if your knowledge is "good enough", then the test where many candidates score 90% is appropriate. If your goal is to determine the absolute level of knowledge of a subject, then the having everyone score 90% isn't realistic because very few people know 90% about any subject (unless the subject is very small and limited).

So maybe I AM just too dumb to understand the college board!

I totally understand the logic of everyone getting a near 100 makes it difficult to understand who knows the material. The guy who taught my combustion class gave fairly easy tests -- you had to think hard, but if you understood the core concepts you could usually tease out the answer. On the flip side though, if you made a dumb math mistake (and I'm pretty good at those) you might lose 10 points and be near the median score even if you knew all the material. I actually liked really hard tests because even if I got a 50 I usually beat the class average and rode the curve to victory.

I still think though in the particular case of the Chemistry AP if I can get 50% of the test correct and be in the top 15% of the country, they are setting the bar too low or the test is too hard. I think you are absolutely right in the broader sense of giving tests though.

Originally Posted by EricH

... not to let academics get in the way of a good education ...

What does this mean? I know I've seen it used on CD before, but I'm just not sure what the lesson is.

[EricH]

Originally Posted by Ian Curtis

What does this mean? I know I've seen it used on CD before, but I'm just not sure what the lesson is.

I actually learned it from my dad...
I'm going to give a couple of definitions that may help, then explain what it means.

Academics: class work, including homework, projects, tests, quizzes, and anything else that has a grade from school.

Good education: Extracurricular activities (say, along the lines of FIRST, band, you get the idea), academics, family chats about life, the universe, and everything--stuff like that.

Academics can fit into a good education, as noted. However, in some cases, for whatever reason the academics are allowed to essentially take over the entire "good education" part. As an example, figure a straight-A student who wants to join the school's FRC team, but is told that he can't because he'll hurt his grades (and thus his chances for college, profession, etc.) Let's also figure that the same is told for any other extra-curricular. That's a rather extreme example of letting academics get in the way of a good education. Or, to use another, semi-related phrasing, not seeing the forest for the trees. (And, in this particular example, hurting the grades by a point or less each may not do anything to future chances.)

Don't get me wrong--the grades are important. But they aren't the be-all and end-all of going to school. Part of that education is the extracurriculars. Where else are you going to apply that knowledge you just gained in school?

[Astrokid248]

Originally Posted by EricH

Part of that education is the extracurriculars. Where else are you going to apply that knowledge you just gained in school?

Here here. The real problem with tests of any sort is that they don't test real world application. So while they serve a purpose, acing a test isn't the same as designing a nested-axle belt-driven transmission. Realistically, there ought to be both in schools, but so many extracurriculars are being dropped that you'd think only academics matter.