Quote:
Originally Posted by Andrew Schreiber
So, what you are saying is that tests should not be used to evaluate absolute understanding but understanding relative to a peer group?
|
Not at all (at least not necessarily). If the test is to cover something you just learned and the test is only on material you just learned, then perhaps 90% should be a typical score for a good student. If the test is to determine your absolute knowledge of the entire subject, then the average should be closer to 50% for a good student. Let's face it - subjects are very broad and you only get a glimpse of it in your studies.
Here's an example:
Let's say I'm hiring for an advanced development position in signal processing. In order to determine knowledge, we will use a written test.
Let's say everyone that applies gets 90%+ on the test and there are a lot of 100% scores. Do I really believe that there are that many people that know 90% of all signal processing knowledge. There's no way that is true. Time to write a new test.
If you're trying to determine absolute knowledge of a subject, then ideally on that test only the elite PhD's who are at the top of the field should get 100% (if they're really, really good). PhD's who are marginal should probably score 85%, showing they know maybe 85% of the field of knowledge. People with master's degrees you would expect to score in the 65 - 85% range. Good bachelor's degree candidates would be expected to get maybe 50%.
The point is, just because the bachelor's candidate scores 50%, that doesn't make him an idiot or a bad candidate for a lot of jobs. It just means he doesn't have PhD level knowledge, but that's okay - we expect that. But if the bachelor's candidate came in and scored 85%, then you know he may be brilliant. You would have never found that with the test where everyone scores 95% or better.
If the goal is to determine if your knowledge is "good enough", then the test where many candidates score 90% is appropriate. If your goal is to determine the absolute level of knowledge of a subject, then the having everyone score 90% isn't realistic because very few people know 90% about any subject (unless the subject is very small and limited).