A report on the judging process used for the 2022 EWCP Scholarship for Cultural Impact:
Background
The EWCP Scholarship for Cultural Impact is a small FIRST scholarship established in 2020 by EWCP, Inc, a nonprofit with the mission of strengthening the diversity, robustness, and impact of youth STEM education through strategic advocacy, expert consensus-building, and technical leadership.
As part of that mission, and to provide a resource for similar community scholarships, we wish to publish some notes on the judging process used to determine the 2022 award winner. We recommend this judging strategy for scholarship programs of similar size.
Anonymization phase
After the essay submission deadline, a group of anonymizers removed identifying information including names, locations, institutions, and specific youth STEM competitions from all essays. Essays were assigned a unique identification number before being sent to the judges to ensure we removed as much potential bias as possible.
Round 1 judging
Essays went through two rounds of judging. The first round of judging saw every essay read by two judges and evaluated using this rubric:
Use the following convention for each criteria:
0 - Does not satisfy this requirement
1 - Somewhat satisfies requirement
2 - Meets requirement
3 - exceeds requirement
Criteria:
Effort: Student has clearly put effort into their essay
Plan - Clearly Communicated: Essay explains the mechanism by which the plan provides a solution to the problem statement
Plan - Demonstrates Thoughtful Understanding: Essay demonstrates empathetic thinking and thoughtful understanding of the problem statement, problem statement is relevant
Plan - Achievable/Actionable: Essay demonstrates that plan is achievable and has actionable items
Plan - Scale: This plan has a significant impact as measured by population size, duration of effect, and/or magnitude of effect
Connected/tied back to FIRST Experience: Essay links to ideas and concepts from FIRST experience
Originality: Essay and plan demonstrate original thinking
Round 1 scores were computed for each essay by adding the scores for each rubric category given by both judges who reviewed that essay.
Downselection for round 2
The essays were sorted by total score. Scholarship administrators selected a position in the ordered list where there was a conspicuously large gap in score between neighboring essays. In the case of the 2022 scholarship, there was a large gap between the 3rd and 4th essays in the sorted list. The 3 top essays were then sent forward to a second round of judging.
Round 2 judging
Next, every judge was asked to read all top essays and identify their 1st, 2nd, and 3rd place choices. A weight was assigned to each of these three ranks, and the overall winner was the essay with the highest weighted sum.
Deanonymization and winner notification
Having identified the winning essay, the scholarship administrators asked the anonymization team to reveal the author.