This (maybe off topic?) quote that appeared earlier in the thread struck me as odd. Having served a number of years as both an FLL and FRC judge (in a different district), I don’t know whether to be bemused by the absurdity of what’s being suggested as common place here, or to be concerned that it could ever possibly be true. It’s certainly diametrically opposed to the way judging works in any panel of which I’ve been part.
I am trying very hard to stay out of this thread for a multitude of reasons but I will corroborate everything Juan has said about NC and I will also tell you that the folks that I have spoken with at FIRST HQ about it are aware and don’t seem bothered. I expect it to be adopted by other regions.
I’m not going to give any personal stories (though I have many at this point). What Juan has described is the norm at events in NC… that and foam tiles.
3 Likes
I wish I was stretching the truth. I really do. I’ve spent many hours on conference calls, providing documentation, reviewing notes, etc. to wind up exactly where @marshall described:
Judges are required to remain mum about what occurs in the judging room, so this would be hard to sort out without help from higher up. If true, it’s deeply disturbing. How could any self respecting judge acquiesce to this? We are asked and expected to be role models.
2 Likes
By virtue of not being familiar with the alternative option. Bluntly - if the training they receive is that they should be using a flawed process… well, that happens. And if the person doing the training happens to be the one who controls access to that group.
Turns out insular groups can be breeding grounds for abuse.
1 Like
This also feels like it ties into the whole idea of classism and unfairness to low resourced teams that’s part of what’s being discussed in the Does the FIRST "Progression of Programs" as is contribute to systemic classism/hurt its own EDI goals? thread.
Teams with “extra” resources and manpower have the ability to print fancy signs and presentation materials, set up televisions for demos, have an awards presenter always standing in your pit, etc. If you have those flashy things, you get on the judges’ list and they’ll find you an award, even if there are more qualified teams for those awards who don’t have the resources to stand out. A team that wins an award at each of their district competitions gets 10 extra district points going into DCMP, 25 going into CMP. That’s a pretty big boost for something that’s selected so objectively.
As a team we’re lucky to have those resources and we do spend a good amount of effort working to earn awards. I don’t work much with the awards side of the team (other than proofreading the chairmans essay for English errors), so I can’t really speak to how judging works. But I know that we’ve won judged awards at 10 of our 14 total events, and that certainly has to do with the amount of resources we’re able to dedicate towards them.
Also, if we’re adding to a list of flawed judging processes, the fact is that most of the volunteer judges here come from a handful of teams, and it’s pretty widely known that they tend to give their teams awards at most events. I’ve never volunteered with anything relating to judging so I don’t have the inside information to be able to claim malfeasance, and these are good teams so it’s hard to say that they don’t deserve the awards. But looking at it as an outsider, the conflict of interest seems staggering.
1 Like
To be clear, the argument here is
Teams that deserve awards being given them
vs.
Awards being given to teams that earn them
?
Yeah going to need clarification.
Sounds like the claim is - judges making arbitrary lists of teams that aren’t judged based off of awards criteria, but instead likability/impressive in other regards? They are then fit into awards after the fact?
List of teams that “we like” or list of teams that haven’t won an award ever/lately? Not that it matters but I’m just curious about the intent.
I’m guessing that neither of you have been a judge. Maybe you should volunteer so you don’t have to rely on what others tell you.
Disclaimer: I was a judge for an FTC event in 2019. It was my first time as a judge. I can confidently say that I wasn’t the best judge. I can also say that I will do better the next time.
From my single judging experience I can safely say that at the beginning of the judging process, EVERY team had an opportunity to win EVERY award. In no way did I feel that step 1 was to list the teams to give awards to. I also did not feel that any other judge was pushing for a team to get an award OR that any judge was trying to coerce other judges to select any particular team.
If at the start of the judging process I was told that “Here’s a list of teams we’re giving awards to.” I would have quit.
So you’re telling me that in NC, all the judges start with a list of “winners”, then talk to every team to make it seem like every team has a chance, then give awards to the pre-chosen few??? To me it seems like you’re questioning the integrity of every judge in NC.
Ohh man did you walk into that fence post. Juan and I met while judging FLL at the state championship level.
I appreciate all of our judges but the judge selection process used in NC for FRC excludes both Juan and myself.
No, we are questioning a process. Not the people, not their commitment to the program, nor their qualifications.
11 Likes
While having first-hand experience is better than trusting someone else’s experience, it doesn’t necessarily mean that someone else’s experiences are invalid. Especially if you trust them and their motivations.
This is a pretty bold claim, given you weren’t in the room at the time.
The universal experience I’ve had while judging: Judges don’t get to choose the process. It’s 100% placed upon them by the Judge Advisor and higher-level leadership. They execute the process and sign off on the final decisions. If the process is broken, you fix it with the people who define the process, not the ones who execute it.
I’d be careful before demonizing individuals who saw issues, raised them with leadership, but had their concerns go unanswered.
3 Likes
I think this might clarify it a bit:
The Story
Let’s say there are (for social distancing reasons) only 3 teams at an event. Team A and Team B have top notch robots. 20 cycles a match, great accuracy, can play the best defense in the world. Their only flaw? Neither has any autonomous mode. They just sit there for 15 seconds. Both are beautiful robots, and deserve an award.
Team C isn’t not as great of a robot. It can do a 2 cycles in a match, and not particularly well. It’s slow, and the drivers aren’t really trained in defense. But! The students of the team are very proud of their auto, which runs forwards off the line in the beginning of the match (Look at him go!) and puts a game piece in the goal. It took 13 hours, a lot of blaming of electrical and a whole lot of pizza, the programmer finally figure it out.
The Big Question
At this event, there are 2 awards, the robot design award and the autonomous award.
Which is the appropriate way for judges to give awards?
-
The judges look at all the teams, and see who has the best robots. They see that Team A and Team B are clearly the best teams, so they mark them down for awards. The judges then go back to there room, figure out which team deserves robot design (Team B) and gives the remaining team (Team A) the auto award.
-
The judges look at all the teams, and try to decide who best fits the robot design award. They settle an Team B. They then look for who has the best auto, which is Team C, even though the rest of their performance is toaster bot quality at best.
2 Likes
I just want to chime in that my experience has been quite different; teams I’ve been on have very frequently won judged (robot) awards with minimal supplemental materials. Usually when we win a judged robot award it’s because:
a. Based on the judges initial questions/reactions, the students successfully deduce what award they’re specifically looking for
b. They tailor the rest of the pitch to specifically target the criteria for that award as per it’s description in the award manual.
Obviously this is hard to tell with 100% certainty, as I don’t recognize every judge, but I want to say that I’ve seen a very healthy mix of judges that are team-affiliated and judges that are a sponsor or VIP that wanted to be a bit more involved in the event (this is for pit speaking about robot awards). I have noticed that usually the event has some long-time judges that are team affiliated, but my speculation is that they typically judge the more “consequential” (i.e. culture changing) awards.
This isn’t to discount your experience but highlight that there seems to be significant variation in how judging is handled between regions.
I’ve only judged FTC, but the process went differently in our regard (it’s been a while, so my memory may be a bit shaky):
a. Listen to all the teams present their engineering notebook
b. List the top x teams for each award
c. “De-conflict” all the lists, ensuring that each award had a different winner while maintaining that it was an award they actually deserved to win.
2 Likes
This is the experience I have had in the judges room as well. It does not seem to be the same one that is being discussed in this thread. Why are their different processes?
This is close to my experience with FLL judging. There are lists of teams that should be up for an award. However, the lists are created independently by groups of judges from their experiences after working with teams at a particular judging station. Judges never share these lists until the end of the day, during consensus meetings.
The tone of consensus is always to come in with lots of options, so that if someone happens to feel strongly about one particular award/team, other judges can easily pick another option to accommodate.
This has 100% been the case for several awards at competitions I’ve been at, where several awards have been handed to teams as a “runner-up” award for the award that they were aiming for, or just because the team was well-known, when there were teams who definitely deserved the awards more
1 Like
More anecdotes, but it feels to me that all of our technical awards have come from kids in the pit talking about the robot, where our non-technical awards have been greatly assisted by fancy marketing materials.
I’m not in NC, but this passes a gut feel check from me. We’ve got a few awards where I’m totally unsure why we won them, other than “general impressiveness” where some judges figured that we probably should have won something.
My understanding of what happens (And it is compatible with my experience judging FLL) I think it is much at Gertworm describes.
- The awards go up on a board, the judges list candidates for each award.
- Strong teams will get listed for multiple awards.
- As general rule a team will get only one award. (Safety award excepted because that is judged by UL.)
- Teams will get removed or points off from consideration for bad behavior of a team members.
- The eventual EI and Chairman’s winner will shake up the awards list. Think about. The EI and chairman’s winners should be strong contenders for the other awards.
- Different JAs will run the process differently. It is really important that First picks the right judge advocates.
- The particulars of what is said in the judges room are confidential. The process is not. Or at least many of the judges I know are willing to discuss the process outside of the event.
1 Like
Be really careful making this judgement with imperfect information about what happened in the judging room, what other teams are doing and what they’re telling the judges. On the FTC side, there have been occasions where the awards we chose to give to certain teams came under fire, but I would absolutely give it to those same teams again. Your perception of who “definitely deserved the award more” is an opinion, and one that the judges may have valid reasons to disagree with based on the information they have.
Not saying this is what you’re doing, but I too often see “the judges were missing something” as an excuse to avoid reflecting on why the judges were missing something, which in my experience has rarely been “they are intentionally ignoring things” and is much more likely to be “the relevant information was not communicated to them effectively.”
Apologies to this thread for a side conversation on a side conversation.
5 Likes