|
|
|
![]() |
|
|||||||
|
||||||||
![]() |
|
|
Thread Tools | Rate Thread | Display Modes |
|
|
|
#1
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
Last edited by angelah : 11-04-2016 at 23:57. |
|
#2
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
I can, rather freely explain the general process that goes into judging though. [1] Most of this could easily be gleaned by a careful observer, for other things I'm going to be intentionally vague. Training Pit Interviews Short List --- I tend to add a another round of Pit Interviews and Short Listing just to get another set of eyes on everyone More Detailed Interviews Deliberations [2] Award Script Writing - NO FREAKING POEMS I'll work on seeing if I can get a more detailed walk through of the process added to the manual for next year[3]. I don't want the process or what Judges are looking for to be a mystery. Here's another fun piece of info for you, that deliberations stage is the single hardest part of judging. Know why? Because there's only a handful of awards and number of teams >>> num of awards. Spoiler - we want to give every team an award. Heck [4] worked with a judge who had only heard stories of how awesome FIRST was from Jess [5], she came in and judged at Dartmouth. Well, guess what? her company NOW sponsors a team. Just from talking to students. Look, I wanna make judging as transparent as possible. I want teams to feel they understand what went into an award decision. But I've been on the other side of someone leaking info from a Judge room. I argued for team B to win over team A. Team A found out... only they only heard "he was arguing against giving you the award". Long story short - it was a crappy experience, I stopped volunteering for a while, nearly quit FIRST it was so crappy. It HAS to be a protected space so that judges can argue without fear of repercussions. I'm not trying to keep the process a secret, only the details. I'll close with some tips on how to maximize chances of getting a judged award. - Read https://frcdesigns.com/2015/07/21/5-...n-more-awards/ Kristine is a former Judge Assistant, current Event Chair, and generally awesome person. - Be prepared, know the award criteria, know what you want to win. Ok, you built some baller vision processing code? Sell the crap outta it, and don't be shy. Go into details! Did you have an issue with a particular filter not working that you worked around? Talk about it. Just remember - some of the judges don't know as much as you do. Explain it to them like they are 5. Plus, that demonstrates you know it. - Listen to what they are asking you. If the judge is asking about your intake mechanism and you start talking about your FLL teams you are wasting everyone involved's time. Now, if you work in "well, our intake was actually based on the intake our FLL team did last year, I was a mentor on the team. We thought back to that problem and .... " That's bonus points right there. Because now the judge has in their mind that not only is it cool, but when they are discussion RAS/EI they can go "wait, they learned from that and it impacted their performance as a team" THAT is a cool memorable story. - Have cool memorable stories. How much time do you spend with judges? Ok, now realize they talked to 10 other teams that afternoon. They are overwhelmed with feet per second, shot percentages, OPR, or whatever other technical details. These are people. Talk to them like people. You know what? You have a cool story, you have a favorite part of the bot. Talk about it. - Don't hand them a binder of crap. A) they have to carry it the rest of the day B) They have to worry about getting it back to you C) Dude, distill this to something I can understand quickly. You know what, it's great you have a record of every shot for any given parameter of your shooter, really, that's cool. But distill it down to an NBA style shot map and it'll stick in the mind a lot better than tables of numbers. - Talk to them like human beings. No, seriously, MOST judges are just normal folks at the end of the day [1] Caveat - every JA runs things slightly differently. I'll point out where I differ from what I've seen most folks. There's a lot of good reasons room processes differ but the biggest one is that each group of judges is different. [2] This is the part I refer to as "chair throwing time" [3] I don't make the rules, I just make a lot of noise and sometimes things change [4] And this happened outside the judge room so I can tell this story! [5] Who is STILL totally at fault for 2Champz /s |
|
#3
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
Once the judging process appears to be unfair, the awards associated with that process cease to be incentives. That undermines the objectives of an otherwise great program and the efforts of people in this community such as the OP, you and I. Rightly or wrongly, there have been whole teams that have left FLL in our area because of awards that were not given out in a rational and transparent way. Having worked as an FLL Judge quite a few times, I know what you mean about throwing chairs. Sometimes, the Judges get about as passionate advocating for the teams they saw as the teams themselves (I think this is a good thing)It would be greatly appreciated if you can help improve the manuals and other materials used to train the Judges. Not being a mystery to the competitors would also mean it would be easier for the Judges to be consistent from one event to another and within an event. |
|
#4
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
I can assure you that the #1 goal of every JA in FRC is to ensure that every team has a positive experience that they feel was fair. Comments like those judges made undermine that. |
|
#5
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
|
|
#6
|
|||||
|
|||||
|
Re: Judge Consistency Between Events
Quote:
Here's an example. I've judged in FLL a couple of times. There was a team that was being considered for the Champions Award. Someone noted that the team wasn't nominated for any of the Project awards; does that indicate they were not a well-rounded team? I mentioned that they were on our short list but not at the top and so they weren't mentioned before. They were good; it's just that there were others that were better. |
|
#7
|
||||||
|
||||||
|
Re: Judge Consistency Between Events
Quote:
Quote:
Quote:
I vaguely recall a webinar floating around perhaps it was IL specific. This is the crappy part, is that judging has significant effort involved. There's a document, video and webinar that you have to go through. Then the day of the event, you show up early and have a few extra hours of training. Then you rush through the judging in one day. As someone who's brand new to FIRST...this is a tough position to fill in. Quote:
First-time Judges: There's unfortunately no way around some percentage of first-time Judges. The solution, specifically for Illinois FTC, is two-fold
No Real Interest: The only way around this is to up our volunteer coordination game. The goal with this is to try and recruit judges at the right time. For Illinois FTC, this just means ensuring our Event Coordinators know this, and it has been getting better. For FRC as far as I can tell, the JA for Midwest Regional seemed really prepared a few months out. Shy Judges: Illinois FTC's way of solving this, is to have well trained Judge Advisors. Ensuring equitable deliberation time and moderating the conversation is the role of the JA. They should bring out as much information as they can. BUT they also need to do it quickly and efficiently (a really hard thing). So we have state-trained JAs (just like FRC has HQ-trained JAs) to make sure we get them communicating about their experiences. Quote:
Quote:
For 1) we only have anecdotal evidence, with very many unknowns and a lack of perspectives. In addition, winning awards should not be the end-all for teams. There are probably other team issues that need fixing too. Frankly, what is broken in the Judge Room (at many events) is 2), there's a constant strain on volunteer pool (in some regions), a lack of volunteers, and a lot of last-minute judges. It's why I'm imploring OP to become a judge. Not only will we get closer to solving 2), but we'll also get more perspectives for 1). I honestly see no resolution to this particular case. In my view, OP is probably a tired mentor (long season...) and got bummed out his students got bummed out. It sounds like his team worked really hard and was a great rookie contender, and got beat out. It also sounds like his team is going to be striving to kick even more butt in the second year. This is the best resolution I can foresee. His team got beat out by great teams, and he's going to motivate his students to work hard. Since I still advocate for solving 2), I also hope he takes my advice and tries to judge or convinces others to judge. |
|
#8
|
|||
|
|||
|
Re: Judge Consistency Between Events
Quote:
Please be aware that the awards are also used to generate support, financial and otherwise, for the teams from their schools and sponsors, whether or not the team members are truly motivated by those awards. Yes, being able to recruit more Judges (and other volunteers) and retaining them would make everyone's experience much better, including the Judges and volunteers. Being able to recruit and retain volunteers is a problem that FIRST, like many other volunteer driven organizations, must solve. |
|
#9
|
||||
|
||||
|
Re: Judge Consistency Between Events
Quote:
Quote:
|
|
#10
|
||||
|
||||
|
Re: Judge Consistency Between Events
Rule number 1 we don't talk about judge club?
|
|
#11
|
||||
|
||||
|
Re: Judge Consistency Between Events
First off, congratulations on getting through your first season! I hope it wasn't too exhausting and your team is excited for a second season!
I want to preface this by saying, I've never been a judge for FRC so I don't know the process. I have been both a student and a mentor. I do have extensive experience with judging for FLL and FTC, so I'll try to draw that in as well. Consistency in subjectivity is really really hard. Unfortunately, the awards are subjective. I base that on the description of the awards themselves, the competition for that award (ie. The teams at the event), and most importantly the ever changing human component. Not only the judges as the human component, but the students and mentors also involved. I understand the students are upset and that you see that there should be more consistency. But even if we were able to get over the logistical nightmare of having judges noted be communicated across events, there's still subjectivity of the event itself. And I think it would not be in the program's interest to ask judges to value other notes over their own observations. I hope I'm not coming off aggressively. I think you're being very calm and collected so I want to approach the same way. The problem you present is basically this, how can we turn something inherently subjective into something more objective? It's something I've been interested in for a long time, especially as a participant. But what I've come to realize is to solve that problem we really have to ask why does the answer matter? That's been a question for me for the past two years as I've transitioned into becoming a Lead Mentor...the lead "guidance" for a bunch of teenagers. And what I've discovered is that while my students may be motivated to win awards, it's my responsibility to ensure they stay motivated, if they win and award, if they don't win, if they get screwed out of one. Rather than trying to figure out what is the fairest way to give the award, my highest priority is ensuring my students continue to learn, work hard and be proud of their failures and successes. This isn't to say that I don't entertain the thought of how to continue to improve the award system. I think a program that doesn't self analyze and seek to improve will stagnate. But my priorities are not on figuring out awards. Anyways, sort of back on subject I highly suggest you give a whirl with judging if you have not already to see how you can improve consistency. |
![]() |
| Thread Tools | |
| Display Modes | Rate This Thread |
|
|