Go to Post Please stand up and be proud of who you are and what you believe in! - MattK [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
 
Thread Tools Rate Thread Display Modes
  #1   Spotlight this post!  
Unread 11-04-2016, 10:51
Sperkowsky's Avatar
Sperkowsky Sperkowsky is offline
Professional Multitasker
AKA: Samuel Perkowsky
FRC #2869 (Regal Eagles)
Team Role: Leadership
 
Join Date: Jan 2015
Rookie Year: 2014
Location: Bethpage, NY
Posts: 1,885
Sperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond reputeSperkowsky has a reputation beyond repute
Re: Judge Consistency Between Events

Especially with a rookie award how hard is it to go down a list of all of the rookies in attendance. There's never too many of them. That way stuff like "you weren't mentioned" doesn't happen.

One thing I recommend is to specifically try not to be forgotten. Say something that's wows them and leaves them thinking of you.
Reply With Quote
  #2   Spotlight this post!  
Unread 11-04-2016, 15:43
philso philso is offline
Mentor
FRC #2587
 
Join Date: Jan 2011
Rookie Year: 2011
Location: Houston, Tx
Posts: 938
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by popnbrown View Post
The way FRC controls this is by training. Judge Advisors are all trained by HQ and Judges are all trained by HQ-trained-Judge-Advisors.

This discussion becomes really difficult at this point, neither of us know what training involves. We haven't been through the process of FRC judging. Frankly we don't even know what the judges at each event you went to thought. We're going to get down to anectodal evidence of one perspective. If we were to have an honest and complete discussion, I think we really owe it to figure out all perspectives before trying to change how things are done.
I would like to propose that FIRST add "The Judges are active advocates for the teams, not passive observers" in a prominent place to guarantee that the Judge Trainees are sure to see it. If the Judges do not advocate for a team, it is the same as a teacher loosing a students term paper or final exam.

The "Robot Design Judging Pre-Tournament Preparation Pack" I received this year did not contain the word "advocate" or any phrasing like it. It had plenty of information about the mechanics of the judging process but only had two sentences about the deliberation process and some boxes that say "Determine Top Teams Seen by Each Pair" and "Review and Discuss Top Teams" that only imply that the Judges should advocate for the teams that they saw.

Perhaps a short video might be appropriate since the document I received was already 34 pages long. It can also cover concepts such as "Gracious Professionalism" and "the kids do all the work (in FLL)" that may not be familiar to the volunteers who are sometimes being trained on the day of the competition and have not had time to have seen these concepts before.


Quote:
Originally Posted by popnbrown View Post
Consistency in subjectivity is really really hard. Unfortunately, the awards are subjective.
Quote:
Originally Posted by Sperkowsky View Post
One thing I recommend is to specifically try not to be forgotten. Say something that's wows them and leaves them thinking of you.
The issue of subjectivity and what "wows" the Judges is irrelevant if

Quote:
Originally Posted by Xavbro View Post
the judges that judged us didn’t advocate or mention us in the meeting when they were deciding on the awards.
I was not a Judge at this event but my wife and I have served as Judges at many FLL tournaments and I can tell you that this does happen. I don't think that the Judges who don't advocate for the teams they see have any nefarious intent. Frequently, they are first-time Judges who were trained the morning of the event and did not know they needed to advocate for the teams they saw. Some might have been doing it just so the event can take place and have no real interest in what is going on. I have also worked with Judges who are naturally shy and quiet individuals and are overwhelmed when a (group of) loud, enthusiastic and outgoing Judge talks up the teams they saw, the shy and quiet one stays quiet. There have been instances where I have politely asked quiet Judges to describe the teams they saw so that those teams got a fair chance.


Quote:
Originally Posted by popnbrown View Post
This isn't to say that I don't entertain the thought of how to continue to improve the award system. I think a program that doesn't self analyze and seek to improve will stagnate.
Yes. FIRST is about changing culture in our society. That take a lot of hard work. The hope of winning the awards given out at these events are the carrots that lead the people doing the work to continue doing the work. If there is the perception that the awards process is arbitrary rather than based on merit, the awards stop being incentives.


Quote:
Originally Posted by Andrew Schreiber View Post
Rule 1 in the Judge Room is "what happens in the Judge Room stays there" There's a lot of reasons for this, but this thread is exactly one of them.
If there is something broken in the Judge Room, it will never be fixed if it is not discussed openly and honestly. Neither the OP nor myself is trying to get them an award after the fact. We are hoping to improve the process.


Quote:
Originally Posted by TheMilkman01 View Post
In regards to them just not bringing up your name, that seems a bit strange. Given that not many rookie teams will attend the same regional, it appears off that they would completely overlook your team in that area. I would say better luck next year, but you won't be rookies then! It's good to see how you're congratulatory of the award winners and already looking forward, though. Who knows, it sounds like you have a good shot at qualifying for Worlds next year if the cards fall in the right place.
Yes, it is very strange. There were 11 rookie teams at Lone Star. Many of them were struggling to put a robot on the field that moved and were unlikely to have been in consideration for any awards. This is not meant as a criticism of those teams. Only a few would have had the foresight to do the work that 5829 did. Probably only 3 or 4 of the those rookie teams would have had the extra resources, bandwidth and foresight to have done any of the things that Judges would have been looking for.


Quote:
Originally Posted by JeffB View Post
This is a fair question to ask only if everything is the same from event to event.

Bayou: http://www.thebluealliance.com/event/2016lake
Lone Star: http://www.thebluealliance.com/event/2016txho

We can look at a variety of stats here to see the two events were a bit different. The highest OPR at Bayou would have been valid only for 4th at Lone Star. 233, one of the winning teams at Bayou, increased their OPR by 10% between these two events. This helps to show the teams you're competing against have also improved during this time. There were 10 more teams at Lone Star. They were different. Even "consistent" judging will appear different when the thing being judged changes. What you're asking for isn't consistent. "We received X award at this event and didn't win at the following event" doesn't show a lack of consistency. You need to step back and evaluate the rest of the events to see if the decisions were really all that different.
I do not think the tougher competition at Lone Star is a factor. One of the Guidelines for the Rookie All-Star Award is "This team has built a robot appropriate to the Game's challenges." The OP's team was ranked 20th at Bayou and 16th at Lone Star, a tougher event as stated by JeffB. They also made it to the Semi-Finals in both events. I think results show that they did a better job of building "a robot appropriate to the Game's challenges" than most that attended either event, including my team.

For full disclosure, the mentors for the OP's team left our team, on good terms, to help a local school start a new team. They (and 3847) very graciously allowed us to work at their facilities when the school we normally work at was closed for Spring Break. We celebrated Bag and Tag with them. You see their facility in our reveal video and the reveal video for another well established Houston area team. It is very likely that we would not have done as well this season if they had not allowed us to test our robot at their facility. I do not feel that our relationship is coloring my views about the inconsistencies in judging at these events. There is now another thread in a similar vein.

Last edited by philso : 11-04-2016 at 15:50.
Reply With Quote
  #3   Spotlight this post!  
Unread 11-04-2016, 16:31
angelah angelah is offline
Registered User
FRC #3547
Team Role: Mentor
 
Join Date: Jul 2013
Rookie Year: 2009
Location: Michigan
Posts: 88
angelah has a spectacular aura aboutangelah has a spectacular aura about
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
I do not think the tougher competition at Lone Star is a factor. One of the Guidelines for the Rookie All-Star Award is "This team has built a robot appropriate to the Game's challenges." The OP's team was ranked 20th at Bayou and 16th at Lone Star, a tougher event as stated by JeffB. They also made it to the Semi-Finals in both events. I think results show that they did a better job of building "a robot appropriate to the Game's challenges" than most that attended either event, including my team.
That is one of four requirements:
■ This team seems like a “Chairman’s Award team in the making.” (Community activities,
leadership, vision, spirit, etc.)
■ The team is a true partnership between school or organization and sponsors.
■ The team understands what FIRST is really trying to accomplish – realizes that technical stuff is
fun, challenging, and offers a future.
■ This team has built a robot appropriate to the game’s challenges.


What you are describing, to me, is the Highest Rookie Seed Award, which I assume they won if they were the highest ranked rookie. The Rookie All-Star, from my understanding (and the Awards section of the manual), is more of a rookie version of the Chairman's Award.

(I wasn't at the event, obviously; I just looked this up to find out whether Rookie All-Star was that different at a regional than at our district events.)
Reply With Quote
  #4   Spotlight this post!  
Unread 11-04-2016, 16:55
Xavbro's Avatar
Xavbro Xavbro is offline
Registered User
AKA: Xavier Eldridge
FRC #5829 (Awtybots)
Team Role: Mentor
 
Join Date: Mar 2013
Rookie Year: 2016
Location: Houston, TX
Posts: 102
Xavbro is a splendid one to beholdXavbro is a splendid one to beholdXavbro is a splendid one to beholdXavbro is a splendid one to beholdXavbro is a splendid one to beholdXavbro is a splendid one to beholdXavbro is a splendid one to behold
Re: Judge Consistency Between Events

Quote:
Originally Posted by JohnBoucher View Post
In addition to this, you need to know that the judges take the process very seriously. There are many checks and balances in place to ensure a fair event.
I know the judges take the process very seriously. We don't know what happened the judges room. We assume that they compared notes and made the best decision based off of the information they received at the event.

Quote:
Originally Posted by JeffB View Post
This is a fair question to ask only if everything is the same from event to event.

Bayou: http://www.thebluealliance.com/event/2016lake
Lone Star: http://www.thebluealliance.com/event/2016txho

We can look at a variety of stats here to see the two events were a bit different. The highest OPR at Bayou would have been valid only for 4th at Lone Star. 233, one of the winning teams at Bayou, increased their OPR by 10% between these two events. This helps to show the teams you're competing against have also improved during this time. There were 10 more teams at Lone Star. They were different. Even "consistent" judging will appear different when the thing being judged changes. What you're asking for isn't consistent. "We received X award at this event and didn't win at the following event" doesn't show a lack of consistency. You need to step back and evaluate the rest of the events to see if the decisions were really all that different.
As Phil stated above, we knew the competition was going to be a lot tougher at Lone Star. We researched the rookies we were competing against with and seeing how we were the only rookie that had been to a previous competition that had won Rookie Inspiration, we thought we had a real shot to win RAS. We stepped up our game, both RAS-wise and robot-wise, and feel we made some really good improvements from Bayou to Lone Star. Once again, nothing against 6133 and 5892, they definitely deserved the awards they won and we congratulate them.

Quote:
Originally Posted by Sperkowsky View Post
Especially with a rookie award how hard is it to go down a list of all of the rookies in attendance. There's never too many of them. That way stuff like "you weren't mentioned" doesn't happen.

One thing I recommend is to specifically try not to be forgotten. Say something that's wows them and leaves them thinking of you.
I knew we needed more LEDs.

I agree and with there being 11 rookies at Lone Star, we accounted for that and thought we would be able to stand out. The rookies in attendance also did their best to stand out too so I'm sure the judges didn't have an easy job either.

Quote:
Originally Posted by angelah View Post
That is one of four requirements:
■ This team seems like a “Chairman’s Award team in the making.” (Community activities,
leadership, vision, spirit, etc.)
■ The team is a true partnership between school or organization and sponsors.
■ The team understands what FIRST is really trying to accomplish – realizes that technical stuff is
fun, challenging, and offers a future.
■ This team has built a robot appropriate to the game’s challenges.


What you are describing, to me, is the Highest Rookie Seed Award, which I assume they won if they were the highest ranked rookie. The Rookie All-Star, from my understanding (and the Awards section of the manual), is more of a rookie version of the Chairman's Award.

(I wasn't at the event, obviously; I just looked this up to find out whether Rookie All-Star was that different at a regional than at our district events.)
Highest Rookie Seed - 6133 (#8)
Rookie Inspiration - 6133
Rookie All-Star - 5892
__________________


TigerBytes (FRC 4209): 2012-2013: Mentor
DiscoBots (FRC 2587): 2009: Member || 2010-2015: Mentor
Impact (FRC 2585): 2016-Present: Mentor
Awtybots (FRC 5829): 2015-Present: Mentor
Ri3D Team oRyon: 2014-Present: Programmer/Strategist

Last edited by Xavbro : 11-04-2016 at 17:08. Reason: typos suck
Reply With Quote
  #5   Spotlight this post!  
Unread 11-04-2016, 16:58
Bob Steele's Avatar
Bob Steele Bob Steele is offline
Professional Steamacrit Hunter
AKA: Bob Steele
FRC #1983 (Skunk Works Robotics)
Team Role: Coach
 
Join Date: Jan 2003
Rookie Year: 2002
Location: Seattle, Washington
Posts: 1,511
Bob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond reputeBob Steele has a reputation beyond repute
Re: Judge Consistency Between Events

First congratulations to you as a team. You are an up and coming star and with the enthusiasm that you are displaying you should be very successful.

After 10 years I have come to one conclusion.

If you base the success of your team on winning awards, you will have a tough time. If you base the success of your team on meeting your team goals (as long as the goals are not we want to win award "x") you will be far better off. Those goals should include inspiring your students and giving them pride in what they accomplish.

Our team has a 10 year history and I can honestly say that having inspired students and active and committed mentors lead to the best rewards. That is what being in FIRST is about. Blue banners and trophies are pretty cold, teamwork, camaraderie, loyalty, gracious professionalism are the marks of success and those lessons last forever.

I know this may sound corny but that is what it is really about.

We love to compete... we love to do our best... we love to celebrate each other's accomplishments.... that is why students keep coming back and that is why this endeavor will change lives.


Keep at it... you guys are already an All Star.... you don't need a trophy or anyone else's opinion when you know who you are.

Woodie Flowers once said: "I'm not going to tell you all that you all are winners. At this point you are smart enough to know whether you are or you aren't." Woodie is smart, listen to him....:0)

Good luck and we hope to see you on the field sometime.
__________________
Raisbeck Aviation High School TEAM 1983 - Seattle, Washington
Las Vegas 07 WINNER w/ 1425/254...Seattle 08 WINNER w/ 2046/949.. Oregon 09 WINNER w/1318/2635..SEA 10 RCA ..Spokane 12 WINNER w/2122/4082 and RCA...Central Wa 13 WINNER w/1425/753..Seattle 13 WINNER w/948/492 & RCA ..Spokane 13 WINNER w/2471/4125.. Spokane 14 - DCA --Auburn 14 - WINNER w/1318/4960..District CMP 14 WINNER w/1318/2907, District CMA.. CMP 14 Newton Finalist w 971/341/3147 ... Auburn Mountainview 15 WINNER w/1318/3049 - Mt Vernon 15 WINNER w/1318/4654 - Philomath 15 WINNER w/955/847 -District CMP 15 WINNER w/955/2930 & District CMA -CMP Newton -Industrial Design Award


Last edited by Bob Steele : 11-04-2016 at 17:03.
Reply With Quote
  #6   Spotlight this post!  
Unread 11-04-2016, 18:55
philso philso is offline
Mentor
FRC #2587
 
Join Date: Jan 2011
Rookie Year: 2011
Location: Houston, Tx
Posts: 938
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by angelah View Post
That is one of four requirements:
■ This team seems like a “Chairman’s Award team in the making.” (Community activities,
leadership, vision, spirit, etc.)
■ The team is a true partnership between school or organization and sponsors.
■ The team understands what FIRST is really trying to accomplish – realizes that technical stuff is
fun, challenging, and offers a future.
■ This team has built a robot appropriate to the game’s challenges.


What you are describing, to me, is the Highest Rookie Seed Award, which I assume they won if they were the highest ranked rookie. The Rookie All-Star, from my understanding (and the Awards section of the manual), is more of a rookie version of the Chairman's Award.

(I wasn't at the event, obviously; I just looked this up to find out whether Rookie All-Star was that different at a regional than at our district events.)
Section 6.2 of the Admin Manual gives a list of all the awards.

Highest Rookie Seed is described as "Celebrates the highest-seeded rookie
team at the conclusion of the qualifying rounds" and is selected by "Robot Performance", not the Judges. This is a non-subjective award and the OP's team could not have won this award at Lone Star since they were ranked below the rookie team that did win that award. Probably, the only thing the Judges have to do with this award is high-fiving the winners and possibly helping to write the script for introducing them.


What you have quoted is from Section 6.14.4 which is specifically for the Rookie All-Star Award.

I noticed that the section of the Admin Manual on Non-Submitted Judged Awards (6.11) does not really say how the Judges are supposed to collect the information on the teams. It also does not say that the Judges have to collect information on all the teams. I would hope that this is in the training material that the Judges are given since there is no schedule for the teams to meet with the Judges that I am aware of. If it isn't clear how and when teams are supposed to communicate their efforts and accomplishments to the Judges, it will lead to a lot of frustration.
Reply With Quote
  #7   Spotlight this post!  
Unread 11-04-2016, 23:47
angelah angelah is offline
Registered User
FRC #3547
Team Role: Mentor
 
Join Date: Jul 2013
Rookie Year: 2009
Location: Michigan
Posts: 88
angelah has a spectacular aura aboutangelah has a spectacular aura about
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
Section 6.2 of the Admin Manual gives a list of all the awards.

Highest Rookie Seed is described as "Celebrates the highest-seeded rookie
team at the conclusion of the qualifying rounds" and is selected by "Robot Performance", not the Judges. This is a non-subjective award and the OP's team could not have won this award at Lone Star since they were ranked below the rookie team that did win that award. Probably, the only thing the Judges have to do with this award is high-fiving the winners and possibly helping to write the script for introducing them.

What you have quoted is from Section 6.14.4 which is specifically for the Rookie All-Star Award.
Sorry to be unclear; yes, I was quoting the requirements from the manual for Rookie All-Star in order to clarify that robot performance is only one piece of the equation and it is more about Chairman's-type attributes. It seemed to me that you were saying they should get RAS based on just robot performance, but only Highest Rookie Seed is determined by robot performance alone. I must have misunderstood.

Last edited by angelah : 11-04-2016 at 23:57.
Reply With Quote
  #8   Spotlight this post!  
Unread 11-04-2016, 16:53
Andrew Schreiber Andrew Schreiber is offline
Data Nerd
FRC #0079
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 4,057
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
If there is something broken in the Judge Room, it will never be fixed if it is not discussed openly and honestly. Neither the OP nor myself is trying to get them an award after the fact. We are hoping to improve the process.
I'm not saying you are. I'm simply explaining that Judges are NOT allowed to share details about why a team did or did not get an award. I'm concerned that they told the op anything because they aren't supposed to. There's some discussion about how do we get feedback to teams, I'm open to ideas on this.

I can, rather freely explain the general process that goes into judging though. [1] Most of this could easily be gleaned by a careful observer, for other things I'm going to be intentionally vague.

Training

Pit Interviews

Short List

--- I tend to add a another round of Pit Interviews and Short Listing just to get another set of eyes on everyone

More Detailed Interviews

Deliberations [2]

Award Script Writing - NO FREAKING POEMS


I'll work on seeing if I can get a more detailed walk through of the process added to the manual for next year[3]. I don't want the process or what Judges are looking for to be a mystery.


Here's another fun piece of info for you, that deliberations stage is the single hardest part of judging. Know why? Because there's only a handful of awards and number of teams >>> num of awards. Spoiler - we want to give every team an award. Heck [4] worked with a judge who had only heard stories of how awesome FIRST was from Jess [5], she came in and judged at Dartmouth. Well, guess what? her company NOW sponsors a team. Just from talking to students.


Look, I wanna make judging as transparent as possible. I want teams to feel they understand what went into an award decision. But I've been on the other side of someone leaking info from a Judge room. I argued for team B to win over team A. Team A found out... only they only heard "he was arguing against giving you the award". Long story short - it was a crappy experience, I stopped volunteering for a while, nearly quit FIRST it was so crappy. It HAS to be a protected space so that judges can argue without fear of repercussions. I'm not trying to keep the process a secret, only the details.


I'll close with some tips on how to maximize chances of getting a judged award.

- Read https://frcdesigns.com/2015/07/21/5-...n-more-awards/ Kristine is a former Judge Assistant, current Event Chair, and generally awesome person.

- Be prepared, know the award criteria, know what you want to win. Ok, you built some baller vision processing code? Sell the crap outta it, and don't be shy. Go into details! Did you have an issue with a particular filter not working that you worked around? Talk about it. Just remember - some of the judges don't know as much as you do. Explain it to them like they are 5. Plus, that demonstrates you know it.

- Listen to what they are asking you. If the judge is asking about your intake mechanism and you start talking about your FLL teams you are wasting everyone involved's time. Now, if you work in "well, our intake was actually based on the intake our FLL team did last year, I was a mentor on the team. We thought back to that problem and .... " That's bonus points right there. Because now the judge has in their mind that not only is it cool, but when they are discussion RAS/EI they can go "wait, they learned from that and it impacted their performance as a team" THAT is a cool memorable story.

- Have cool memorable stories. How much time do you spend with judges? Ok, now realize they talked to 10 other teams that afternoon. They are overwhelmed with feet per second, shot percentages, OPR, or whatever other technical details. These are people. Talk to them like people. You know what? You have a cool story, you have a favorite part of the bot. Talk about it.

- Don't hand them a binder of crap. A) they have to carry it the rest of the day B) They have to worry about getting it back to you C) Dude, distill this to something I can understand quickly. You know what, it's great you have a record of every shot for any given parameter of your shooter, really, that's cool. But distill it down to an NBA style shot map and it'll stick in the mind a lot better than tables of numbers.

- Talk to them like human beings. No, seriously, MOST judges are just normal folks at the end of the day




[1] Caveat - every JA runs things slightly differently. I'll point out where I differ from what I've seen most folks. There's a lot of good reasons room processes differ but the biggest one is that each group of judges is different.


[2] This is the part I refer to as "chair throwing time"


[3] I don't make the rules, I just make a lot of noise and sometimes things change


[4] And this happened outside the judge room so I can tell this story!


[5] Who is STILL totally at fault for 2Champz /s
__________________




.
Reply With Quote
  #9   Spotlight this post!  
Unread 11-04-2016, 19:31
philso philso is offline
Mentor
FRC #2587
 
Join Date: Jan 2011
Rookie Year: 2011
Location: Houston, Tx
Posts: 938
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by Andrew Schreiber View Post
...

Deliberations [2]

...

I'll work on seeing if I can get a more detailed walk through of the process added to the manual for next year[3]. I don't want the process or what Judges are looking for to be a mystery.


Here's another fun piece of info for you, that deliberations stage is the single hardest part of judging. Know why? Because there's only a handful of awards and number of teams >>> num of awards. Spoiler - we want to give every team an award. Heck [4] worked with a judge who had only heard stories of how awesome FIRST was from Jess [5], she came in and judged at Dartmouth. Well, guess what? her company NOW sponsors a team. Just from talking to students.


Look, I wanna make judging as transparent as possible. I want teams to feel they understand what went into an award decision. But I've been on the other side of someone leaking info from a Judge room. I argued for team B to win over team A. Team A found out... only they only heard "he was arguing against giving you the award". Long story short - it was a crappy experience, I stopped volunteering for a while, nearly quit FIRST it was so crappy. It HAS to be a protected space so that judges can argue without fear of repercussions. I'm not trying to keep the process a secret, only the details.

...

[2] This is the part I refer to as "chair throwing time"
I totally understand your concern about the need for the Judging Room to stay private and that there should be no leaks. The fact of the matter is that there was a leak and the content of the leak appears to indicate there are flaws in the judging process. Considering the time, effort and resources that teams commit to this competition, it is unacceptable to be forgotten, just like it would be unacceptable for your professor to tell you that you cannot get a grade because he lost your exam papers.

Once the judging process appears to be unfair, the awards associated with that process cease to be incentives. That undermines the objectives of an otherwise great program and the efforts of people in this community such as the OP, you and I. Rightly or wrongly, there have been whole teams that have left FLL in our area because of awards that were not given out in a rational and transparent way.

Having worked as an FLL Judge quite a few times, I know what you mean about throwing chairs. Sometimes, the Judges get about as passionate advocating for the teams they saw as the teams themselves (I think this is a good thing)

It would be greatly appreciated if you can help improve the manuals and other materials used to train the Judges. Not being a mystery to the competitors would also mean it would be easier for the Judges to be consistent from one event to another and within an event.
Reply With Quote
  #10   Spotlight this post!  
Unread 11-04-2016, 19:57
Andrew Schreiber Andrew Schreiber is offline
Data Nerd
FRC #0079
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 4,057
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
I totally understand your concern about the need for the Judging Room to stay private and that there should be no leaks. The fact of the matter is that there was a leak and the content of the leak appears to indicate there are flaws in the judging process. Considering the time, effort and resources that teams commit to this competition, it is unacceptable to be forgotten, just like it would be unacceptable for your professor to tell you that you cannot get a grade because he lost your exam papers.

Once the judging process appears to be unfair, the awards associated with that process cease to be incentives. That undermines the objectives of an otherwise great program and the efforts of people in this community such as the OP, you and I. Rightly or wrongly, there have been whole teams that have left FLL in our area because of awards that were not given out in a rational and transparent way.

Having worked as an FLL Judge quite a few times, I know what you mean about throwing chairs. Sometimes, the Judges get about as passionate advocating for the teams they saw as the teams themselves (I think this is a good thing)

It would be greatly appreciated if you can help improve the manuals and other materials used to train the Judges. Not being a mystery to the competitors would also mean it would be easier for the Judges to be consistent from one event to another and within an event.
Again, I question that the wording used by the judges implied the team was forgotten. More accurately it likely meant that the other team had impressed more folks, or the folks were more adept at swaying others. Or any of a dozen things.

I can assure you that the #1 goal of every JA in FRC is to ensure that every team has a positive experience that they feel was fair. Comments like those judges made undermine that.
__________________




.
Reply With Quote
  #11   Spotlight this post!  
Unread 11-04-2016, 20:14
philso philso is offline
Mentor
FRC #2587
 
Join Date: Jan 2011
Rookie Year: 2011
Location: Houston, Tx
Posts: 938
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by Andrew Schreiber View Post
Again, I question that the wording used by the judges implied the team was forgotten. More accurately it likely meant that the other team had impressed more folks, or the folks were more adept at swaying others. Or any of a dozen things.

I can assure you that the #1 goal of every JA in FRC is to ensure that every team has a positive experience that they feel was fair. Comments like those judges made undermine that.
I am not questioning the integrity of the Judges. I know it is a difficult job for them. I am now also wondering if they were "Miss Universed" because of a simple "clerical error". That would go a long way towards explaining why "their number was not mentioned". This year, the Lone Star FLL Championship Tournament had an issue with a team being disqualified erroneously (different organizers). The error was only discovered after the event had ended.
Reply With Quote
  #12   Spotlight this post!  
Unread 12-04-2016, 06:32
GaryVoshol's Avatar
GaryVoshol GaryVoshol is offline
Cogito ergo arbitro
no team
 
Join Date: Aug 2005
Rookie Year: 2000
Location: Royal Oak, MI
Posts: 5,726
GaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond reputeGaryVoshol has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by Andrew Schreiber View Post
Again, I question that the wording used by the judges implied the team was forgotten. More accurately it likely meant that the other team had impressed more folks, or the folks were more adept at swaying others. Or any of a dozen things.
My thoughts as well. "Your team didn't come up" may mean that the judges that saw your team had another candidate that they thought was better qualified and they presented that team to the rest of the judges.

Here's an example. I've judged in FLL a couple of times. There was a team that was being considered for the Champions Award. Someone noted that the team wasn't nominated for any of the Project awards; does that indicate they were not a well-rounded team? I mentioned that they were on our short list but not at the top and so they weren't mentioned before. They were good; it's just that there were others that were better.
__________________
(since 2004)
Reply With Quote
  #13   Spotlight this post!  
Unread 11-04-2016, 19:42
popnbrown's Avatar
popnbrown popnbrown is offline
FIRST 5125 HOTH Lead Mentor
AKA: Sravan S
FRC #5125 (Hawks on the Horizon)
Team Role: Mentor
 
Join Date: Feb 2007
Rookie Year: 2007
Location: Illinois
Posts: 367
popnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
I would like to propose that FIRST add "The Judges are active advocates for the teams, not passive observers" in a prominent place to guarantee that the Judge Trainees are sure to see it.
Again, can't comment to FRC, but as a FLL and FTC Judge Advisor, I use training time to clarify this. Deliberations are one of the toughest things and I make sure to tell my judges to "Speak up if there's anything to say".

Quote:
but only had two sentences about the deliberation process and some boxes that say "Determine Top Teams Seen by Each Pair" and "Review and Discuss Top Teams" that only imply that the Judges should advocate for the teams that they saw.
Technically, these are the teams that they should be advocating primarily for because they have the most familiarity with them. Also the second "Review and Discuss Top Teams" doesn't limit them to the ones they saw. To me the two phrases sound like give top teams you saw, and discuss top teams in general.

Quote:
Perhaps a short video might be appropriate since the document I received was already 34 pages long.
Again...all FLL/FTC based.
I vaguely recall a webinar floating around perhaps it was IL specific. This is the crappy part, is that judging has significant effort involved. There's a document, video and webinar that you have to go through. Then the day of the event, you show up early and have a few extra hours of training. Then you rush through the judging in one day. As someone who's brand new to FIRST...this is a tough position to fill in.


Quote:
Frequently, they are first-time Judges...have no real interest in what is going on...are naturally shy and quiet individuals and are overwhelmed when a (group of) loud, enthusiastic and outgoing Judge talks up
These are interesting issues we can try to tackle.

First-time Judges:
There's unfortunately no way around some percentage of first-time Judges. The solution, specifically for Illinois FTC, is two-fold
  1. have enough veteran judges to pair up
  2. give as many opportunities for training as possible (Judging Manual, Webinar/Call 1wk b4, Morning Training)

No Real Interest:
The only way around this is to up our volunteer coordination game. The goal with this is to try and recruit judges at the right time. For Illinois FTC, this just means ensuring our Event Coordinators know this, and it has been getting better. For FRC as far as I can tell, the JA for Midwest Regional seemed really prepared a few months out.

Shy Judges:
Illinois FTC's way of solving this, is to have well trained Judge Advisors. Ensuring equitable deliberation time and moderating the conversation is the role of the JA. They should bring out as much information as they can. BUT they also need to do it quickly and efficiently (a really hard thing). So we have state-trained JAs (just like FRC has HQ-trained JAs) to make sure we get them communicating about their experiences.


Quote:
The hope of winning the awards given out at these events are the carrots that lead the people doing the work to continue doing the work.
I think we've raised in this thread whether it should be the sole carrot. Personally as a mentor, it is not the carrot that leads me to do the work. I would consider not submitting for Chairman's but would not (unless students had really good reasons) stop continuing to host FTC events, mentor FLL teams, and plant tulips at our school. These activities allow me to teach students how to be organized, how to be a leader/mentor, and how we affect our environment/how "it's-not-easy-to-just-plant-tulips".

Quote:
If there is something broken in the Judge Room, it will never be fixed if it is not discussed openly and honestly.
The number one issue is that a lot of our first-time judges stay as first-time judges. There's tons of MN threads going on, and a lot of people discussing how volunteer shortage is a big issue. FIRST, as an organization, is like 90% (I want to even say 99%) volunteer-driven and FUNDAMENTALLY relies on its volunteer pool for operation. While I understand that perceived inconsistency is an issue, my point is this we need to determine what is the root and which is more important, more necessary to solve. 1) Bringing greater consistency into judging or 2) ensuring that we continue to have judges, good judges.

For 1) we only have anecdotal evidence, with very many unknowns and a lack of perspectives. In addition, winning awards should not be the end-all for teams. There are probably other team issues that need fixing too. Frankly, what is broken in the Judge Room (at many events) is 2), there's a constant strain on volunteer pool (in some regions), a lack of volunteers, and a lot of last-minute judges.

It's why I'm imploring OP to become a judge. Not only will we get closer to solving 2), but we'll also get more perspectives for 1).

I honestly see no resolution to this particular case. In my view, OP is probably a tired mentor (long season...) and got bummed out his students got bummed out. It sounds like his team worked really hard and was a great rookie contender, and got beat out. It also sounds like his team is going to be striving to kick even more butt in the second year. This is the best resolution I can foresee. His team got beat out by great teams, and he's going to motivate his students to work hard.

Since I still advocate for solving 2), I also hope he takes my advice and tries to judge or convinces others to judge.
__________________
I am an employee of FIRST. However, the postings on this site are of my own perspective as a FIRST mentor and volunteer and do not necessarily reflect the views of FIRST.

FIRST Team 5125 Hawks on the Horizon Lead Mentor
FRC Team 4096 Ctrl-Z Former Mentor
FTC Team 5203 #19@! Former Mentor
FRC Team 1403 Cougar Robotics Alumni
Reply With Quote
  #14   Spotlight this post!  
Unread 11-04-2016, 20:05
philso philso is offline
Mentor
FRC #2587
 
Join Date: Jan 2011
Rookie Year: 2011
Location: Houston, Tx
Posts: 938
philso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond reputephilso has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by popnbrown View Post
Again, can't comment to FRC, but as a FLL and FTC Judge Advisor, I use training time to clarify this. Deliberations are one of the toughest things and I make sure to tell my judges to "Speak up if there's anything to say".

...


I think we've raised in this thread whether it should be the sole carrot. Personally as a mentor, it is not the carrot that leads me to do the work. I would consider not submitting for Chairman's but would not (unless students had really good reasons) stop continuing to host FTC events, mentor FLL teams, and plant tulips at our school. These activities allow me to teach students how to be organized, how to be a leader/mentor, and how we affect our environment/how "it's-not-easy-to-just-plant-tulips".

...

Since I still advocate for solving 2), I also hope he takes my advice and tries to judge or convinces others to judge.
It is great that you have a more thorough training process for the Judges. It would be great if more of FIRST could follow your example, especially if it is a matter of saying a few more words. Perhaps you can provide your training materials to the FIRST.

Please be aware that the awards are also used to generate support, financial and otherwise, for the teams from their schools and sponsors, whether or not the team members are truly motivated by those awards.

Yes, being able to recruit more Judges (and other volunteers) and retaining them would make everyone's experience much better, including the Judges and volunteers. Being able to recruit and retain volunteers is a problem that FIRST, like many other volunteer driven organizations, must solve.
Reply With Quote
  #15   Spotlight this post!  
Unread 11-04-2016, 20:10
popnbrown's Avatar
popnbrown popnbrown is offline
FIRST 5125 HOTH Lead Mentor
AKA: Sravan S
FRC #5125 (Hawks on the Horizon)
Team Role: Mentor
 
Join Date: Feb 2007
Rookie Year: 2007
Location: Illinois
Posts: 367
popnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond reputepopnbrown has a reputation beyond repute
Re: Judge Consistency Between Events

Quote:
Originally Posted by philso View Post
It would be great if more of FIRST could follow your example, especially if it is a matter of saying a few more words. Perhaps you can provide your training materials to the FIRST.
I feel like I'm in an experimental stage, and there are certainly flaws to my proposal. So as they get worked out they will surely be shared with FIRST, but a lot of the inspiration was from current judging processes from FIRST. Really..it's just been localized.

Quote:
Please be aware that the awards are also used to generate support, financial and otherwise, for the teams from their schools and sponsors, whether or not the team members are truly motivated by those awards.
I'm well aware, and I do think there's merit to having awards be a motivation. I just want to raise the point that it shouldn't be the only or primary motivation.
__________________
I am an employee of FIRST. However, the postings on this site are of my own perspective as a FIRST mentor and volunteer and do not necessarily reflect the views of FIRST.

FIRST Team 5125 Hawks on the Horizon Lead Mentor
FRC Team 4096 Ctrl-Z Former Mentor
FTC Team 5203 #19@! Former Mentor
FRC Team 1403 Cougar Robotics Alumni
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 15:33.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi