Experienced Judges vs Unbiased Judges

So i was thinking about the difficulties of, well, any growing region in FRC competition wise. One of the struggles is always finding experienced and dedicated volunteers.
Here comes the dilemma, and i would like to get some feedback from various regions and see how they handle it.

We all want experienced judges, who know what FIRST is about, who know what are the difficulties of building a robot, a team and a community outreach program. In most cases, the best people for that job would be experienced mentors and team alumni. Problem is, if you gather your ‘local’ pool of volunteers, you will find that you have judges volunteering in event where teams they are associated with are competing, and in that case even the best once will be at least a little biased. It’s beyond a persons own self control.

This means that one has to decide, between experienced and slightly biased judges, or just having inexperienced, unbiased judges (AKA volunteers from the local industry/academy, unrelated to any team).

What does your region do?

At Bayou, the judges are from local tech companies and academia who are not directly involved in any FIRST teams. At the FLL events (and presumably the FTC events), judges are a mix of non-FIRSTers and FRC mentors; I have judged several FLL events, and an FLL-like event at the National WWII Museum.

As to referees (you didn’t ask, but maybe that’s what you meant), I know that the FLL referees are usually associated with FRC teams, as our students and mentors have served in this capacity regularly, at local and state FLL events and offseasons. I don’t know about the Bayou FRC referees.

Edit: Another possibility is to induce FRC mentors & coaches from surrounding areas to come referee.

I’ll take experienced, personally, bias or no.

And the reason is that bias can be worked around, and there are a number of ways to do that–the easiest is that you simply have the person not participate in discussions involving their own team, second easiest is to “balance the bias” and have only one person biased towards any given team. Experience can only be gained by doing in most cases. And with that experience, you learn how to shut down the biases, or mitigate them. It takes time, and discussion, but it can be mostly taken care of.

Now, that being said…

In my area, judges tend to be professionals, not exactly team-affiliated. But many have been doing this for years. And the JAs are either heavily experienced or team-affiliated or both.

And there is the “surrounding area” routine as well. What that does is it gives a bias, but none of the biases are towards any given local team (other than maybe some “friend teams”). It’s a lot easier than you might think–unless you’re in districts, or a rural area…

I think an issue with this is that you are making it seem mutually exclusive. You can have dedicated volunteers that are unaffiliated, like those who come in from the industry. While Judges might not know the detailed specifics of what we do and don’t do, that’s not saying they’re unaware of how difficult the robot production and team building processes are. FIRST tries to mirror the real world in many cases, so it wouldn’t be out of the question for someone from the industry to have relatable experience with what we do.

And even as a Judge, you’re not supposed to be reviewing your own team or one you have any affiliation with. At least that’s the experience I had Judging for FLL. I trust that any volunteer worth their salt is going to put the spirit of the event and the other volunteers’ opinions before their own agenda.

What’s most important to creating this “Dedicated but unaffiliated” volunteer is making sure that we get volunteers just as hooked on this program as we are. We need to make the experience fun, and show that competitions are an event worth giving your time to. I have a zero-tolerance policy when it comes to my students (or other volunteers) being disrespectful to volunteers - it certainly isn’t a compelling case to return, and personally is a massive reason as to why I will not volunteer in certain parts of the West Coast.

tl;dr Make unaffiliated volunteers want to come back to volunteering, and you will fix both problems.

For a rural. district event like i’m coming from (Israel) it would be impossible to involve people from close by areas, and i’d assume the same goes for Australia, Canada and maybe even CA.

Wev’e had judges from the industry here for years, and iv’e witnessed cases on yearly basis when it just doesn’t work.
Every team will go out and tell their own story, and for judges who don’t really go into depth in FRC, everything will seem amazing as told by students. Incredible motion profiling and vision done by elite teams can seem just as incredible as a team that will demonstrate their use of CAN and encoders. A person who doesn’t do FIIRST will find it hard to tell why a turret is so hard to execute well, and might sometimes find easy and elegant solution worse than complex once that don’t work.

Furthermore, when watching a game, it will be hard for them to tell a good robot from an incredible robot.

I really like the ‘balancing it out’ method. How has that worked for regions that tried it out? Has it improved the over-all judging quality?

Dead wrong on CA. 8 regional events, in two groups of 4, not uncommon to have volunteers from a nearby area whose team isn’t at the event. (L.A., Orange County, Ventura, and San Diego are within about 4 hours of each other–from far north to far south–so it’s not terribly difficult travelwise.)

Basically, it’s not getting someone from another area nearby, it’s getting someone from relatively close whose team is not competing at the event. For Israel, that might be a little more challenging, true, but I would think that that would still be possible to pull off.

This year, Israel had 4 competitions plus DCMP. There were two sets of competitions back-to-back weeks, where each set was back-to-back in the middle of the same week in the same location. Two competitions were in Haifa (north) and two were in Tel Aviv (middle). It’s about an hour and a half drive between the two. Many teams went to one competition in each location so they would have time to rest and make changes between their events, which under normal circumstances would mean that mentors would be available at the other competitions to volunteer.

The thing that makes this situation more difficult than usual is that if you are at one competition and want to volunteer at the next, that’s pretty much your entire week gone (Mon-Thu). Not to mention that you would then be taking off at least part of the next week for the second competition. Whereas in CA you can be at a number of competitions in a row only missing one day of work each. I expect that would make mentors less eager to volunteer.

In my opinion, Israel FIRST needs to look for volunteers from the local industry and technological colleges (which are plentiful both in Haifa and Tel Aviv). The important thing will be retaining these volunteers from year to year. They may not understand the intricacies of the robots perfectly their first year, but over time they will learn to distinguish between the expected and the extraordinary.

Of the 4 SoCal regionals, only 1 was on a Sunday. If someone was at all 4 SoCal events, they’d miss 7 days. It might have been different up north–I think two of the NorCal events were F-S-S–but that’s out of range for volunteering if the team isn’t going, in general. (There are exceptions, I might add.)

Now, in terms of missing work, I consider F-S-S as bad as T-F-S, because I’ll take the Monday off afterwards and recover. I’m not everyone, though. (I took the entire week between L.A. and OCR off because I figured that showing up for two days midweek wasn’t worth the trouble.)

The rest of the post is spot on, just a reminder to check the calendar of the area you’re discussing if you aren’t familiar with it.

Isn’t there an unspoken rule that judges with any affiliation to a team up for an award are to ‘back out’ and not judge a particular team they would think they have a bias toward? That’s what our mentor does, when our team is up for an award. Wouldn’t that remove a large part of the issue if all judges did that?

But to answer the question, I’d rather experienced judges. I think those who truly understand a team and think they deserve to win an award are going to be inherently biased, but this is good if that team is truly deserving of the award, when placed against all other eligable teams.

Without going into details, I can assure you that the rule is more than just unspoken.

Google “FIRST FRC judge info”, the following page comes up:

Direct your attention to “*Judges must sign a Conflict of Interest and Disclosure Statement prior to start of service.” Following the link to the Conflict of Interest form:

Quote the form “By signing this document, I, the undersigned agree to disclose any direct relationship I may have with a FIRST team. I also agree, if I do have a relationship with a competing FIRST team, to follow the directions given to me during official training for my position this year, or if no specific instructions were given, to recuse myself from any discussions which involve my team.”

You can judge (as I have) with a conflict of interest, you will typically be assigned where you will not need to judge your team and if it does happen to come up you just recuse yourself from any discussions directly involving your team conflict(s). No matter what you follow the directions of those appointed above you.

As to the new judges versus experienced team related people, it is typically easier to bring in outside technical minded engineering people to a robotics competition. They are often assigned to judge the machine based awards as that is what they are most familiar with, depending on their field of expertise of course. After judging a few times and becoming more familiar with the culture and mission of FIRST, then they are more comfortable judging the culture and team awards.

I will say that judging is extremely rewarding. The drive, intelligence, ingenuity, outside of the box thinking and just pure passion that the students show is extremely inspiring. From what I have seen, bringing in outside people even once to judge can and has resulted in many long time repeating judges/volunteers.

Indeed. At both local and state FLL competitions where I’ve been a judge, it appears that teams were assigned so as to make sure that no one was in the position of judgeing a team they might even be be indirectly associated with. Even when judges “give evidence” regarding a team with an indirect link, many (and presumably all) are careful to give “full disclosure” of that indirect link.

Same with other groups at the FRC level.

As an inspector, I will flatly refuse to inspect any team I’m affiliated with. That’s easy enough, there’s other inspectors around. If I’m in stripes, I can’t always avoid it, so I do my best to stay out of “discussions” involving my team(s). (And I remember this line… Someone out there is counting on the game being called right.)

A good JA can usually handle any conflicts of interest as long as they know about them (judges note conflict of interest on their forms).

From my experience, having judges that know what an FRC team is all about is valuable to have in the judging room. And it’s nice as a team to have someone with FRC experience as your judge.

IMO, having a mix of FRC experience and real world experience with your judges and a good JA is the most ideal situation.

And then there’s reality. It’s not always easy to recruit judges of either skill set. So if you know someone that would be a good judge, pass that info along to the Volunteer Coordinator, Regional Director, Planning Committee, FIRST Senior Mentor so they can be recruited.

When you sign on new judges at a particular event do you ever walk them through what they should expect to see and what they should look out such as whether a team is using can which can usually today be easily identified if the team is using srx’s with all of them daisy chained?

It’s not possible or practical. There is barely enough time to introduce them to what FIRST and FRC is let alone educating them about what to expect on the robots. Teaming a new judge with an experienced judge is the best way to get them acclimated to what they are seeing on the robot.

And if you focus on volunteer retention the same way we focus on student and team retention, those new judges will become experienced judges in a few years, ready to teach new judges. The problem comes when there is no volunteer retention from year to year and you are left with all new judges with no FIRST experience every year.

OK, so here comes the next question.
Judges for Rookie All Star, EI and Chairman’s.
The teams that reach the highest level of community programs are usually well known for that. Judges who really get that, in many cases, will be, at least somehow affiliated with those teams (Involved in some of their projects, worked together before, or just good friends of involved individuals).
This means that however they try, it will be challenging to be completely unbiased.
What is done in your region to handle that issue?

*Disclaimer: I’m not a regional director, or a volunteer coordinator. Just looking for solutions to help FIRST Israel grow.

I’m pretty positive to be a judge for these you have to inform the event of any conflicts of interest, but I may be in wrong in which case that sounds highly inappropriate. Being a friend of a team is more likely though.

Question is, what happens after informing?