Chief Delphi - Call Inconsistencies Between Regionals

Chief Delphi (http://www.chiefdelphi.com/forums/index.php)

- Rules/Strategy (http://www.chiefdelphi.com/forums/forumdisplay.php?f=6)

- - Call Inconsistencies Between Regionals (http://www.chiefdelphi.com/forums/showthread.php?t=65868)

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by Anne Shade (Post 719502)

2. Provide for a method of evaluating the referees, specifically the head referees. Training is only half of the equation. Training provides the knowledge but is it being applied correctly? Referees in all professional sports are evaluated by their organization after all games. If too many mistakes are made, they are re-trained or released. Where is the evaluation for FIRST referees? How do we know that the training received was effective if their is no evaluation of performance? Some people are just not cut out for the job but we have no way to determine that until something goes grossly wrong.

Quote:

Originally Posted by Brandon Holley (Post 719545)

My suggestions:
1. There absolutely must be a way to train these referees better than just a test. Corey balint had suggested a video of possible scenarios played out via animation or even people robots like done in the kickoff examples, and do this pre regional events.

Perhaps we can take care of both of issues with one stone, maybe 3 issues with one stone. Ever thought about focus groups of FIRST Referees?

The difficulties of using video footages to train refs is an excellent idea, but I don't think animation is going to cut it. We literally need footages from the first week regionals. It's true that there will be no footages until the first week, but that is not that different from how things are now.

The Refs and the GDC go into week 1 without a clear vision of how things will played out, but their vision is refined after that, and refined again after another week, and so on. So, its a matter of how you can capture the first week to help you train the Refs better in the weeks that follow. Besides, there are certain elements that will continue to come back year after year, such as entanglement, tipping, and high speed ramping.

Anyway, back to the focus group. Here is the idea. You look at videos from week one. You look for instances where rule violations occur. And you look for them so much that you begin build up a spectrum of instances between clearly acceptable acts and clear violations. Say you get 5 video clips of ramming that give you an understanding of, on a scale of 1 to 5, what's acceptable ramming and what's unacceptable, and the gray area in between.

Here is how you apply these videos to training and evaluation. During the training, you show 1 clip as an example of clear violation of a certain rule. After the training is over, you show 5 clips of the same rule, and you ask the person in training which ones they feel are violation and which ones aren't. You give them a chance to make mistakes so they have a better understanding of the variation.

Then you show them another 5 clips for another evaluation. After you repeat that once or twice, you should have a good idea whether that person has caught on to what's acceptable and what isn't.

For training purposes, it is very important for the head Referee at the headquarter to decide, on scales of 1 to 5, what is acceptable and what isn't. Once a line is drawn, you drill that line into all referees' head in the weeks that follow. Consistency is what's important here. And you continue to use these clips and newer ones to train and evaluate refs before and after the events, and before and after the years.

Won't be for every rule of course, just the difficult ones.

Here are some side bonuses:

1. You release some of these clips to the teams so they too will get a better understanding of what's acceptable and what isn't.
2. You collect data as people evaluate various clips. The more data you collect, the better you are at understanding, in general, how people are calling certain rules. This understanding will help you decide whether a rule need to be better written, or if the training need to be improved.

Sounds complicated, right? You betcha! There is probably a simplier idea in here somewhere. On the other hand, there are many people watching many videos of many competition during the weeks. Won't be hard to find volunteers whose job is to identify these clips for the GDC and the head Ref at headquarter. Once a system is in place, Refs and refs in training just need to go to the training site and watch some videos.

Re: Call Inconsistencies Between Regionals

Ken, I think you have some valid points but the evaluation I was refering to is an evaluation of a head referees' performance at a regional. There needs to be a way to say that a head ref did a good or bad job and to determine who is not right for the job. Tests and questionaires only get you so far in the evaluation of a person's skills and performance.

Part of that evaluation could be team and key regional personel feedback but most really should be video review or in person evaluation. Why not have the committee of championship head referees review random matches from each regional and evaluate the play calling and the refereeing procedures utilized by the head referee? Head referees would receive feedback on how they can improve and those that are not performing up to par can be replaced.

This is the second part of training. Without the evaluation, the training has very little value.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by Ken Streeter (Post 719958)

Hmm. In cases of unrestrained motion, the trackball positions are scored based upon where they do finally come to rest, not where they are when the clock reaches zero. If BMR was scoring all trackballs based upon where they were when the clock reached zero, that was incorrect based upon Team Update #13, which resulted in rule G14 being updated as follows:

Quote:

Originally Posted by Ryan Dognaux (Post 719969)

Maybe they weren't counting them because they were knocked off after the clock hit zero. Not exactly sure, but it could be the case. At least in one of our matches, our partner (1018 I think) knocked a ball off right at the end and it was still counted as being on the overpass. The officiating was pretty consistent either way. Tons of penalties of course.

Quote:

Originally Posted by IndySam (Post 719981)

I didn't see that, what I saw were balls that were knocked off right at the buzzer and were rightfully counted because they were in contact with the robot knocking them off at the time of the buzzer.

Another possibility is that Update 13 came out on Thursday evening of the competition. Perhaps the information was not given to the Head Ref.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by Anne Shade (Post 720162)

There needs to be a way to say that a head ref did a good or bad job and to determine who is not right for the job. Tests and questionaires only get you so far in the evaluation of a person's skills and performance.

Part of that evaluation could be team and key regional personel feedback but most really should be video review or in person evaluation. Why not have the committee of championship head referees review random matches from each regional and evaluate the play calling and the refereeing procedures utilized by the head referee? Head referees would receive feedback on how they can improve and those that are not performing up to par can be replaced.

This is the second part of training. Without the evaluation, the training has very little value.

And how do you know whether that is or is not happening now? (I don't mean to be harsh here. I'm just pointing out that we don't know everything that goes on in the background.)

Head Referees have been replaced in the past.

Re: Call Inconsistencies Between Regionals

Gary,

I have been a referee for the past three years and a head referee the past two. I have not been a head ref this year (started a new team). Video review has not occured in the past. I've spoken to Aidan on this before.

Head Referees have been replaced in the past, that is true. They were replaced when the situation got so bad that FIRST had no choice but to act. We need to diagnose problems before that point. Hence my comment, evaluations are the next step after training.

Re: Call Inconsistencies Between Regionals

chesapeake had several instances where robots crossed lane dividers after the bell sounded for hybrid period ending, but no score awarded, especially the 3rd line.
How is that different from how the ball ends up on the overpass, not when the bell sounds, but where it eventually ends up at rest.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by sumadin (Post 719572)

I agree with Brandon. It does not make any sense, in any way, for there to be different rules for different venues. There is one set of rules - that is the one written in the manual.

I think the adult volunteers need to be better trained about the meaning and the reason for the yellow challenge box. Luckily for us, in SVR, we were never kicked out out of the challenge box by any adult volunteer. If that happens, then perhaps this needs to be emphasized with the adult volunteers.

I wont mention his name, but at Chesapeake, I got a "this isn't VCU remark" when I questioned an inconsistency during inspection and to the same person as he was a referee during matches for a penalty call.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by waialua359 (Post 720374)

It might not make sense, but that is how it is supposed to be called. Team Update 13 clarified this

Quote:

In reference to the <G14> rule change, referees will make the calls as follows:

determine the scores contributed by the Robots based on where they are
when the clock reaches zero.
determine the scores contributed by the Trackballs based on where they are
when the clock reaches zero – unless they are in unrestrained motion (i.e.
not in contact with a Robot) at the time, in which case determine the score
based on when they come to rest

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by waialua359 (Post 720378)

The Chesapeake Regional Planning Committee, along with all the lead positions will be having a formal debriefing. I am happy to pass along your constructive criticism if you send them to me in a brief document. jbeatty@usfirst.org

There is always room for improvement.

My understanding was that questions about anything that happened on the field were to only be addressed by the head referee. If that process was not followed, please include that in your comments.

It was a pleasure and an honor having Team 359 at Chesapeake.

Re: Call Inconsistencies Between Regionals

Wow, I glad I'm not a ref this year.

I have ref in the past and know it takes a great deal of time away from my paying job and family to work at an event. I wasn't paid and did not expect to be.

We must remimber that for the most of the people working these events, are unpaid and do so to the best of their abitilty. This year, in order to increase the quality of the refs, FIRST has tried to balance training with the refs schedule. I believe that overall, the refs have improved over previous years. Are they perfect? No.

To increase the amount of time a ref must be trained for an event is a great idea however, can all the people take the time off work and away from family to do it? If not, do we not use that person? It's hard enough to get people to give up their time without the criticism (constructive and not).

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by Rick Thornbro (Post 720420)

To increase the amount of time a ref must be trained for an event is a great idea however, can all the people take the time off work and away from family to do it? If not, do we not use that person? It's hard enough to get people to give up their time without the criticism (constructive and not).

I agree. I to have a paying job that i must leave to attend a competition, as do many other mentors and people who are NOT volunteering. The point is that if you volunteer to be a referee, expect criticism, plain and simple. I am more than willing to be a referee, but as expressed through this forum, it appears, on the east coast at least, that you need an in to become one.

I think everyone here understands that "they're just volunteers", but they hold a great deal of responsibility. The culmination of a lot of things runs directly through their hands, so shrugging it off as they're trying, lets leave it at that, is not going to get the job done here.

Re: Call Inconsistencies Between Regionals

I think college students are the best bets for referees. Most of team are able to to get to competitions on Thursday and Friday, have a passion for the game, rules, and fairness left over from high school, and have lots of potential to open up doors for the program if they come back over the years. (Thats right, the STUDENTS can open doors for the PROGRAM, not the other way around...)

Instead I've seen these students turned away as they are "unexperienced" or "not the right fit". To this I say: What gives? If college students were not allowed to ref because of these reasons and this thread didn't exist, I may not care so much. But, obviously people have been dissatisfied with the performance so far, so I would like to call this matter to attention.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by Corey Balint (Post 719610)

Also, it is very hard for an adult my age to be able to get a ref position on the Eastern Seaboard. Most regionals here tend to have some bias in the selections.

Four of the Chesapeake Referees were adults your age, all FIRST alums from 4 different teams.

Re: Call Inconsistencies Between Regionals

Detroit had 4 referees in or near college age, and another one not much older.

Re: Call Inconsistencies Between Regionals

Quote:

Originally Posted by RoboMom (Post 720383)

Is there a Silicon Valley Regional Planning Committee contact?