Video Review Needs to Happen Now

[patar8746]

Please share this, tweet this, etc. FIRST students put too much effort into their robots to be cheated out of their achievements like this anymore.

As much as FIRST has done for me over the years, one of my biggest problems with the organization is the rule that forces Referees to ignore video evidence exposing bad calls made on the field. Just today at the New York City Regional, teams 395, 2869, and 1546 were cheated out of their win in the second quarterfinals match by a failure to count a defense cross that would have earned 5 points for cross as well as a 20 point bonus for a successful breach of defenses. This forced a third match which this alliance, flustered and confused by the flawed results, lost, causing a deserving group of students to miss a chance to play in the semifinal matches. Footage was provided from 2 different angles but not considered as per rules, and the match remained in the wrong team's hands. Even a replay of the match couldn't be considered. We all know from experience how hard all these students worked for this moment, and to lose in an unfairly judged match is an insult to their efforts and could drive them away from their interests in science and technology, the exact opposite of what FIRST stands for. The teams' hopes were destroyed because the rules don't allow Referees to use all available information to admit their mistakes. This is not what FIRST is about and I am ashamed of how this situation wasn't handled. Its probably too late to correct this individual incident, but we can stop this from happening again. FIRST, please change this policy against video reviews, and referees, please at least consider replays of the whole match. Some already do this, but it needs be standardized to stop this from happening again. I'm open to discussion here, because this is something that needs to be talked about, but please spread the word.

I've heard counterarguments that this would prolong regional events and video could be unreliable. These are no excuses to discredit the effort of these students, as I witnessed firsthand Referees spending more time refusing to watch video than it would have taken to watch the clips 3 times over. In addition, professional sports have already solved the video reliability issue by necessitating that the calls be overturned beyond any reasonable doubt by the evidence, and it's worked well. This can happen, and I hope as a community we can come together to make sure matches are fair and teams earn the credit they deserve.

Note regarding edits: the original edit of this post contained content unfairly judging the way the Referees handled this situation that I sincerely regret ever writing, and I sincerely apologize for any damage I may have caused. For every one mistake made, volunteer referees have made thousands upon thousands of fair calls and correct decisions, without which FIRST wouldn't be able to exist. Thank you to all involved in this tough rule writing and enforcing process for taking the time to help students despite harsh negative feedback and disrespect. The few and far between problems have to this point been handled as well as they could have been, but I hope to be able to help further improve the process for teams in the future to have the best experiences they can.

[Gregor]

Regardless of your feeling towards the rules, the referees acted fully within the rules with regards to video rules and replays, so don't express your anger towards them.

Referee mistakes suck (hello 2014), and they happen, and I'm not going to tell you to get over it, but don't hate the player, hate the game.

[patar8746]

My apologies, I was rather angry when I wrote that and attacked the wrong thing. Edited to reflect the real problem in the rule.

Refs, thanks for volunteering, regardless of calls events don't run without you guys at all.

[EmileH]

I agree with Gregor on this one, but also, I'd shoot an email to frcteams@firstinspires.org regarding your concern. I think it's a good idea for refs to accept video evidence (many a time have bad referee calls been detrimental to my own team's performance).

[Lil' Lavery]

I strongly disagree. Video review opens a can of worms that should be left closed. Beyond simple time delays, it poses concerns regarding what source of video reviews should be allowed, how many videos should be allowed, what level of evidence is needed, what is the timing/procedure for calling video reviews*, how many video reviews are allowed, etc.

*Anyone who's been an alliance captain before already knows the confusion and heartburn of figuring out timeout and back-up coupon timings.

Video review in sports has been anything but a panacea, and they have countless more camera angles and video analysis tools than will be at the disposal of referees in FRC.

[JohnFogarty]

Originally Posted by Lil' Lavery

I strongly disagree. Video review opens a can of worms that should be left closed. Beyond simple time delays, it poses concerns regarding what source of video reviews should be allowed, how many videos should be allowed, what level of evidence is needed, what is the timing/procedure for calling video reviews*, how many video reviews are allowed, etc.

*Anyone who's been an alliance captain before already knows the confusion and heartburn of figuring out timeout and back-up coupon timings.

Video review in sports has been anything but a panacea, and they have countless more camera angles and video analysis tools than will be at the disposal of referees in FRC.

Here's where I start to get rustled.

A simple overhead view of the field can be used to solve all disputes in FRC.
There is very little room to argue with this. FIRST could implement the system themselves as its not that complicated of a system to put in place.

As to who's video? Obviously the only video that would be reviewed would be from FIRST's own camera system. There would be no 3rd party camera footage viewing allowed, similarly to how things are now.

This isn't sports. In FRC the game pieces are simple, the rules are simple, and the problem is no where as complicated as you think it is.

My perspective: I've been a Head Ref for FTC events for several years now.

[Lil' Lavery]

Originally Posted by JohnFogarty

Here's where I start to get rustled.

A simple overhead view of the field can be used to solve all disputes in FRC.
There is very little room to argue with this. FIRST could implement the system themselves as its not that Complicated of a system to put in place.

As to who's video? Obviously the only video that would be reviewed would be from FIRST's own camera system. There would be no 3rd party camera footage viewing allowed, similarly to how things are now.

This isn't sports. In FRC the game pieces are simple, the rules are simple, and the problem is no where as complicated as you think it is.

I call BS.

A simple overhead camera? I'm assuming you mean similar to the fisheye cameras used by FiM/MAR/Indiana. Because if you mean a true overhead camera, that's simply not a realistic option at the vast majority of FRC venues (and certainly not an option without additional rigging costs on the venue side). And if you mean some other static camera (or even a true overhead), there are still plenty of disputes that will not be able to solve. Was a robot's wheel touching the outerworks at the end of autonomous, or just their bumper overhanging? Did their mechanism extend more than 15" beyond the frame perimeter? Did their robot fully stop contacting the drawbridge door momentarily? What was the game clock at that second during the overhead view? Heck, I'm watching the Tippecanoe stream right now, and there are still quite literal blind spots behind easy drawbridge/portcullis (as well as smaller ones behind the towers). No single camera is going to solve all FRC disputes.

Nor did you address any of the other questions I raised.


e; To demonstrate my point, I just captured this from the NYC webstream. While the lack of quality is due to stream compression, the general point still stands. This is what typically constitutes an overhead view in FRC. Did the robot in the orange circle cross the defense? Commit a penalty?

[JohnFogarty]

You'd like me to go through every part of your argument? I have loads of time today.

Originally Posted by Lil' Lavery

Video review opens a can of worms that should be left closed. Beyond simple time delays, it poses....what level of evidence is needed, what is the timing/procedure for calling video reviews*, how many video reviews are allowed

Time delays are a valid concern. I'll use the Orlando Regional as a benchmark for a lot of things I'll talk about here since that was my most recent experience. We were 1.5 hours behind schedule pretty much all the time. Taking up the time of normal refs with this new video system I'm proposing would not work. Yet, I see the solution as simple and I'll provide my solution.

In qualifications if there is a disputed match where a team believes there was something not scored correctly we could have an additional referee who's entire job it was is to review video to sort these problems out. You don't even have to take the time of the normal match refs to do this sort of after-match verification. The video ref could take a look at the camera view footage to determine if the appropriate call was made.

The review time for videos would be kept short if a designated video review ref could not find indisputable evidence that the call was botched then there would be no changes.

In eliminations where the match scores are in my opinion even more critical to maintaining the quality of the event you can follow a similar procedure. Give at max 5 minutes to determine the call.

Originally Posted by Lil' Lavery

concerns regarding what source of video reviews should be allowed, how many videos should be allowed

As I said previously. The only video that would be reviewed would be that from an official FIRST mandated system.

Originally Posted by Lil' Lavery

A simple overhead camera? I'm assuming you mean similar to the fisheye cameras used by FiM/MAR/Indiana. Because if you mean a true overhead camera, that's simply not a realistic option at the vast majority of FRC venues (and certainly not an option without additional rigging costs on the venue side). And if you mean some other static camera (or even a true overhead), there are still plenty of disputes that will not be able to solve. Was a robot's wheel touching the outerworks at the end of autonomous, or just their bumper overhanging? Did their mechanism extend more than 15" beyond the frame perimeter? Did their robot fully stop contacting the drawbridge door momentarily? What was the game clock at that second during the overhead view? Heck, I'm watching the Tippecanoe stream right now, and there are still quite literal blind spots behind easy drawbridge/portcullis (as well as smaller ones behind the towers). No single camera is going to solve all FRC disputes.


e; To demonstrate my point, I just captured this from the NYC webstream. While the lack of quality is due to stream compression, the general point still stands. This is what typically constitutes an overhead view in FRC. Did the robot in the orange circle cross the defense? Commit a penalty?

A overhead camera like this one. https://youtu.be/PNs40CrPWUk?t=10s
A solution would have to be found. FIRST provides the field, the ref system, etc. This would have to become part of it.

It would very obvious that the camera would have to provide a high enough resolution video of the entire field to be validly able to determine calls.

Some of those calls that you just mentioned can't even be called consistently by the refs with their own eyes during the course of a match let alone an event. (i.e 15" perimeter rule, crossings, etc.) If you watched that video you'd even see that crossings were not being counted correctly even over the simple defenses.

Blind spots are valid and just like in football where sometimes a call can't be made definitively even with camera angles a call would just have to be left to stand. HOWEVER, if it can be proven with a simple system like the overhead camera like I am proposing than that alone is a drastic improvement.

[EricH]

John, I'm assuming that you're volunteering as the video referee for any and all events you're attending. If you're not attending one event per week, you're not attending enough events. You just added one volunteer--in an extraordinarily tough position--that will need training. I don't see anybody volunteering for that one. We've got enough problems finding referees as it is.

Unless you're volunteering to head out to some of the areas that can't find refs and give them a hand, of course.

[dodar]

Originally Posted by EricH

John, I'm assuming that you're volunteering as the video referee for any and all events you're attending. If you're not attending one event per week, you're not attending enough events. You just added one volunteer--in an extraordinarily tough position--that will need training. I don't see anybody volunteering for that one. We've got enough problems finding referees as it is.

Unless you're volunteering to head out to some of the areas that can't find refs and give them a hand, of course.

Why does it need to be a ref? All it needs to be is someone who knows the game. They wont be looking for anything unless a team came forward with a specific request. And even then the team could point out the exact moment they believed something happened. The person doesnt need to be a true referee.

[EricH]

Originally Posted by dodar

Why does it need to be a ref? All it needs to be is someone who knows the game. They wont be looking for anything unless a team came forward with a specific request. And even then the team could point out the exact moment they believed something happened. The person doesnt need to be a true referee.

Because the vast majority of the calls that would be made are judgement calls. FIRST seems to have made the determination that only referees can provide that judgement in some years (2014 and 2016 come to mind). I can't say I'd trust someone who just "knows the game" with determining whether team X crossed a defense--there is a very specific definition of Crossing, and that very specific definition has led to numerous teams not quite meeting it and then complaining about it not being called. Just trust me on that.

If you don't have the training of a referee, why would you try to make a referee's call?

[Kevin Leonard]

So here's a situation that occurred last year:

2015 Tech Valley Regional, 20-5254-3624 had an incredible match in a make-it-or-break-it 2nd quarterfinal match, placing 4 stacks total for the first time. When the scores came up, it had our alliance with something like 119 points, and as we analyzed the score, we realized they had only credited us for 3 stacks!

We sent some students to the question box, and the referees came together and discussed that they did remember us having 4 stacks up.

From my understanding of the situation (I was not in the question box nor in the referee's discussion), they then looked at video provided by two different teams that showed 4 stacks built 5-6 high and ended up reversing the call of that match, which ended up allowing us to move on to the semifinals.

Tech Valley is a generally relaxed event, with some great referees and teams who are always gracious, and I don't think anyone involved thought what the referees did was unfair. Am I wrong?

[Rangel(kf7fdb)]

Originally Posted by EricH

Because the vast majority of the calls that would be made are judgement calls. FIRST seems to have made the determination that only referees can provide that judgement in some years (2014 and 2016 come to mind). I can't say I'd trust someone who just "knows the game" with determining whether team X crossed a defense--there is a very specific definition of Crossing, and that very specific definition has led to numerous teams not quite meeting it and then complaining about it not being called. Just trust me on that.

If you don't have the training of a referee, why would you try to make a referee's call?

The person running it simply has to go to the spot where the student says something was missed or penalty was called wrong and bring a ref over to look at it real quick. Could honestly be in between matches too and only if a student has a valid complaint and a general time in the match in which it happened. And hey if the videos are all saved too then now we have videos of every match at every regional and we solved that problem too! Idk maybe that's the real reason I would like to see this happen.

[JohnFogarty]

Originally Posted by EricH

John, I'm assuming that you're volunteering as the video referee for any and all events you're attending. If you're not attending one event per week, you're not attending enough events. You just added one volunteer--in an extraordinarily tough position--that will need training. I don't see anybody volunteering for that one. We've got enough problems finding referees as it is.

Unless you're volunteering to head out to some of the areas that can't find refs and give them a hand, of course.

Sure. If that's what it takes I'd volunteer. I'll make the whole system if I have to.

[EmileH]

Also, one thing with FRC that is different than traditional sports is that the game changes every year. Let that sink in for a moment. In traditional sports, let's say, soccer for example, the game never changes. Albeit maybe a few minor rules (I know the offside rule changes frequently) but nowhere near the frequency that new FRC games get introduced. This essentially means that at this time in the season (week 2), any FRC referee can only have 2-3 months of experience and by champs they could have 4-5 months (maybe) of referee experience, whereas traditional sports referees have 20 to 30 years playing the game with only minor variations.

May I add that FRC games are WAY more complicated than traditional sports games. I could argue this but I do not have the energy nor the full keyboard to use.