Video Review Needs to Happen Now

[Briansmithtown]

Heres my input on this...

Back in 2014, out last match (match 90) at NYC was.... horrible to say the least. 2 of out alliances failed in the first 3 seconds, and we were the only running robot (you would not want to hear the anger that was behind the alliance wall) During the match, we pushed an opponent bot into our ball, and the ball scored low goal... that caused mass confusion with the field crew... 14 seconds later, we finally got the ball, and we have video of it taking that long... now this is where i flip the table. No, there should not be video reviewing, and you should just accept the results. BUT if there were video reviewing, the rule should be that you get one per regional/district. Now why one? Because there is simply no time. Especially this year when the field reset crew has a lot to do. Yes, me and my team were angry with the results, and did it help make us lose? Sure it did. But over time you just learn to take one on the chin, raise your head up high, and be the better man. You can sorta turn it into a life lesson if you want, by saying its like the real world where a mistake happens, and it doest work out for you at all, but you learn to deal with it and just keep moving on. Because trust me, there have been plenty of times where I wish I could show video and get things right, but you simply just can't. Thats a FIRST lesson.

[Headphones]

I agree that something needs to be done to address situations where the referees miss a score by an alliance. In 2014, we were sent home from two separate events when referees missed an assist. Both times we had video evidence refuting the calls. It's ridiculous to invest hundreds of man-hours, pay thousands of dollars, and then be sent packing by a distracted referee.

[VacioArconte]

While it has its merits, I think that a full video replay system is only going to be yet another stressor placed on event organizers, referees and FIRST.

Initially I was thinking: Why not just change the current rule? Often, teams go to the question box with a complaint about the way a foul was called or match was scored, and they try to argue their case without video evidence. If teams that go to the question box could simply provide the replay of the exact moment of the missed call, would it be too unreasonable for the refs to use that as evidence to overturn a call?

But this only introduces the potential for more delays. The Alamo Regional was a full 2 hours behind schedule, and lord knows how much more time would be lost scrutinizing every detail of a match just played.

One other (smaller) factor: video replay of every match would mean that scores would be delayed 2-3 minutes after a match. I can't see that being well-received by the audience

[EricH]

Originally Posted by Headphones

I agree that something needs to be done to address situations where the referees miss a score by an alliance. In 2014, we were sent home from two separate events when referees missed an assist. Both times we had video evidence refuting the calls. It's ridiculous to invest hundreds of man-hours, pay thousands of dollars, and then be sent packing by a distracted referee.

Just be careful with the inflammatory language there. I agree that it's frustrating, but to call the referees distracted should be very far from the truth, and taken as a little bit of unnecessary "taunting" (for lack of a better word).


That being said: I can't say for sure, but I might be thinking about trying something involving replay at the local offseason out here. Too many variables need to fall into place, though, to make a call one way or the other at this point--including who ends up being the head ref at that event.

[Donut]

Originally Posted by alicen

I'd be curious to see how many people who have volunteered as refs AND also previously been on a drive team would choose to solve this problem that everyone has. I say that because they have the perspective of having calls missed, and being in the position of seeing how it's possible that they can miss a call.

I'll bite. I was a student Coach and Driver for 3 years, mentor Coach for 2, and have reffed the last 2 years (I'll be a Coach at AZ West later this year also). As alicen and EricH noted I'm sure there are a growing number of alumni who have been on both sides.

Despite the missed calls that have happened FIRST actually has made some strides to improve referee calls. The refs do not have to watch Boulder scores at all this year (automated scoring is the best way to not miss scores even if past failures like 2006 make us wary of it) and the training has very detailed questions on difficult scenarios that could happen in match. There was also a referee update after week 1 to put more focus on Outer Works Crossings.

I think video review could help with getting calls correct, but it needs to be used in limited cases to keep events running smooth and to limit its use to fixing errors that are significant and can be reasonably evaluated. Building off of other suggestions in this thread I would propose:

• Each Alliance is allowed one challenge/review in the playoffs. Unlimited challenges will result in reviewing every somewhat close match since no one wants their event/season to end. If the system works it could be expanded to each team having one challenge for all qualification matches also but not more to limit review quantities.
• Head Ref leads the review process and delivers the final call, with other refs who were involved in that call assisting if necessary. The rest of the refs keep the next match getting setup correctly so that minimal time is added to normal field reset. Adding a review process should not require an increase in referee headcount given the difficulty in finding referee volunteers already.
• Only match scoring errors (and penalties that lead to an automatic score) can be reviewed. This year that would mean defense crossings, challenges, scales, autonomous points, G13, G28, and boulders (though I am not sure if allowing review of an automatically scored element is reasonable to review, counting balls/disks scored in a match from a video is time consuming and more prone to error). Fouls are not reviewable as it is not easy to determine what fouls were assessed from a video and many involve a judgement call by a referee who has a better view than a camera or driver in their station will.
• The score or lack of being reviewed must be significant enough to affect the outcome of the match (or an RP being awarded for games like this year). Reviewing whether a crossing was awarded in autonomous or teleop in a 40 point blowout is a waste of time and the implications on ranking tiebreakers are not significant enough to justify the resources for that.
• Video evidence must be indisputable to change a call. The point is to receive credit for an obviously missed score, not debate further a close call that a referee already used their best judgement on (such as barely breaking contact with the Outer Works and Sally Port door).

I'm not sure that anything other than a full field view all match should be allowed, as it would be difficult to determine anything from a video flying all over the field following one robot. I don't have good suggestions on implementing a video system.

Overall I think that a review system would only be helpful in a small handful of circumstances, such as a missed stack last year or an obvious defense crossing this year. A video is not going to give a good view of things like bumpers being barely in or out of the Outer Works for when a crossing was not awarded as complete, and those are the more common scenarios that as a driver I would think "we should have scored for that" but as a referee I call a no score because of a different view on field.

Also I thought I'd throw in that I love Stronghold, the referee jobs are hard this year but this is one of the best games (especially for spectators) I've seen in FRC.

[gblake]

The OP wrote

Originally Posted by patar8746

Please ...

My reply is, "No."

[patar8746]

Originally Posted by PayneTrain

...Don't we all want FIRST, an organization that by its very nature must always be sprinting forward instead of dragging its feet, to constantly be improving and tweaking the status quo to make the program better?

Some disagreements lie in both parties wanting to make the organization better in fundamentally different ways (2 champs), some disagreements lie in two parties wanting some change but have differences in methods (Chairman's feedback). The worst ones, and the ones that get me really fired up, are ones where under an almost instinctive impetus to make a change, the controlling party wants to actively fight against the change (Cali districts).

I think piloting a replay system at an event like IRI and other offseasons should lead FRC down the path to pilot it throughout the season in certain regions or globally, and see if it works by 2champs.

Raise your hand if you think missed calls are not a buzzkill and something no one ever feels bad about.

Stand and be counted if you think field disconnects make FRC better.

Shout from the high horse on which you ride, proclaiming "yes, I love to be in a program that says 'No! We cannot come up with any way of improving a problem that has existed for years! There is not even a problem! Scorekeeping errors enable teams, not disable them!"

I know who some of you are, I just want to see how this line of thinking will be swiftly dragged into the mud. FIRST can be all things to all people. It can be a program that transforms people of all ages and backgrounds. It can always be better.

Well said. We all know no one will stand for those reasons when put the right way. FIRST is about more than the robot, its about inspiring students and always becoming better, whether it be robot capability, gracious professionalism, or running of the competition itself.

Originally Posted by Briansmithtown

Heres my input on this...

Back in 2014, out last match (match 90) at NYC was.... horrible to say the least. 2 of out alliances failed in the first 3 seconds, and we were the only running robot (you would not want to hear the anger that was behind the alliance wall) During the match, we pushed an opponent bot into our ball, and the ball scored low goal... that caused mass confusion with the field crew... 14 seconds later, we finally got the ball, and we have video of it taking that long... now this is where i flip the table. No, there should not be video reviewing, and you should just accept the results. BUT if there were video reviewing, the rule should be that you get one per regional/district. Now why one? Because there is simply no time. Especially this year when the field reset crew has a lot to do. Yes, me and my team were angry with the results, and did it help make us lose? Sure it did. But over time you just learn to take one on the chin, raise your head up high, and be the better man. You can sorta turn it into a life lesson if you want, by saying its like the real world where a mistake happens, and it doest work out for you at all, but you learn to deal with it and just keep moving on. Because trust me, there have been plenty of times where I wish I could show video and get things right, but you simply just can't. Thats a FIRST lesson.

That to me is a lesson in just giving up. Do we want our future engineers to accept defeat when they fail? No, that is not a FIRST lesson. You wished you could show a video, what if you could? We identified a problem, and are working to solve it. Being the better man is not giving up, though solid GP for not getting mad, but while part of being the better man is accepting what you can't change, the other part that is just as important is working at the things you CAN. We can't change results from past regionals but we can do our part to stop it from happening again by implementing a review system. This was never about complaining about the past. Over time, things do get better, often teams get another shot, but there are plenty of Seniors that dont get another chance. You say it can't happen but don't say why. How about you help us try? I'd love to be proven wrong, but at least give it a chance.

See earlier posts concerning the time issues, and proposals have already been made regarding limiting challenges and I fully agree, though I'd like to allow teams to continue to challenge if they get the call right. Losing your ability to challenge for being wrong would be a HUGE incentive for teams to not overuse that option, eliminating time problems resulting from too many reviews.

Originally Posted by VacioArconte

While it has its merits, I think that a full video replay system is only going to be yet another stressor placed on event organizers, referees and FIRST...

...The Alamo Regional was a full 2 hours behind schedule, and lord knows how much more time would be lost scrutinizing every detail of a match just played.

One other (smaller) factor: video replay of every match would mean that scores would be delayed 2-3 minutes after a match. I can't see that being well-received by the audience

Again, video replays would only be called upon in contested situations, and incentives can easily be provided to make sure this privilege is not abused (described above).

Also, whether you're for or against replays, I think we should all work together to test ideas at offseason events. How else can we say with any sort of confidence which option is better? Contact your local offseason coordinators to see how you can help get this together, I know I will.

[patar8746]

Originally Posted by Donut

I'll bite. I was a student Coach and Driver for 3 years, mentor Coach for 2, and have reffed the last 2 years (I'll be a Coach at AZ West later this year also). As alicen and EricH noted I'm sure there are a growing number of alumni who have been on both sides...

...I think video review could help with getting calls correct, but it needs to be used in limited cases to keep events running smooth and to limit its use to fixing errors that are significant and can be reasonably evaluated. Building off of other suggestions in this thread I would propose:

• Each Alliance is allowed one challenge/review in the playoffs. Unlimited challenges will result in reviewing every somewhat close match since no one wants their event/season to end. If the system works it could be expanded to each team having one challenge for all qualification matches also but not more to limit review quantities.
• Head Ref leads the review process and delivers the final call, with other refs who were involved in that call assisting if necessary. The rest of the refs keep the next match getting setup correctly so that minimal time is added to normal field reset. Adding a review process should not require an increase in referee headcount given the difficulty in finding referee volunteers already.
• Only match scoring errors (and penalties that lead to an automatic score) can be reviewed. This year that would mean defense crossings, challenges, scales, autonomous points, G13, G28, and boulders (though I am not sure if allowing review of an automatically scored element is reasonable to review, counting balls/disks scored in a match from a video is time consuming and more prone to error). Fouls are not reviewable as it is not easy to determine what fouls were assessed from a video and many involve a judgement call by a referee who has a better view than a camera or driver in their station will.
• The score or lack of being reviewed must be significant enough to affect the outcome of the match (or an RP being awarded for games like this year). Reviewing whether a crossing was awarded in autonomous or teleop in a 40 point blowout is a waste of time and the implications on ranking tiebreakers are not significant enough to justify the resources for that.
• Video evidence must be indisputable to change a call. The point is to receive credit for an obviously missed score, not debate further a close call that a referee already used their best judgement on (such as barely breaking contact with the Outer Works and Sally Port door)...

Overall I think that a review system would only be helpful in a small handful of circumstances, such as a missed stack last year or an obvious defense crossing this year...

...Also I thought I'd throw in that I love Stronghold, the referee jobs are hard this year but this is one of the best games (especially for spectators) I've seen in FRC.

^^^EXPERIENCED REF AND DRIVER

This ruleset is great, and I think this could be our base rules, minus defining where cameras are to be placed and what video feeds can be used. Some variations to try would be changing how many reviews a team gets (especially allowing them to keep reviewing if they aren't ever wrong), allowing foul reviews, implementing other penalities for baseless video requests, and whether or not to allow review in quals vs elims. Great start.

For camera placement, I'd suggest using existing stream setups to start. Variations could be placing cameras at or as close to bird's eye as possible, and not adding new cameras but allowing fans/media reps to submit video, and referee body cams?

Originally Posted by gblake

My reply is, "No."

No chute door?

[Ryan Dognaux]

Originally Posted by gblake

The OP wrote
My reply is, "No."

Super insightful, thanks Blake. Added a lot to this discussion. /s

[gblake]

Originally Posted by Ryan Dognaux

Super insightful, thanks Blake. Added a lot to this discussion. /s

Folks you can PM me if you care to. A longer post by me would repeat what other experienced people already wrote.

Whatever faith people put in my opinions should simply be added to the "No" column, if anyone is keeping score.

Hopefully this added context explains the purpose of my earlier, succint, no-soapbox, horse-is-already-dead post.

Blake

[patar8746]

Originally Posted by gblake

Folks you can PM me if you care to. A longer post by me would repeat what other experienced people already wrote.

Whatever faith people put in my opinions should simply be added to the "No" column, if anyone is keeping score.

Hopefully this added context explains the purpose of my earlier, succint, no-soapbox, horse-is-already-dead post.

Blake

Horse is both alive and dead until we open the box. We should get this tested and if we're wrong we're wrong, if not FIRST can be made better. Why not take that chance?

[FlyingHedgeHog]

Originally Posted by connoc1

I've been skimming through these posts (there are a lot), and after being on an alliance that lost out on a ranking point today because of one of these types of errors, I would like to hear from any referee of Stronghold so far.

Do the refs simply have too much on their plates to be watching for defense crossings, fouls, and making sure stuff generally doesn't go sour? It would be good to get input on the people actually making the calls as they will have the best insight towards this issue.

Hi there, I was a ref this last week at the PNW Wilsonville district event. Personally I think while being an outerworks ref is challenging, it's still doable. The biggest issue in my experience has been the Sallyport, which opens TOWARDS the referee, such that it is often difficult to determine if a robot began their crossing free of contact with said defense.

That said, there have been many instances at my event alone where teams were in the question box after talking about missed crossings, when many attempts were not valid in the first place. I remember several specific instances where a team either didn't begin the crossing entirely within the neutral zone, or never completed the crossing entirely into the courtyard before backing up and crossing again. While there are certainly missed calls, as with every game, I don't think there are nearly the number of uncalled crossing that teams are asserting there are. Many teams seem to be confused as to the exact definition of a crossing, and it's often hard to see exact robot positioning across the field. As such, teams should make it incredibly clear where they start and end their crossings, to help avoid any problems on both ends.

[gblake]

Originally Posted by patar8746

... Why not take that chance?

For all the reasons other people I agree with have already explained accurately.

[patar8746]

Originally Posted by gblake

For all the reasons other people I agree with have already explained accurately.

I've seen reasons it might not work, but nothing refuting trying the system over the off-season

[svpracer]

Being on drive team this year, the validity of this argument is quite clear... It's not a matter of if but when. We clearly crossed a defense 2 times but the refs didn't see it/ made it unclear if we crossed to we had to waste more time going back through it. This clearly could be solved if an "over head" camera was installed (use the top of either tower). But the case is when to use a video replay. I believe that if the refs themselves are unsure of a call or if a valid question is raised (post match, in the question area) that they can call a match under review and sort it out.