Video Review Needs to Happen Now

[commentingonly]

Originally Posted by bkahl

Any reason you felt the need to make an anonymous account for this thread?

Just curious

Partially because, I don't actually have another account otherwise. (long time reader though.) And partially to abstract away my background, and just focus on the information that I have to provide as a technical professional and my experience as a volunteer.

Originally Posted by frcguy

I'd say it took us around 5-10 minutes to check everything, decide, and edit the scores.

I think your right that the time could be improved, but for something such as a re-scoring a match, you can't do that in less then 3 minutes. and that does not include the time needed to verify that you entered the right scores and everyone agrees. Let alone if you wish to analyze other aspects of the match on video as well.

Originally Posted by wilsonmw04

Wow,
it puts the emphasis on the wrong part of the event: who won. We need to take the best of the sports model, not the entire thing.

I also as a alumni and mentor, have to agree 100% with wilsonmw04. Yes, winning the event is a great experience. As someone who never made it to finals while on a team, I hope that every student has the chance to get there, but FIRST is not about winning, its about learning and spreading STEM, business and professionalism. In college and business, you can't go to your professor, boss, or a client and tell them that they did something wrong and expect them to re evaluate a grade or business decision just cause you have evidence supporting your case. As a free lance software developer, I have lost bids, and when seeing the final product produced by the competition, known that I should have won it, but I can't change the clients mind when they made the decision.

If teams members think that matches are reffed poorly, then those teams should be providing mentors to volunteer as refs so that they have someone that will do a better job, and at that point you might see that its not as easy as you think to make the calls they do.

[Chris is me]

I know this is another Opinion / Hot Take post, and it does echo some themes in the thread, but essentially the problem instant replay solves can be solved in other ways - but instant replay is probably the only way that we as event planning volunteers can make an impact.

The "right" way to solve the problem is in game design. While I respect and understand the many constraints the GDC are under in the game design process (and thus this whole paragraph is easier said than done), there are certainly changes to the game rules that would make refereeing more fair. Removing tasks that are scored subjectively by humans in real time is the #1 change that can be made. Assists in 2014 and crossings in 2016 are two perfect examples. To some extent these get better if the human scoring is dedicated solely to the task and focused on a small area of the field, but in both years this task was spread out over several positions and could happen at the same time in multiple areas. Other areas of rules ambiguity could be tightened up and made either more objective or removed entirely. There is a tendency for the GDC to "patch" holes in the game design with specific and excessively subjective rules to cover for a variety of convoluted situations, and that leads to a lot of these problems.

The thing is though, we have no pull on the GDC, and no opportunity to change how the game rules are written whatsoever. So we can't solve these problems the "right" way. We can ask and hope, that's it. If we want to get more calls right, in games where calls are done like this, instant replay is worth exploring. The way it's been done at various offseasons is great - let's keep trying stuff at offseasons to balance the constraints between volunteer requirements, equipment, rules interactions, etc. What we don't need is an essential moratorium on even considering the slightest change to a broken process from people used to the status quo. Let people try it at off-seasons! If it is truly doom and gloom as it is made out to be that will become obvious very quickly, and if it's not, we learned something.

Nobody in the entire thread wants video replay because they believe refs are incompetent, not trying hard enough, or biased. There is universal recognition of the difficulty of the task of refereeing. This is not a matter of people going "oh, now that I know being a ref is Hard, I won't complain anymore" - because what comfort is that to a team that's season is over on a blatant missed call that everyone can see but no one can change? You guys say "it's not about winning", which is really easy to say if you get to the Championship every year anyway, but the fact is in FRC winning is more than symbolic - it creates the future opportunity to compete and be inspired. As long as qualification is merit based, winning, and getting the calls right, will absolutely, tangibly matter. We should try and get it right.

[Ryan Dognaux]

Originally Posted by gblake

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

GRC had one instance of it being used by a team that wanted to make sure all of the high goal shots were correctly counted. As the next match set up, the head referee sat down with me at the webcast table and we pulled up the previous match video. We re-watched the match and the head referee was able to easily count each ball being scored in the high goal. Afterwards they confirmed it with the team and that was it. There was no delay to the event or anything event-related. Overall I thought it was a success.

Originally Posted by wilsonmw04

Wow,
This is such a bad idea for so many reason, forget the logistics of it: technology required, time and energy, etc; it puts the emphasis on the wrong part of the event: who won. We need to take the best of the sports model, not the entire thing.

At least to your first points - technology and energy required are basically the same if you're already webcasting the event. See my post from another thread on the technology aspect - https://www.chiefdelphi.com/forums/s...0&postcount=30 Don't make a mountain out a mole hill on the technology piece.

[Sperkowsky]

Originally Posted by commentingonly

n college you can't go to your professor and tell them that they did something wrong and expect them to re evaluate a grade just cause you have evidence supporting your case. A

Off topic a bit but this doesn't work out in my mind. Someone in college can correct me if I am wrong but if a professor made a mistake grading a test that caused hundreds of kids to fail and you alerted them of it you are telling me they wouldn't fix it?

[Siri]

Originally Posted by Ryan Dognaux

At least to your first points - technology and energy required are basically the same if you're already webcasting the event. See my post from another thread on the technology aspect - https://www.chiefdelphi.com/forums/s...0&postcount=30 Don't make a mountain out a mole hill on the technology piece.

Seriously. I would very much like to see this implemented as: 1) improve the webcasts to Make It Louder 2) experiment with replay methods behind the scenes, 3) go from there.

We all agree that the system is not perfect. We're likely to disagree about the proper effort to devote to various improvements. But this desire to shoot down any potentialities before even investigating their manifested difficulties is very aggravating. The biggest obstacle to experimentation is the technological investment, which largely piggybacks off other vast improvements that go directly to public FInspirationRST. Then experiment with how much different cameras help, how to integrate and navigate feeds, how to handle FMS and turnaround issues, etc, behind the scenes. Then handle actual implementation and restrictions thereon.

I also take serious issue with the "it's unfair if it's not X, and you definitely can't do X" strawmen. The status quo is unfair; insisting that an improvement become perfectly fair is unreasonable. Calls will still be missed whether replay is automatic for every second of every match, available throughout quals, available only by challenge in elims, available only to the head ref, what have you. The fact that calls will be missed doesn't mean missing fewer in some systematic way is equivalent. (Though you're of course free to argue it's irrelevant to the goal of FIRST, which is at best going to land on agree-to-disagree again.)

Originally Posted by Sperkowsky

Off topic a bit but this doesn't work out in my mind. Someone in college can correct me if I am wrong but if a professor made a mistake grading a test that caused hundreds of kids to fail and you alerted them of it you are telling me they wouldn't fix it?

I've been both a grader and a student in this situation, and it's a black-and-white call, it's far rarer that it won't be changed. And I've of course also corrected my own bosses with evidence; I do not understand this argument. If the evidence is ambiguous they may debate it or ignore you, but if it's a cut-and-dry equivalent of "they crossed the defense"--and there's still a way to correct it--every good boss I've had will. And I certainly don't hold the initial call against them if it was reasonable and they correct it responsibly. In fact, the whole thing is very analogous to the way providing video evidence in any situation should work. And I'm saying this as a veteran referee. I have reservations about replay, but that's certainly not one of them.

[commentingonly]

Originally Posted by Sperkowsky

Off topic a bit but this doesn't work out in my mind. Someone in college can correct me if I am wrong but if a professor made a mistake grading a test that caused hundreds of kids to fail and you alerted them of it you are telling me they wouldn't fix it?

You are both right and wrong. Some professors do, but others don't. I have had more then one professor who, refused to regrade assignments due to their own error. It may seem a little silly, in concept, but often times it relates to needing to regrade the entire class or its a project, where the final grade stands as is and they don't want to make the page. In one instance, the professor did offer to, when my grade was in correctly assigned due to their error, but when regrading they ended at the same score finding other reasons to deduct points. So yes, in some cases, but its not as easy as you would think to get grades to be changed or re evaluated.

[gblake]

Originally Posted by Chris is me

I know this is another Opinion / Hot Take post, and it does echo some themes in the thread, but essentially the problem instant replay solves can be solved in other ways - but instant replay is probably the only way that we as event planning volunteers can make an impact.

The "right" way to solve the problem is in game design. ...

Chris,

I broadly agree with your comments about game design, but I agree far less with the assertion about off-season event hosts being left with (probably) only video replay in their bag of tricks for making things better.

Off-season events can and do change rules every year, and not just the rule about using video. Changes other than video are definitely on the table.

One option that doesn't require changing any game rules would be to simply scrounge up 6 volunteers from the crowd and give them the job of watching one robot apiece (there are details to be dealt with, but you get my point).

[/EDIT]
When I wrote the two paragraphs above, my brain was stuck in a thinking-about-off-season-events/experiments rut; and it just dawned on me that Chris was almost certainly thinking about regular-season events. Doh!

That said, the bigger picture point is that if off-season experiments such as getting teams attending events to supply a small handful of students/adults to do a few low-skill scorekeeping tasks (or rule tweaks, or ...), create a dramatic error reduction during off-season events, the GDC would probably notice (notice both the errors and the simple(ish) way to treat the root cause's symptom).
[/EDIT]

I'm not saying in this post that using video evidence it's bad or good, wise or foolish, etc.

I'm also not saying in this post whether or not I think there is a difference between inspiring someone to try something new, and that person later on being excited or depressed by they way an FRC competition unfolds.

I am saying that I don't think video is the only lever that can be pulled.

Now, I'm going back to waiting for the "hard" data posts.

Blake

PS: The one time I got to spend some time with an FRC GDC, they seemed like nice people . I think they would welcome well-organized feedback from event organizers; especially if it took the form of a video-based, post-mortem of a game's rules.

I'm thinking about the sort of review and analysis that would use a large number of hours of video from multiple events to identify the sorts of calls/rules that are hardest for humans to make/enforce correctly.

That sort of info could definitely influence future games (and treat a cause instead of a symptom), especially if could be put into a simple checklist of things to avoid, or do.

Maybe a pro-video person reading this thread will contact the GDC in order to volunteer to do that for the next 1-3 seasons?

[EricH]

Originally Posted by Sperkowsky

Video replay will probably be used as much as the question box. A box used what? Probably less then 15 times per regional with at least half just being simple questions.

Uh, are you basing that just on your experience reffing an offseason, or do you have some other source? I've seen lines at those boxes in the regular season! I'd put it at a minimum of 1 match in 4, somebody is in the box, average over the event (more during playoffs, mind you).

Volunteer role is simple Livestream role. Almost all FIRST events already have livestreams and there is someone volunteering already unofficially to run it... many times without even being recognized as a real volunteer.

ONLY for equipment operation. If that person makes the call--and makes it wrong--you know that HQ will be swamped with complaints about how instant replay inhales audibly. Any calls resulting from replay need to be made by the ref crew, particularly the Head Ref.


One other data point: I did once review a replay at an offseason--the webcast, to be exact. I won't go into the details, but what I'll term a "integrity of the tournament" question was raised. A quick look at the right place, and the question was answered to the tune of "integrity of tournament not affected". Had to wait a match or so to get the webcast set in playback mode, though.

I'm not against replays, per se. I'm against unnecessary use and excessive use, as well as use without proper equipment (read: without decent video). Basically, what that boils down to is that if the replay is available, it is necessary to use it to confirm a specific non-judgement call, and the team-requested use is kept to a reasonable level, great, provided that it's legal for the event in question.

I also agree with Siri on a very key item: If it is clear that something wasn't done properly, and the mistake is caught, it should be fixed. See also: Question box.

[ratdude747]

Originally Posted by Chris is me

The "right" way to solve the problem is in game design. While I respect and understand the many constraints the GDC are under in the game design process (and thus this whole paragraph is easier said than done), there are certainly changes to the game rules that would make refereeing more fair. Removing tasks that are scored subjectively by humans in real time is the #1 change that can be made. Assists in 2014 and crossings in 2016 are two perfect examples. To some extent these get better if the human scoring is dedicated solely to the task and focused on a small area of the field, but in both years this task was spread out over several positions and could happen at the same time in multiple areas. Other areas of rules ambiguity could be tightened up and made either more objective or removed entirely. There is a tendency for the GDC to "patch" holes in the game design with specific and excessively subjective rules to cover for a variety of convoluted situations, and that leads to a lot of these problems.

Great idea in theory. In practice, I see this as only a partial solution. One cannot assume the field hardware works perfectly. Such as in 2014 with the hot goal indicators not flipping during auto once in a while. The head ref made the call whether or not the flippers worked (and if not, called for a replay of match). That's an automated system with a subjective element; head ref's are only human, and I could see one missing a missed flip (especially if it was only one one side) and the resulting questions after the match from the alliances.

When a field scoring fault is suggested, things become subjective by default. Yes, there are places where game design can reduce the number of subjective calls ref's have to make, but subjective calls, at least by the head ref, are unavoidable.

[EricH]

Originally Posted by ratdude747

When a field scoring fault is suggested, things become subjective by default. Yes, there are places where game design can reduce the number of subjective calls ref's have to make, but subjective calls, at least by the head ref, are unavoidable.

You know, it's really hard for the game designers--and the refs.

If everything is black and white/objective, everybody complains that they got screwed over because there should have been some leeway/room for judgement (or the ref missed the call), and that everything should be a judgement call.

If everything is a judgement/subjective call, everybody complains that they got screwed over because the refs' interpretation of the game rules was bad, and that everything should be black and white.

And if the two are mixed, everybody's complaining that the objective calls should be subjective, and the subjective should be objective, and the refs still miss/misinterpret everything, so the teams are screwed over!



Oh, and then someone is bound to bring up instant replay as the only cure-all (instead of what it actually is, one possible tool in the toolbox full of solutions to issues that some teams don't even realize exist). Cue everybody repeating their statements from above paragraphs, followed by debates as to feasibility/fairness/volunteer POV.

[Ryan Dognaux]

Originally Posted by gblake

Now, I'm going back to waiting for the "hard" data posts.

What would hard data for this subject look like to you? Most implementations will only have anecdotal evidence at this point because video review probably only gets used once or twice during an event (so far.)

[Lil' Lavery]

Originally Posted by Ryan Dognaux

What would hard data for this subject look like to you? Most implementations will only have anecdotal evidence at this point because video review probably only gets used once or twice during an event (so far.)

That lack of usage is a valid data point.

[Ryan Dognaux]

Originally Posted by Lil' Lavery

That lack of usage is a valid data point.

What's interesting is I know some argued it would be used so much that it would slow events down, or that would be a risk at least, correct? So far it seems that is not the case for the few events that have tried it out.

[Lil' Lavery]

Originally Posted by Ryan Dognaux

What's interesting is I know some argued it would be used so much that it would slow events down, or that would be a risk at least, correct? So far it seems that is not the case for the few events that have tried it out.

Have any events actually had an overturned call yet? There was one rescoring that was mentioned, but otherwise no event has had something overturned or adjusted. The primary concern is that most video review overturns would result in replaying matches, and thus the added schedule risk. It's not fair to penalize alliances for their strategic behavior based on the score/field conditions that are presented to them in real time, and then go back and adjust those via video review. For instance, if a team crosses a defense 3 or 4 times to damage it, they can't get the extra time they wasted crossing that defense back. Or if a team makes a call to ensure a capture at ~15seconds rather than scoring an extra couple balls based on their real time scoring, then have that scoring adjusted when you give the other alliance a breach that didn't exist previously. Virtually any case where errors are found in video review should mandate a replayed match. Thus the schedule concern.

[gblake]

Originally Posted by Ryan Dognaux

What would hard data for this subject look like to you? Most implementations will only have anecdotal evidence at this point because video review probably only gets used once or twice during an event (so far.)

TL, DR: A) Ask the right question(s). B) Measure the baseline. C) Understand the relationships. D) Tweak the independent variables. E) Measure the results. F) Decide what changes/(re)allocations, if any, to implement.

- - - - - - - - - - - - - - - - - - - - - -

To prep to answer this, in addition to shooting from the hip , I wanted to refresh my recollection of what has been said so far in the last few months. so, I reviewed this thread, and an adjacent thread. I found these posts I and a few others wrote. There is nothing Earth-shaking in them; but they supply some context.

https://www.chiefdelphi.com/forums/s...0&postcount=64
https://www.chiefdelphi.com/forums/s...0&postcount=72
https://www.chiefdelphi.com/forums/s...&postcount=182
https://www.chiefdelphi.com/forums/s...&postcount=193
https://www.chiefdelphi.com/forums/s...&postcount=207
https://www.chiefdelphi.com/forums/s...&postcount=223
https://www.chiefdelphi.com/forums/s...&postcount=265
https://www.chiefdelphi.com/forums/s...&postcount=268
https://www.chiefdelphi.com/forums/s...5&postcount=11
https://www.chiefdelphi.com/forums/s...2&postcount=13
https://www.chiefdelphi.com/forums/s...2&postcount=17

For me, the outline that follows is the way I would want to approach A) creating a solid understanding of the need (or lack thereof) for adding video to the refs' tools, and B) coming up with a first version of a video system, if developing one is warranted.

The "hard data" would be the results (measurements & statistics) produced by the experiments.

Obviously this is a back of the napkin, discussion-forum-quality sort of an outline - Not even PowerPoint quality yet.

The current system (FIRST) being discussed is a system containing many things, including competition events that contain, at the least, a Playing Field & Game Pieces, the Field Staff (announcers, refs, etc.), the participating Teams, the Match/Game/Robot rules, the Audience, the Matches/Schedule, and the field Computers/Sensors/Software.

We are talking about introducing Video Replays into the FRC (and FTC ...) event part of that FIRST system.

We need to know the pertinent parts of the current baseline system's status/performance, the current system's purpose, and the sensitivity of the system's ability-to-achieve-it's-purpose(s) to changes in the independent variables we are going to adjust.

Some useful metrics might be

• Call accuracy
• Current call challenges outcomes
  • Was a call changed
  • Was the result accurate
• Match outcomes affected by call accuracy and by challenges
• Event outcomes affected by call accuracy and by challenges
• System-purpose outcomes affected by call accuracy and challenges
• Calls (i.e. rules) that could/would/should be affected
  • Perfect video
  • Less-than-perfect video
    • Video usefulness vs equipment performance/placement
    • Equipment purchase costs vs equipment performance
    • Non-equipment-purchase costs: time, labor, maintenance, shipping, training, etc.
  • Video-alternatives (more humans, rule/game changes, anything else?)

In the experiments I would want to

• Compare and contrast off-season events with regular season, and various championships to identify how they are alike, and how they are different; and what effect that has on the results collected in each type of event.
• Compare and contrast multiple locations, times-of-day, levels of human-training, etc. as part of trying each candidate method for accomplishing the event's purposes (and sub-purposes).
  • These locations would include change-nothing "control" events.
• Compare and contrast multiple years' (multiple games') results
• When testing each/any alternative, determine "truth" (the baseline data) by analyzing data (not in real time) collected by a plethora of sensors (these are separate from the sensors/methods that are being tested, and the "truth" is not shared during the event).
• In some circumstances (off-season events?), purposefully stress each alternative by having teams challenge calls in bursts and/or continuously. Do this to stress-test the alternative, not because of actual disagreements.
  • If necessary have teams create difficult-to-assess situations, so that reviewing calls requires more than a trivial glance at the video records or other evidence.

What's above is a quick-and-dirty outline of what I would *want* to do to produce "hard data". After dealing with real-world constraints, thinking a bit more deeply, and getting some preliminary results; I, or whoever, might decide the experiments could be simplified without violating the integrity of the results, or they might add something.

I know there are folks who firmly believe that the need for (or cost of) video reviews is/isn't so obvious, that what I outlined above isn't necessary. I don't disagree that they feel that way. I do say that nothing in this thread so far *proves* that the need does/doesn't exist, and/or that a need would justify the investment (instead of investing in satisfying other needs).

Blake

PS: In the past, I and at least one other person have wished for detailed camera/lens specs and placement info. That would be one example of "hard data", and could be used to answer some important questions; but it's just one part of the bigger picture under the heading of "Video Review Needs to Happen Now".

PPS: Above I have some bullets about identifying which calls could/should/would be affected by reviewing video. Complementing that, I'm not sure whether deciding what the effect of a changed call should be, is part designing each/any experiment (it probably is). Regardless, it is certainly something that would factor into any decisions to introduce (or not) video replays into the system.