Video Review Needs to Happen Now

[gblake]

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

[Cothron Theiss]

Originally Posted by gblake

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

Small anecdote here.
I reffed a small, one-day off season event recently. We were pretty short-staffed, so I was doing field reset, counting defense crossings, and keeping tabs on all the stuff behind one alliance wall. In two cases, I know I might've missed a crossing. In one case, it would have made the difference between a Breach or not for a team that was in the top 8 after the Qualification matches. I would have REALLY liked the chance to request a video replay, because I'm honestly not sure if I made the right call in a situation that would have affected a team's ranking somewhat significantly.

Not exactly a new or groundbreaking sentiment, but at some point, I'd like to ref again. I'd feel more confident reffing if I knew I could access video replays to give every team the results they earn.

[Sperkowsky]

Originally Posted by gblake

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

Not related exactly to video review but it's worth sharing.

Yesterday I tried my hand at reffing at a fairly large off season event. After my experience in that position I realise how easy it is to miss something like a crossing or a pin or a g43 violation. I'm sure I missed atleast 1 crossing yesterday and that wasn't for lack of trying or knowing the rules. On an Frc field there is a ton of stuff going on and despite 1 ref watching courtyard fouls and 1 watching crossings its extremely easy to miss something.

So why not have video review? If I missed something that caused a team to lose I'd feel horrible and if there was a way for my mistakes to be realized before ruining hundreds of kids competition it would be great.

Video review needs to happen. After my experience as a ref I belive that even more. I've been on both sides of the coin now and on both sides I want video review. I don't care if it means I leave the completion an hour later.

[Bryce Clegg]

What if we had video review only for playoffs/eliminations and only the referees can call for video review? This wouldn't slow down the competition too much, and the refs would know exactly what happened. While some argue that "this isn't what FRC is about", I disagree. This allows for teams that should have gone to the next level, which could be the District Championship or World Championship. Those championships allow for much more inspiration than just a regional or district event. Also, the public opinion of teams about certain events would become more positive because they know that there was no error that caused them not to advance to the next level.

Imagine this: there is a finals match like the final Stronghold match we saw this year at an event. The teams are tied, and a referee mistake could allow for a team to advance that maybe shouldn't have.

I believe that this is something we need to experiment with, however the power should not be overused,and would need to be monitored very closely.

[EricH]

Originally Posted by Bryce Clegg

What if we had video review only for playoffs/eliminations and only the referees can call for video review? This wouldn't slow down the competition too much, and the refs would know exactly what happened.

I would make that the Head Referee only, myself--and it's actually easier than you might think to implement, IF you ignore the technological part of the equation. Basically, all the GDC would need to do would be to include video replays in the allowable sources the Head Referee is allowed to consult, for playoffs only, and clarify the "will not review" part of that to only be during qualifications. (The list currently includes GDC members, FIRST personnel, the FTA, and technical staff, per the manual in 5.5.3, just for reference.)

That being said, that's ignoring the technical part--camera placement, clear views, etc. That's been discussed ad nauseum, so I'll just leave the discussion there as solutions exist.

The other half is that by allowing the Head Ref to use it, use is not required. This could lead to variation between events, similar to the classic complaint to the inspectors of "But the Magnolia Regional allowed _______!" Probably some consistent standard would be needed, but I suspect that that would end up in the ref training materials, and thus the teams might not be able to know what it was.

This allows for teams that should have gone to the next level, which could be the District Championship or World Championship. Those championships allow for much more inspiration than just a regional or district event. Also, the public opinion of teams about certain events would become more positive because they know that there was no error that caused them not to advance to the next level.

First question I have is, who determines what teams "should have gone" to the next level? Honestly, I would hope that that's determined by the teams on the field of play--even refs don't like swinging the winner by (debatable) penalty, folks!

Second question stems from major sports. They all use replay, to some extent, so the question now becomes why "bad" calls are still making it through the replay system? Paid refs, one game piece (and however many players), $$$$$ replay system, and calls still get through??? Huh, funny how that happens. How are we sure that no error was made?

Now, there is something that would deal with the public opinion BETTER than instant replay, IMO: Transparency about the call, at least with the affected teams if not the entire event. In the playoffs especially, I would say that if the head referee takes the time to go over why the call was made (or missed) with the teams, it's actually better than a replay--and if there's a score correction that needs to be made, then it needs to be made. The teams might not like the call--at least three of them won't!--but I'd be hopeful that they'd understand why the call was made the way it was made.

Imagine this: there is a finals match like the final Stronghold match we saw this year at an event. The teams are tied, and a referee mistake could allow for a team to advance that maybe shouldn't have.

You'll notice that the only reason for the tie was, in fact, a penalty--and whether that penalty was a good call has been discussed, with the conclusion that there was grounds in the rules for it. Just some food for thought, as that wasn't the only time during the season that a single penalty flipped winner to loser of an elims match. I think I saw that happen two or three times prior to that.


The other part of the problem is that FIRST may just need to do a better job of determining which games need dedicated scorers. 2014 and 2016 didn't have them (at official events--though by the end of 2014 the number of refs had increased to allow some to be scorers). 2015 did. Take a wild guess which games actually needed the scorers? (I'll go on record as saying that having scorers in 2016 really helped at the offseasons I was at, even if they were just recording a referee call.)

[commentingonly]

I would like to reiterate match logistics and explain a few reasons why Video review is out of the question, and should never be considered. I am a field staff volunteer, and have work with a few different teams.

Let us start with event logistics from the point of view of field staff, and refs for that matter. A district event typically has around 36 to 40 teams. For that matter lets actually use an event for logistics. How about FIM Southfield event this year.

The event had 39 teams, and 10 hours worth of time scheduled for qualification matches on the public schedule. Each team was set to play 12 matches each giving you 78 matches. That gives you about 7.5 minutes per match.

Now let’s think about game play. What has to happen for a match? The field needs to be configured, robots need to connect, teams need to be announced, the match needs to be played, the scores need to be submitted, then posted, and robots need to clear the field. How long does this all take? About 7 minutes. Exactly the time allotted to each match, and that’s without having any problems with robots or the field that delay the start of a match, match replays (when needed), field repairs, and any number of other factors that may result in a delay of a match on the part of teams or the field. This 7-minute cycle time is just about the limit. it takes 30 seconds or so to prep the field for team connections, 3 to 4 minutes for the teams to setup robots on the field and connect, match time running about 2.5 minutes, then give the refs 30 seconds to confirm the score and for it to be posted. That adds up to about the 7 minutes depending on how long each of these steps takes. Can we make this faster? not without teams setting up robots faster. Even then, with field configuration, robot connection time and the time it takes for matches to run, and scores to be submitted and posted, the fastest time that matches tend to run is 5 and a half minutes, and that only gets hit at most once an event if that. Matches tend to run between 6.5 and 8 minute cycles.

Where in there do you plan to fit in video replay? Replaying a full match would take 2.5 minutes. Not to mention times to analyze what they are seeing, and re watch parts if needed. Maybe add 30 second to adjust and verify and scores that need to be after that? that gives you about 3 minutes to add onto match cycles. so that scores can be posted, so the field can be configured for the next match. Plus, who is to say that there won't me a few additional minutes of discussion about what’s being watched and if those points were counted. How much time do you really want to add to score review? how much is too much? well if you add the three minutes that we gave above, probably the lower end of the range, that gives us 10 minutes per match, and with 78 matches as at Southfield, that gives you 13 hours of game play. You now have to add 3 more hours on to the first day of qualification matches so that you can get done in time. This means keeping students at the venue as late as 10 PM in the case of Southfield. Some people would say that’s reasonable, but some teams traveling more than an hour to events, that’s unreasonable as an hour back to the school and then time to get home, puts students arriving home at 12 PM. That won't fly with many school districts, and do you really want students, mentors and volunteers running on that little sleep? Well think about it. Southfield's event opened at 8 AM the last day. with an hour bus rides, students need to meet before 7, and wake up at 6 to get there by then, and well we all know they don't go to sleep right away.

Now let’s think more about how this entire replay thing would work anyways. Okay so you record the video from the audience screen right? well what’s that? a full field camera? reasonable, but it can't get everything, can't see close details. Okay so you record all the cameras? Well Most setups have 3. Full field, and one for each alliance. Most of the feeds for the walls did not fully catch the defenses, so you don't get to see the exact defense crossed, or even this year, having the portcullis or drawbridge block the view of part of the camera feed. So you add more camera. that works to solve those problems. say one more for each alliance. covering the parts that the others don't see? Now you end up having to allow for video mixing, feed switching, rewind replay, and watching multiple camera screens. That is extra time in each match review that you want to have. maybe say 30 seconds or a minutes? That’s reasonable since the entire point of this is to not miss any detail right? well we have now added at least another hour onto our matches. Where does that fit into the schedule.

Now well you have your 11-minute cycle times, your cameras, your all ready for match review. What’s the cost? well say $500 for a good camera. so $3000 for all of them. Let’s add $500, for wires tripods, and another $500 for the screens and controls so that the video could be watched. You also need a device to record the video. We can use a Tricaster 40 which is $5,000 on B&H. That says our video replay system starts at around $10,000 once you get everything you need. Well that’s not too bad for a video setup, it’s fairly low end, but probably all you need to watch the matches. And remember that needs to be on 20 fields per week. So that is $200,000 for just your video replay system. Let’s also remember that additional time and cost needs to go into training for use of the system, a volunteer spot to possibly man the computer to help the refs, and time and the resources that are needed to prepare the system and vet the options. Still think it’s reasonable?

Okay you want to argue that not every second of every match needs to be replayed? So, you want to play part of a match, you have to seek through the full footage to find those 10 seconds for review, and maybe you still watch it twice? well the time to find the clip, might be 20 seconds and then 20 seconds to watch it twice. so that's 40 seconds. 20 seconds of thinking about how to proceed (that would be super-fast for most people. Please make a decision in 20 seconds for me). and there we have an entire minute added onto game play. Or maybe refs have their own station to watch matches? Okay fork over an additional $20,000 per field for a more advanced streaming device. And we still have the concept of not watching every second of every match. Arguable if you are going to do video review, then it need to happen on every match, since if you only do it when you think you have missed something, or a team asks a question, then that's unfair, since what if a team did not realize the ref did not count a defense crossing they made? Well why should they not get those points and have another team get them since they were being picky? So that means every match gets watched in full. That’s the only way to make it fair for every team.

So you want to argue the time problem? reasonable. Teams can setup for a match while video review is going on. That works. except, it does not change that hard cutoff that it takes the field itself to pre start and run a match. You need to leave 3 minutes for that. and your field staff (FTA, FTAA, CSA, Scorekeeper) won't have any idea about what robots are having connection problems until the field has been pre started, and then fixing any problems could take a minute or two. so you won't save yourself much time here.

You also need to consider the technical aspect of running the equipment. In an ideal world, nothing goes wrong. But that never happens. Cameras will go offline, break, the video streaming boxes won't work, cables will die, balls will fly out of the field and knock over or break video equipment. What happens when something goes wrong. It’s not fair to give a team extra points for crossing a missed defense in one match, but in the next when it happens again, but the opposing alliance hit the camera making it record the ground during those 20 seconds, none because it "was not on camera?" that's not fair now is it. And don't tell me "it won't happen," because you know that it will, and it will be your team that lost points because of it.

All in all, Yes the idea is great. It works for Football, Soccer, Baseball, or other sports that run for 3 hour long games, since 5 minutes don't matter, it’s just a commercial break. but in the fast passed games of FIRST, it’s not the time and place, and in the end you will be doing more bad then good. Feel free to pick this apart, but take a chance to see why it’s really unpractical for this to work.

[Foster]

Wait. Stop. Lets not do, what will be the third/fourth rehash of this.

A few events said they would offer replay counts for teams. Blake (and I) are looking for the data from those events.

What does the data show?

Commentingonly, you make great points, all of which have been posted before. But I'll give you points for a first post that has that much detail in it. For someone so new to FRC, I think it's very impressive.

Thanks

[commentingonly]

Foster, I am not new to FRC I have volunteered in almost every aspect of events, was a student on a team as a critical part of build and drive, ane I help mentior 4 teams. You must understand that i did not wish to rehash the data, and information. I have read a good deal of this thread maybe missing a few posts, so i know its all be said. I (and i would hope that many other field staff members, refs, event coordinates, and gdc/first staff) cant understand how one would expect to achieve this. The reasons i listed are just a few of those that i could list to explain how this would end badly and become more of a stuggle to implement then it would achieve in the long run.

The data that needs to be analized is how long it would take you to walk around the field, seek through a video, find a clip that you want to watch, verify what you expect, play it for two other people so you can be sure, disguss it, and then make the adjustments. Thats process, it would take atleast 2 minutes to complete. and if your talking about recounting scoring, it takes no less then 2.5 minutes to watch a full match, and for something like this years game? thats absolutely required in order to confirm crossings. So it would garentee adding atleast 3 minutes onto any match cycle you have, and thats without debate over any other penaties or other field related problems.

I will say i do agree that every point matters and that missed points suck, but you can't guarantee that a ref won't get distracted. And it can be a disapointment to students and teams. I personally when i was a student drive team memver was on the receiving end of a few bad calls, but never once did we think the wrong call was made after talking with the head ref.

The only feasable solution, is making refs more accountable for their actions. This could be as simple as adding a scorer or additional refs in future years. But providing video replay would more likely then not creat more problems by allowing refs to say "we will just review this later" and stop watching scores or fouls and forget about what they wanted to review at the end of the match.

[MrTechCenter]

We did not offer teams the chance to challenge via video replay at CCC (frankly, we had so much going on during the planning this year that we couldn't even think about it).

However, in the elimination rounds, there were two instances where we believe the match was scored incorrectly. We went back and looked at it on video to see and, if necessary, recount boulder scores and/or defense crossings. It made a significant difference having the video review.

In Finals Match 3, the final score that was initially posted showed that the blue alliance did not get a breach and, thus, did not earn the bonus points. Once the score went up I (Scorekeeper/FTA) immediately knew that it was wrong, the blue alliance was not given credit for the breach and red had won the match and, thus, the tournament initially. Others who were fieldside weren't so sure if they did get the breach or not. We replayed the stream back on one of the monitors and surely enough, the blue alliance did get the breach and not only that but their autonomous wasn't scored correctly either. The scoring adjustments were made and it put them over the red alliance and they ended up winning the tournament.

Not the best implementation, but it made a huge difference.

[Foster]

Originally Posted by MrTechCenter

Not the best implementation, but it made a huge difference.

Thank you very much for your data point with the detailed description on how it worked out.

Question, how long did it take you to do the replay and make a decision?

and again, we've beaten this horse to death with opinions, still looking for the events that did it and the FACTS around it.

[Sperkowsky]

Originally Posted by commentingonly

snip

While I agree with some of your points, I majorly disagree with some.

First of all with time. You seem to assume every single match will get replayed and frankly for lack of better words that's stupid.
Video replay will probably be used as much as the question box. A box used what? Probably less then 15 times per regional with at least half just being simple questions. Even if we ridiculously assume that every match is replayed in its entirety then lets go to your rebuttal about watching while set up is happening for the next match. You mentioned the match could not be pre started without the score being in and you are correct RIGHT NOW. If FIRST does implement video review I am sure they will make a few tweaks in FMS code and make it so the match can be pre started while video review was happening.

Now for cameras. I think video review could fix 99% of calls with 3 wide angle action cameras(gopros) costing around $900. You also mentioned a tricaster. While I love tricasters it is not needed. Something like a Blackmagic Atem would entirely suffice and you can get a nice one for $1,500. After that, there would need to be a streaming computer which Ill say costs $500 and then another $100 for cables. In the end that is about $3,000 per setup. So to outfit all 20 fields we are talking about $60,000. Now that may seem like a huge number but when you realize that its only 12 teams initial registrations it seems much smaller. Not to mention that setup can FINALLY make it so FIRST can standardize livestreams. Volunteer role is simple Livestream role. Almost all FIRST events already have livestreams and there is someone volunteering already unofficially to run it... many times without even being recognized as a real volunteer.

Now your point about equipment breaking I somewhat understand although your main center camera should be able to see 90% of what is going on, on the field. Not to mention this year is a major outlier. Any other year a single wide angle camera could catch 99.999% of calls.

I like you have done a ton of roles at an FRC competition. This year I have done field reset, been the scorekeeper, reffed, and run regional livestreams. I know video review can happen and I really really hope FIRST wakes up and it makes it happen.

[commentingonly]

Sam, you make some good points. Ones which I did consider. I have actually done work with video recording systems and have setup a similar camera setup to what your talking about.

First you suggest GoPros and an Atem. The Atem is a good piece of hardware, and would do what you need, for video mixing in a production environment. In this case however, you need something better. It only offers 1 video out which works for the audience video and webs stream, but does not allow you to record all 3 of the camera in your setup.

The second point that I would like to make is the GoPros, they are a good example for a camera for this. I personally have used them in this type of environment, and I believe the Mid-Atlantic had done this as well with their video system last year. The GoPros don't hold up in a streaming environment. They easily over heat when run for a long time. Getting them up and running takes a bit of work. They also did not play nicely with a black magic Atem that I used, and required multiple adapters (costing about $200 for each camera) to get them to work reliably.

Third, your video streaming server is severely under budget. A cheep black magic card, to handle HD video recording on one feed, costs around $200. Something to handle 3 or more recorded feeds starts around $300-$500. There goes your budget for that one, even if you get cheep hardware, you are still looking at around $1000 for that setup. You can get cheaper hardware, but I would not expect it to hold up well during full event.

I should also mention about camera setup. I have tried to place camera around the FRC field to effectively cover the full field. Its hard especially this year, and don't forget last year too. Once your stacks got above 4 high, it blocked robots, chutes, human players, and drivers. There was no way to ensure full video coverage over this, or last years, games with only 3 cameras. Also i would not say that its over yet either. With the increased production value of this years game, and the looks that it will be carried over to next year, I think we will see another low visibility game in our near future.

Second, i do agree that my every event gets video replay is a little crazy, but I don't think its out of the question. And yes, the question box is under used right now, since often there is nothing that can be done to re score matches. In my years of first, I would expect that if you watched match replays, then you could re-score about 50% of matches based on the video recordings of them. It happens that a ref is watching a robot for a penalty while another that they are not watching scores points, yeah this can very easily get over looked. I as a field staff member have caught myself focusing on the movements of a single robot for long parts of a match and ignoring the others. The game is exciting and that's really easy to do when your 3 feet away. You said it your self a few posts ago.

Originally Posted by Sperkowsky

I'm sure I missed at least 1 crossing yesterday and that wasn't for lack of trying or knowing the rules. On an Frc field there is a ton of stuff going on and despite 1 ref watching courtyard fouls and 1 watching crossings its extremely easy to miss something.

How many matches would you have wanted to replay at that off season event? Are you telling me that you trusted every score you entered and would have only done match review is someone came to the question box? and even then, are you telling me that one team should get their match re-scored just cause they came to the question box? If you do it then, it should happen always in order to make it fair for all teams.

I do agree with you that its a great idea. Video replay would be a handy tool for refs when scoring matches, but i don't think there is a good implementation that would not make more of a hassle for teams and volunteers. There would be countless delays that this would create and i don't think any event could accurately predict its own running time with this as an added factor.

[EricH]

Originally Posted by Sperkowsky

Video replay will probably be used as much as the question box. A box used what? Probably less then 15 times per regional with at least half just being simple questions.

Uh, are you basing that just on your experience reffing an offseason, or do you have some other source? I've seen lines at those boxes in the regular season! I'd put it at a minimum of 1 match in 4, somebody is in the box, average over the event (more during playoffs, mind you).

Volunteer role is simple Livestream role. Almost all FIRST events already have livestreams and there is someone volunteering already unofficially to run it... many times without even being recognized as a real volunteer.

ONLY for equipment operation. If that person makes the call--and makes it wrong--you know that HQ will be swamped with complaints about how instant replay inhales audibly. Any calls resulting from replay need to be made by the ref crew, particularly the Head Ref.


One other data point: I did once review a replay at an offseason--the webcast, to be exact. I won't go into the details, but what I'll term a "integrity of the tournament" question was raised. A quick look at the right place, and the question was answered to the tune of "integrity of tournament not affected". Had to wait a match or so to get the webcast set in playback mode, though.

I'm not against replays, per se. I'm against unnecessary use and excessive use, as well as use without proper equipment (read: without decent video). Basically, what that boils down to is that if the replay is available, it is necessary to use it to confirm a specific non-judgement call, and the team-requested use is kept to a reasonable level, great, provided that it's legal for the event in question.

I also agree with Siri on a very key item: If it is clear that something wasn't done properly, and the mistake is caught, it should be fixed. See also: Question box.

[ratdude747]

Originally Posted by Chris is me

The "right" way to solve the problem is in game design. While I respect and understand the many constraints the GDC are under in the game design process (and thus this whole paragraph is easier said than done), there are certainly changes to the game rules that would make refereeing more fair. Removing tasks that are scored subjectively by humans in real time is the #1 change that can be made. Assists in 2014 and crossings in 2016 are two perfect examples. To some extent these get better if the human scoring is dedicated solely to the task and focused on a small area of the field, but in both years this task was spread out over several positions and could happen at the same time in multiple areas. Other areas of rules ambiguity could be tightened up and made either more objective or removed entirely. There is a tendency for the GDC to "patch" holes in the game design with specific and excessively subjective rules to cover for a variety of convoluted situations, and that leads to a lot of these problems.

Great idea in theory. In practice, I see this as only a partial solution. One cannot assume the field hardware works perfectly. Such as in 2014 with the hot goal indicators not flipping during auto once in a while. The head ref made the call whether or not the flippers worked (and if not, called for a replay of match). That's an automated system with a subjective element; head ref's are only human, and I could see one missing a missed flip (especially if it was only one one side) and the resulting questions after the match from the alliances.

When a field scoring fault is suggested, things become subjective by default. Yes, there are places where game design can reduce the number of subjective calls ref's have to make, but subjective calls, at least by the head ref, are unavoidable.

[EricH]

Originally Posted by ratdude747

When a field scoring fault is suggested, things become subjective by default. Yes, there are places where game design can reduce the number of subjective calls ref's have to make, but subjective calls, at least by the head ref, are unavoidable.

You know, it's really hard for the game designers--and the refs.

If everything is black and white/objective, everybody complains that they got screwed over because there should have been some leeway/room for judgement (or the ref missed the call), and that everything should be a judgement call.

If everything is a judgement/subjective call, everybody complains that they got screwed over because the refs' interpretation of the game rules was bad, and that everything should be black and white.

And if the two are mixed, everybody's complaining that the objective calls should be subjective, and the subjective should be objective, and the refs still miss/misinterpret everything, so the teams are screwed over!



Oh, and then someone is bound to bring up instant replay as the only cure-all (instead of what it actually is, one possible tool in the toolbox full of solutions to issues that some teams don't even realize exist). Cue everybody repeating their statements from above paragraphs, followed by debates as to feasibility/fairness/volunteer POV.