Video Review Needs to Happen Now

[Cothron Theiss]

Originally Posted by gblake

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

Small anecdote here.
I reffed a small, one-day off season event recently. We were pretty short-staffed, so I was doing field reset, counting defense crossings, and keeping tabs on all the stuff behind one alliance wall. In two cases, I know I might've missed a crossing. In one case, it would have made the difference between a Breach or not for a team that was in the top 8 after the Qualification matches. I would have REALLY liked the chance to request a video replay, because I'm honestly not sure if I made the right call in a situation that would have affected a team's ranking somewhat significantly.

Not exactly a new or groundbreaking sentiment, but at some point, I'd like to ref again. I'd feel more confident reffing if I knew I could access video replays to give every team the results they earn.

[Sperkowsky]

Originally Posted by gblake

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

Not related exactly to video review but it's worth sharing.

Yesterday I tried my hand at reffing at a fairly large off season event. After my experience in that position I realise how easy it is to miss something like a crossing or a pin or a g43 violation. I'm sure I missed atleast 1 crossing yesterday and that wasn't for lack of trying or knowing the rules. On an Frc field there is a ton of stuff going on and despite 1 ref watching courtyard fouls and 1 watching crossings its extremely easy to miss something.

So why not have video review? If I missed something that caused a team to lose I'd feel horrible and if there was a way for my mistakes to be realized before ruining hundreds of kids competition it would be great.

Video review needs to happen. After my experience as a ref I belive that even more. I've been on both sides of the coin now and on both sides I want video review. I don't care if it means I leave the completion an hour later.

[Bryce Clegg]

What if we had video review only for playoffs/eliminations and only the referees can call for video review? This wouldn't slow down the competition too much, and the refs would know exactly what happened. While some argue that "this isn't what FRC is about", I disagree. This allows for teams that should have gone to the next level, which could be the District Championship or World Championship. Those championships allow for much more inspiration than just a regional or district event. Also, the public opinion of teams about certain events would become more positive because they know that there was no error that caused them not to advance to the next level.

Imagine this: there is a finals match like the final Stronghold match we saw this year at an event. The teams are tied, and a referee mistake could allow for a team to advance that maybe shouldn't have.

I believe that this is something we need to experiment with, however the power should not be overused,and would need to be monitored very closely.

[EricH]

Originally Posted by Bryce Clegg

What if we had video review only for playoffs/eliminations and only the referees can call for video review? This wouldn't slow down the competition too much, and the refs would know exactly what happened.

I would make that the Head Referee only, myself--and it's actually easier than you might think to implement, IF you ignore the technological part of the equation. Basically, all the GDC would need to do would be to include video replays in the allowable sources the Head Referee is allowed to consult, for playoffs only, and clarify the "will not review" part of that to only be during qualifications. (The list currently includes GDC members, FIRST personnel, the FTA, and technical staff, per the manual in 5.5.3, just for reference.)

That being said, that's ignoring the technical part--camera placement, clear views, etc. That's been discussed ad nauseum, so I'll just leave the discussion there as solutions exist.

The other half is that by allowing the Head Ref to use it, use is not required. This could lead to variation between events, similar to the classic complaint to the inspectors of "But the Magnolia Regional allowed _______!" Probably some consistent standard would be needed, but I suspect that that would end up in the ref training materials, and thus the teams might not be able to know what it was.

This allows for teams that should have gone to the next level, which could be the District Championship or World Championship. Those championships allow for much more inspiration than just a regional or district event. Also, the public opinion of teams about certain events would become more positive because they know that there was no error that caused them not to advance to the next level.

First question I have is, who determines what teams "should have gone" to the next level? Honestly, I would hope that that's determined by the teams on the field of play--even refs don't like swinging the winner by (debatable) penalty, folks!

Second question stems from major sports. They all use replay, to some extent, so the question now becomes why "bad" calls are still making it through the replay system? Paid refs, one game piece (and however many players), $$$$$ replay system, and calls still get through??? Huh, funny how that happens. How are we sure that no error was made?

Now, there is something that would deal with the public opinion BETTER than instant replay, IMO: Transparency about the call, at least with the affected teams if not the entire event. In the playoffs especially, I would say that if the head referee takes the time to go over why the call was made (or missed) with the teams, it's actually better than a replay--and if there's a score correction that needs to be made, then it needs to be made. The teams might not like the call--at least three of them won't!--but I'd be hopeful that they'd understand why the call was made the way it was made.

Imagine this: there is a finals match like the final Stronghold match we saw this year at an event. The teams are tied, and a referee mistake could allow for a team to advance that maybe shouldn't have.

You'll notice that the only reason for the tie was, in fact, a penalty--and whether that penalty was a good call has been discussed, with the conclusion that there was grounds in the rules for it. Just some food for thought, as that wasn't the only time during the season that a single penalty flipped winner to loser of an elims match. I think I saw that happen two or three times prior to that.


The other part of the problem is that FIRST may just need to do a better job of determining which games need dedicated scorers. 2014 and 2016 didn't have them (at official events--though by the end of 2014 the number of refs had increased to allow some to be scorers). 2015 did. Take a wild guess which games actually needed the scorers? (I'll go on record as saying that having scorers in 2016 really helped at the offseasons I was at, even if they were just recording a referee call.)

[commentingonly]

I would like to reiterate match logistics and explain a few reasons why Video review is out of the question, and should never be considered. I am a field staff volunteer, and have work with a few different teams.

Let us start with event logistics from the point of view of field staff, and refs for that matter. A district event typically has around 36 to 40 teams. For that matter lets actually use an event for logistics. How about FIM Southfield event this year.

The event had 39 teams, and 10 hours worth of time scheduled for qualification matches on the public schedule. Each team was set to play 12 matches each giving you 78 matches. That gives you about 7.5 minutes per match.

Now let’s think about game play. What has to happen for a match? The field needs to be configured, robots need to connect, teams need to be announced, the match needs to be played, the scores need to be submitted, then posted, and robots need to clear the field. How long does this all take? About 7 minutes. Exactly the time allotted to each match, and that’s without having any problems with robots or the field that delay the start of a match, match replays (when needed), field repairs, and any number of other factors that may result in a delay of a match on the part of teams or the field. This 7-minute cycle time is just about the limit. it takes 30 seconds or so to prep the field for team connections, 3 to 4 minutes for the teams to setup robots on the field and connect, match time running about 2.5 minutes, then give the refs 30 seconds to confirm the score and for it to be posted. That adds up to about the 7 minutes depending on how long each of these steps takes. Can we make this faster? not without teams setting up robots faster. Even then, with field configuration, robot connection time and the time it takes for matches to run, and scores to be submitted and posted, the fastest time that matches tend to run is 5 and a half minutes, and that only gets hit at most once an event if that. Matches tend to run between 6.5 and 8 minute cycles.

Where in there do you plan to fit in video replay? Replaying a full match would take 2.5 minutes. Not to mention times to analyze what they are seeing, and re watch parts if needed. Maybe add 30 second to adjust and verify and scores that need to be after that? that gives you about 3 minutes to add onto match cycles. so that scores can be posted, so the field can be configured for the next match. Plus, who is to say that there won't me a few additional minutes of discussion about what’s being watched and if those points were counted. How much time do you really want to add to score review? how much is too much? well if you add the three minutes that we gave above, probably the lower end of the range, that gives us 10 minutes per match, and with 78 matches as at Southfield, that gives you 13 hours of game play. You now have to add 3 more hours on to the first day of qualification matches so that you can get done in time. This means keeping students at the venue as late as 10 PM in the case of Southfield. Some people would say that’s reasonable, but some teams traveling more than an hour to events, that’s unreasonable as an hour back to the school and then time to get home, puts students arriving home at 12 PM. That won't fly with many school districts, and do you really want students, mentors and volunteers running on that little sleep? Well think about it. Southfield's event opened at 8 AM the last day. with an hour bus rides, students need to meet before 7, and wake up at 6 to get there by then, and well we all know they don't go to sleep right away.

Now let’s think more about how this entire replay thing would work anyways. Okay so you record the video from the audience screen right? well what’s that? a full field camera? reasonable, but it can't get everything, can't see close details. Okay so you record all the cameras? Well Most setups have 3. Full field, and one for each alliance. Most of the feeds for the walls did not fully catch the defenses, so you don't get to see the exact defense crossed, or even this year, having the portcullis or drawbridge block the view of part of the camera feed. So you add more camera. that works to solve those problems. say one more for each alliance. covering the parts that the others don't see? Now you end up having to allow for video mixing, feed switching, rewind replay, and watching multiple camera screens. That is extra time in each match review that you want to have. maybe say 30 seconds or a minutes? That’s reasonable since the entire point of this is to not miss any detail right? well we have now added at least another hour onto our matches. Where does that fit into the schedule.

Now well you have your 11-minute cycle times, your cameras, your all ready for match review. What’s the cost? well say $500 for a good camera. so $3000 for all of them. Let’s add $500, for wires tripods, and another $500 for the screens and controls so that the video could be watched. You also need a device to record the video. We can use a Tricaster 40 which is $5,000 on B&H. That says our video replay system starts at around $10,000 once you get everything you need. Well that’s not too bad for a video setup, it’s fairly low end, but probably all you need to watch the matches. And remember that needs to be on 20 fields per week. So that is $200,000 for just your video replay system. Let’s also remember that additional time and cost needs to go into training for use of the system, a volunteer spot to possibly man the computer to help the refs, and time and the resources that are needed to prepare the system and vet the options. Still think it’s reasonable?

Okay you want to argue that not every second of every match needs to be replayed? So, you want to play part of a match, you have to seek through the full footage to find those 10 seconds for review, and maybe you still watch it twice? well the time to find the clip, might be 20 seconds and then 20 seconds to watch it twice. so that's 40 seconds. 20 seconds of thinking about how to proceed (that would be super-fast for most people. Please make a decision in 20 seconds for me). and there we have an entire minute added onto game play. Or maybe refs have their own station to watch matches? Okay fork over an additional $20,000 per field for a more advanced streaming device. And we still have the concept of not watching every second of every match. Arguable if you are going to do video review, then it need to happen on every match, since if you only do it when you think you have missed something, or a team asks a question, then that's unfair, since what if a team did not realize the ref did not count a defense crossing they made? Well why should they not get those points and have another team get them since they were being picky? So that means every match gets watched in full. That’s the only way to make it fair for every team.

So you want to argue the time problem? reasonable. Teams can setup for a match while video review is going on. That works. except, it does not change that hard cutoff that it takes the field itself to pre start and run a match. You need to leave 3 minutes for that. and your field staff (FTA, FTAA, CSA, Scorekeeper) won't have any idea about what robots are having connection problems until the field has been pre started, and then fixing any problems could take a minute or two. so you won't save yourself much time here.

You also need to consider the technical aspect of running the equipment. In an ideal world, nothing goes wrong. But that never happens. Cameras will go offline, break, the video streaming boxes won't work, cables will die, balls will fly out of the field and knock over or break video equipment. What happens when something goes wrong. It’s not fair to give a team extra points for crossing a missed defense in one match, but in the next when it happens again, but the opposing alliance hit the camera making it record the ground during those 20 seconds, none because it "was not on camera?" that's not fair now is it. And don't tell me "it won't happen," because you know that it will, and it will be your team that lost points because of it.

All in all, Yes the idea is great. It works for Football, Soccer, Baseball, or other sports that run for 3 hour long games, since 5 minutes don't matter, it’s just a commercial break. but in the fast passed games of FIRST, it’s not the time and place, and in the end you will be doing more bad then good. Feel free to pick this apart, but take a chance to see why it’s really unpractical for this to work.

[Foster]

Wait. Stop. Lets not do, what will be the third/fourth rehash of this.

A few events said they would offer replay counts for teams. Blake (and I) are looking for the data from those events.

What does the data show?

Commentingonly, you make great points, all of which have been posted before. But I'll give you points for a first post that has that much detail in it. For someone so new to FRC, I think it's very impressive.

Thanks

[commentingonly]

Foster, I am not new to FRC I have volunteered in almost every aspect of events, was a student on a team as a critical part of build and drive, ane I help mentior 4 teams. You must understand that i did not wish to rehash the data, and information. I have read a good deal of this thread maybe missing a few posts, so i know its all be said. I (and i would hope that many other field staff members, refs, event coordinates, and gdc/first staff) cant understand how one would expect to achieve this. The reasons i listed are just a few of those that i could list to explain how this would end badly and become more of a stuggle to implement then it would achieve in the long run.

The data that needs to be analized is how long it would take you to walk around the field, seek through a video, find a clip that you want to watch, verify what you expect, play it for two other people so you can be sure, disguss it, and then make the adjustments. Thats process, it would take atleast 2 minutes to complete. and if your talking about recounting scoring, it takes no less then 2.5 minutes to watch a full match, and for something like this years game? thats absolutely required in order to confirm crossings. So it would garentee adding atleast 3 minutes onto any match cycle you have, and thats without debate over any other penaties or other field related problems.

I will say i do agree that every point matters and that missed points suck, but you can't guarantee that a ref won't get distracted. And it can be a disapointment to students and teams. I personally when i was a student drive team memver was on the receiving end of a few bad calls, but never once did we think the wrong call was made after talking with the head ref.

The only feasable solution, is making refs more accountable for their actions. This could be as simple as adding a scorer or additional refs in future years. But providing video replay would more likely then not creat more problems by allowing refs to say "we will just review this later" and stop watching scores or fouls and forget about what they wanted to review at the end of the match.

[MrTechCenter]

We did not offer teams the chance to challenge via video replay at CCC (frankly, we had so much going on during the planning this year that we couldn't even think about it).

However, in the elimination rounds, there were two instances where we believe the match was scored incorrectly. We went back and looked at it on video to see and, if necessary, recount boulder scores and/or defense crossings. It made a significant difference having the video review.

In Finals Match 3, the final score that was initially posted showed that the blue alliance did not get a breach and, thus, did not earn the bonus points. Once the score went up I (Scorekeeper/FTA) immediately knew that it was wrong, the blue alliance was not given credit for the breach and red had won the match and, thus, the tournament initially. Others who were fieldside weren't so sure if they did get the breach or not. We replayed the stream back on one of the monitors and surely enough, the blue alliance did get the breach and not only that but their autonomous wasn't scored correctly either. The scoring adjustments were made and it put them over the red alliance and they ended up winning the tournament.

Not the best implementation, but it made a huge difference.

[Sperkowsky]

Originally Posted by commentingonly

snip

While I agree with some of your points, I majorly disagree with some.

First of all with time. You seem to assume every single match will get replayed and frankly for lack of better words that's stupid.
Video replay will probably be used as much as the question box. A box used what? Probably less then 15 times per regional with at least half just being simple questions. Even if we ridiculously assume that every match is replayed in its entirety then lets go to your rebuttal about watching while set up is happening for the next match. You mentioned the match could not be pre started without the score being in and you are correct RIGHT NOW. If FIRST does implement video review I am sure they will make a few tweaks in FMS code and make it so the match can be pre started while video review was happening.

Now for cameras. I think video review could fix 99% of calls with 3 wide angle action cameras(gopros) costing around $900. You also mentioned a tricaster. While I love tricasters it is not needed. Something like a Blackmagic Atem would entirely suffice and you can get a nice one for $1,500. After that, there would need to be a streaming computer which Ill say costs $500 and then another $100 for cables. In the end that is about $3,000 per setup. So to outfit all 20 fields we are talking about $60,000. Now that may seem like a huge number but when you realize that its only 12 teams initial registrations it seems much smaller. Not to mention that setup can FINALLY make it so FIRST can standardize livestreams. Volunteer role is simple Livestream role. Almost all FIRST events already have livestreams and there is someone volunteering already unofficially to run it... many times without even being recognized as a real volunteer.

Now your point about equipment breaking I somewhat understand although your main center camera should be able to see 90% of what is going on, on the field. Not to mention this year is a major outlier. Any other year a single wide angle camera could catch 99.999% of calls.

I like you have done a ton of roles at an FRC competition. This year I have done field reset, been the scorekeeper, reffed, and run regional livestreams. I know video review can happen and I really really hope FIRST wakes up and it makes it happen.

[Foster]

Originally Posted by MrTechCenter

Not the best implementation, but it made a huge difference.

Thank you very much for your data point with the detailed description on how it worked out.

Question, how long did it take you to do the replay and make a decision?

and again, we've beaten this horse to death with opinions, still looking for the events that did it and the FACTS around it.

[commentingonly]

Sam, you make some good points. Ones which I did consider. I have actually done work with video recording systems and have setup a similar camera setup to what your talking about.

First you suggest GoPros and an Atem. The Atem is a good piece of hardware, and would do what you need, for video mixing in a production environment. In this case however, you need something better. It only offers 1 video out which works for the audience video and webs stream, but does not allow you to record all 3 of the camera in your setup.

The second point that I would like to make is the GoPros, they are a good example for a camera for this. I personally have used them in this type of environment, and I believe the Mid-Atlantic had done this as well with their video system last year. The GoPros don't hold up in a streaming environment. They easily over heat when run for a long time. Getting them up and running takes a bit of work. They also did not play nicely with a black magic Atem that I used, and required multiple adapters (costing about $200 for each camera) to get them to work reliably.

Third, your video streaming server is severely under budget. A cheep black magic card, to handle HD video recording on one feed, costs around $200. Something to handle 3 or more recorded feeds starts around $300-$500. There goes your budget for that one, even if you get cheep hardware, you are still looking at around $1000 for that setup. You can get cheaper hardware, but I would not expect it to hold up well during full event.

I should also mention about camera setup. I have tried to place camera around the FRC field to effectively cover the full field. Its hard especially this year, and don't forget last year too. Once your stacks got above 4 high, it blocked robots, chutes, human players, and drivers. There was no way to ensure full video coverage over this, or last years, games with only 3 cameras. Also i would not say that its over yet either. With the increased production value of this years game, and the looks that it will be carried over to next year, I think we will see another low visibility game in our near future.

Second, i do agree that my every event gets video replay is a little crazy, but I don't think its out of the question. And yes, the question box is under used right now, since often there is nothing that can be done to re score matches. In my years of first, I would expect that if you watched match replays, then you could re-score about 50% of matches based on the video recordings of them. It happens that a ref is watching a robot for a penalty while another that they are not watching scores points, yeah this can very easily get over looked. I as a field staff member have caught myself focusing on the movements of a single robot for long parts of a match and ignoring the others. The game is exciting and that's really easy to do when your 3 feet away. You said it your self a few posts ago.

Originally Posted by Sperkowsky

I'm sure I missed at least 1 crossing yesterday and that wasn't for lack of trying or knowing the rules. On an Frc field there is a ton of stuff going on and despite 1 ref watching courtyard fouls and 1 watching crossings its extremely easy to miss something.

How many matches would you have wanted to replay at that off season event? Are you telling me that you trusted every score you entered and would have only done match review is someone came to the question box? and even then, are you telling me that one team should get their match re-scored just cause they came to the question box? If you do it then, it should happen always in order to make it fair for all teams.

I do agree with you that its a great idea. Video replay would be a handy tool for refs when scoring matches, but i don't think there is a good implementation that would not make more of a hassle for teams and volunteers. There would be countless delays that this would create and i don't think any event could accurately predict its own running time with this as an added factor.

[bkahl]

Originally Posted by commentingonly

...

Any reason you felt the need to make an anonymous account for this thread?

Just curious

[frcguy]

Originally Posted by Foster

Thank you very much for your data point with the detailed description on how it worked out.

Question, how long did it take you to do the replay and make a decision?

and again, we've beaten this horse to death with opinions, still looking for the events that did it and the FACTS around it.

I'd say it took us around 5-10 minutes to check everything, decide, and edit the scores. That's only because we didn't really have an actual system in place and we were just playing back the webcast. If we actually planned out and came up with a process, dedicated equipment, etc. it would probably take significantly less time.

[wilsonmw04]

Wow,
This is such a bad idea for so many reason, forget the logistics of it: technology required, time and energy, etc; it puts the emphasis on the wrong part of the event: who won. We need to take the best of the sports model, not the entire thing.

[bdaroz]

Another possible data point for you all....

It may not be necessary to re-watch an entire 2 1/2 minute match to determine if an error occurred.

At our off-season event we had a situation where in the last ~15s of our first quarterfinals match our alliance damaged 2 defenses (2 bots nearly simultaneously) and breached before barely making it to the batter. Prior to crossings, the lights on the defenses both indicated one crossing had been completed, but prior to end of match (when they all went off) only one had been scored as damaged. The failure to score the 2nd crossing and breech was the difference between a win and a loss.

We had no replay (or webcast) in place but we sent our student to the ref to object, to no avail. All in all it took almost 3 minutes between quarterfinals matches to have the discussion and return to the queue.

When we reviewed match video later, we only needed to review the final ~30s of the match, enough to see the lights on the defense prior to the crossing and that two crossings occurred, confirming the scoring error.

To those that say we should focus on the positive of the game, and not emphasize winning or losing, I offer this observation. Coming out of the match we lost, the mentors of all three alliance teams attempted to do just that. In fact the closest to success we had was pointing out that we get to play an extra match as a result (knowing we'd win against this alliance). However, until you stand there with ~15-20 students who all know they took a loss due to a clear error, it's difficult to understand the severe lack of "inspiration" that results.

(To be clear, I hold no ill-will or resentment of the refs at the event -- I strongly believe they did their level best within the rules FIRST laid out, and it was a fun competition overall. Also, these options are mine, and I do not speak for my team.)