Video Review Needs to Happen Now

[JohnFogarty]

Originally Posted by Ryan Dognaux

I don't know that I 100% agree with video replay, but people are blowing the level of difficulty WAY out of proportion here. It's 2016, there are some very affordable & simple A/V setups that are possible now.

Here's what it would take to implement a basic level of high quality video review -

- 1 volunteer to man the webcast PC. I get that volunteers are scarce but that's a bad excuse to not do something that will improve events big time.
- 1 GoPro on a tall pole. The one we used at St. Louis this past weekend used a $20 speaker stand, a 7-8 ft. tall PVC pipe, a GoPro and a cell phone charger with a USB cable to give the GoPro power all weekend.
- An HDMI input recorder like the Elegato to allow for recording of the GoPro's view.
- Software to record the stream locally on the PC. We use XSplit because it's so easy to use, but there are other options too. Match files are saved automatically to the PC's hard drive and can be opened immediately after the match ends.

Here's what I envision the process looking like -

1. Each alliance gets one challenge flag during the elimination tournament. The challenge must be issued within 2 minutes of the match ending. Once the match has been challenged, the head referee must watch the match / incident in question.

2. Head referee coordinates with the webcast PC volunteer and pulls up the locally recorded file of the last match. This would literally take a minute to do.

3. Head referee watches the video and based on the evidence shown makes a call to replay the match or let the match stand. Similar to the NFL, the video would need to show overwhelming evidence that the match should be replayed i.e. no close calls.

My opinion - if we want FRC to be represented as a truly competitive sport then we need to present it as most sports are presented. One great example of this is how E-sports have exploded over the past few years. The coverage of online gaming tournaments is incredible and is a model FRC should look to follow. For roughly $1000 in equipment, every event could implement a basic level of coverage that would up the home viewing experience ten fold. There's no reason this same setup couldn't be used for a basic level of video replay. Will it be like the NFL? Of course not. But it has to be better than what we have today - which is nothing. FIRST could easily include the kit I described to travel with the fields from event to event and include a tip sheet on how to set it up. Anyone that can hook up their Xbox to their TV could handle setting it up.

Saying we can't do this because 'it's hard' and 'would take too much effort' is a total cop out. This is FIRST, we're supposed to be doing incredible stuff right? What happened to trying to make it loud - or is that not a thing anymore?

I like what you are proposing a lot and I know exactly what you are referencing when it comes to e-sports as well.

I hope what everyone understands from what I was posting earlier in response to LL was just an idea. I do think that a key volunteer for this position should be created. I'm very adamant about that much.

As dodar has mentioned. I'm looking out for the kids, and not just mine.

Without the competition, FRC wouldnt be nearly as inspirational as it is. It isnt the ultimate goal, but it is a major part to the ultimate goal.

This^

[James1902]

Originally Posted by Ryan Dognaux

I don't know that I 100% agree with video replay, but people are blowing the level of difficulty WAY out of proportion here. It's 2016, there are some very affordable & simple A/V setups that are possible now.

I just wanted to clarify that the "GoPro on a stick" setup wasn't what I was considering cost or labor prohibitive. I was talking about a pure top down shot like John Fogarty was suggesting.

And, I don't want to speak for Sean Lavery, but I believe his point about that particular setup being prone to blind spots on this field still stands.

That being said, let's give it a shot at an offseason! I'm always for more and better webcasts from events and if the side effect is that we may have a way to improve team experiences, i'll take it.

I think the main disagreement we're having here is between people who think a system that will be able to catch most missed calls is too expensive so why bother, and people who think an affordable system would have flaws (blind spots) but is worth trying to improve the experience at least a little.

We should be able to come up with something in between the two that fixes some problems but still bothers everybody equally in it's own way.

Oh and this:

Originally Posted by Ryan Dognaux

if we want FRC to be represented as a truly competitive sport then we need to present it as most sports are presented. One great example of this is how E-sports have exploded over the past few years. The coverage of online gaming tournaments is incredible and is a model FRC should look to follow. For roughly $1000 in equipment, every event could implement a basic level of coverage that would up the home viewing experience ten fold.

is why we started the RoboShow in the first place. I disagree that the GoPro on the stick is the best way to go about it in the long run, but it'd certainly be a start. Would love to talk about getting better coverage to more regionals.

[Ryan Dognaux]

Originally Posted by James1902

And, I don't want to speak for Sean Lavery, but I believe his point about that particular setup being prone to blind spots on this field still stands.

...

I disagree that the GoPro on the stick is the best way to go about it in the long run, but it'd certainly be a start.

Agree that one camera will miss some things, especially with the height of drawbridges and sally ports this year. I would think instant replay would be used for major missed items such as defense crossings, scored boulders and egregious fouls for tipping and such. Maybe the head ref should reserve the right to deny the challenge if it won't affect the outcome over the match overall.

The GoPro definitely isn't on the level of what RoboShow, GameSense, IRI Live, etc. are doing but it's a start. Crawl > walk > run.

[bdaroz]

Originally Posted by EricH

(Snip)
But the answer to your question is both yes and no. Many events try to have one extra ref "on staff" so they can carry on with a full on-field crew if someone has to drop out, or more usually to give refs a break every so often. Some head refs would put that "extra" ref on-field for key matches like playoffs. That "extra" position is now an official one, and with a full crew can be manned and still have a ref taking a break.

Ok, so I take it the number of "regularly-staffed ref positions" is the same between 2015 and 2016 (and my take away from your entire post is that it may have been the same even longer). The fact there is an "extra" ref to allow for breaks who can be pulled in for eliminations is a possibly recent addition that doesn't affect qualifications.

The point I'm inclined to make is that the issue of missed calls (as opposed to flat-out errors) clearly has a "difficulty of game" component to it. Perhaps not a 1:1 correlation, but a strong causation nonetheless. Taking this further, these kinds of situations are, I'd argue to a large degree, a result of the GDC game design, either in referee difficulty, or in failing to allocate more resources (refs/video) to compensate.

Our competition is this coming week, so I have no first-hand experience to base this on, yet... But I understand it can be difficult to find an extra referee, or a volunteer to sit and work video all day for several days. I also have the utmost respect for the referees, and all the volunteers, and the time they invest and volunteer to do what can be a thankless job.

I think, however, with this complicated a game, another set of eyes would have been a wise investment. Perhaps it would be easier to fill a "Video Replay Official" position rather than an additional referee.

[EricH]

Originally Posted by bdaroz

Ok, so I take it the number of "regularly-staffed ref positions" is the same between 2015 and 2016 (and my take away from your entire post is that it may have been the same even longer). The fact there is an "extra" ref to allow for breaks who can be pulled in for eliminations is a possibly recent addition that doesn't affect qualifications.

Read my post again. In 2015, and 2014 Week 1, there were 5 refs on the field. One extra was on staff, for a full crew of 6 refs, and during playoffs there could be 6 on the field depending on the head ref. (2014 kept ramping up the ref count, though--went to 6 on-field later in the season, I was on a crew with 8 on-field at one point; CMP had 10 or so I hear.)

This year, there are 6 refs on field with one ref sitting out (ideally). There is an increase.

The point I'm inclined to make is that the issue of missed calls (as opposed to flat-out errors) clearly has a "difficulty of game" component to it. Perhaps not a 1:1 correlation, but a strong causation nonetheless. Taking this further, these kinds of situations are, I'd argue to a large degree, a result of the GDC game design, either in referee difficulty, or in failing to allocate more resources (refs/video) to compensate.

That's a possibility. Or it's a case of the refs having eyes away from the area of the missed call for some reason. As far as game design... There's one aspect of the game that SOMEBODY made a trivial mistake on that really makes life difficult for the refs; what that is is left as an exercise for the game viewer (with a hint to look at the defenses on the left side of your screen when you're watching the webcasts).

[alicen]

Just a small note to add to all this, but in my experience, people are either mad (to some degree) at the GDC for a boring or broken game, or they're mad at the refs for not calling every perceived penalty.

If a game is exciting, a lot of times that means there is A LOT going on. Given that most of the time you're looking at a team of 4-6 refs + head ref, that's a lot to look at when refs have to do some amount of scoring, while also calling penalties. To make matters worse, those penalties aren't always in the same place. Penalties can occur behind the driver's station, right in front of you, over in the spy box, across the field, heck even after the match!

More to the point of this thread - yes video review is used extensively for all other sports for making the right calls. Think about it this way, the average football game would last according to google:

"An average professional football game lasts 3 hours and 12 minutes, but if you tally up the time when the ball is actually in play, the action amounts to a mere 11 minutes."

11 minutes of play turns into 3 hours and 12 minutes. I'm fairly certain that if we reviewed video every time someone had a complaint or saw something that they felt a ref missed, regionals would take upwards of a week. Or you'd get about 4 matches total out of the whole thing.

I'm not saying the system is perfect and to leave it alone, I'm just saying that instead of kicking and screaming about not getting your way, propose a reasonable solution.

side note to the line in the original post for the thread about ruining the chances of some very deserving students -- were the students on the opposing alliance not deserving? I think everyone is deserving, but please be GP!

[dodar]

Originally Posted by alicen

Just a small note to add to all this, but in my experience, people are either mad (to some degree) at the GDC for a boring or broken game, or they're mad at the refs for not calling every perceived penalty.

If a game is exciting, a lot of times that means there is A LOT going on. Given that most of the time you're looking at a team of 4-6 refs + head ref, that's a lot to look at when refs have to do some amount of scoring, while also calling penalties. To make matters worse, those penalties aren't always in the same place. Penalties can occur behind the driver's station, right in front of you, over in the spy box, across the field, heck even after the match!

More to the point of this thread - yes video review is used extensively for all other sports for making the right calls. Think about it this way, the average football game would last according to google:

"An average professional football game lasts 3 hours and 12 minutes, but if you tally up the time when the ball is actually in play, the action amounts to a mere 11 minutes."

11 minutes of play turns into 3 hours and 12 minutes. I'm fairly certain that if we reviewed video every time someone had a complaint or saw something that they felt a ref missed, regionals would take upwards of a week. Or you'd get about 4 matches total out of the whole thing.

I'm not saying the system is perfect and to leave it alone, I'm just saying that instead of kicking and screaming about not getting your way, propose a reasonable solution.

side note to the line in the original post for the thread about ruining the chances of some very deserving students -- were the students on the opposing alliance not deserving? I think everyone is deserving, but please be GP!

Taking 2-3 minutes of a ref's time during 7 minute field reset wont add any time.

[EricH]

Originally Posted by dodar

Taking 2-3 minutes of a ref's time during 7 minute field reset wont add any time.

You mean a 4:30 field reset. It's a 7-minute cycle (officially).

But, of course, the refs don't have anything else to do like monitor traffic onto and off of the field for safety, check robot starting positions, check ball starting positions, take a quick scan for frame perimeter violations and call teams out to fix position/perimeter issues, direct team members to the question box (or answer questions there, if you happen to be the Head Referee), grab a drink, help take care of problems with the field (or point staff to them), look up the rules from the last match's tough call, circle the zebra herd to finish discussing a call...



Honestly, I'd use the off-field referee, if available and at leisure. If it's confirmed to be a missed call, he/she advises the head ref of what was missed at the next available point in time and--here's the key thing--whether it would have changed the outcome of the match. Missing one crossing in a 60-point blowout? Sorry, folks, not makin' a difference. Missing a 20-second courtyard contact violation in a 2-point match? Yep, that one's going to be reviewed by the head ref for what the action is going to be.

[It'sAScoot]

Say what you have to say and please turn down the passive aggressiveness. We all get that you disagree with video review but there is no reason to be snarky.

[patar8746]

Originally Posted by dodar

Taking 2-3 minutes of a ref's time during 7 minute field reset wont add any time.

Preach.

[Lil' Lavery]

Originally Posted by JohnFogarty

Time delays are a valid concern. I'll use the Orlando Regional as a benchmark for a lot of things I'll talk about here since that was my most recent experience. We were 1.5 hours behind schedule pretty much all the time. Taking up the time of normal refs with this new video system I'm proposing would not work. Yet, I see the solution as simple and I'll provide my solution.

In qualifications if there is a disputed match where a team believes there was something not scored correctly we could have an additional referee who's entire job it was is to review video to sort these problems out. You don't even have to take the time of the normal match refs to do this sort of after-match verification. The video ref could take a look at the camera view footage to determine if the appropriate call was made.

The review time for videos would be kept short if a designated video review ref could not find indisputable evidence that the call was botched then there would be no changes.

In eliminations where the match scores are in my opinion even more critical to maintaining the quality of the event you can follow a similar procedure. Give at max 5 minutes to determine the call.

So, your solution to the time delay is to add additional volunteers? With a potentially different interpretation of the rules, and who quite possibly did not see the event first-hand given that they were reviewing another match?

You're essentially adding another "key volunteer" position. Something that many events already struggle to fill.

Originally Posted by JohnFogarty

A overhead camera like this one. https://youtu.be/PNs40CrPWUk?t=10s
A solution would have to be found. FIRST provides the field, the ref system, etc. This would have to become part of it.

It would very obvious that the camera would have to provide a high enough resolution video of the entire field to be validly able to determine calls.

So, the district system is over then? Because you're not going to find many high school gyms with the scaffolding/catwalk to support an overhead camera like that. Heck, I'd bet more than 50% of regionals would have to find new venues as well, as plenty of college gyms and convention centers wouldn't be able to support that.

Originally Posted by JohnFogarty

Some of those calls that you just mentioned can't even be called consistently by the refs with their own eyes during the course of a match let alone an event. (i.e 15" perimeter rule, crossings, etc.) If you watched that video you'd even see that crossings were not being counted correctly even over the simple defenses.

I've watched plenty of videos. Mistakes will always be made, but mistakes are part of any sport. Not just FRC. Video replay is not a panacea to fix those officiating errors, which is my point. It's a lot of added cost and complexity, for minimal return. There's still plenty of opportunity for blown calls. Anyone who has followed the NHL this season, and its roll out of expanded video review, will attest to that. More people are complaining about the additional video review than any other officiating issue this season.

Originally Posted by JohnFogarty

Blind spots are valid and just like in football where sometimes a call can't be made definitively even with camera angles a call would just have to be left to stand. HOWEVER, if it can be proven with a simple system like the overhead camera like I am proposing than that alone is a drastic improvement.

That "simple solution" isn't viable in the majority of FRC venues. That simple solution doesn't solve anything regarding interactions underneath bumpers. Specifically, that system cannot answer close calls in terms of CROSSINGS this year, because it cannot see underneath the frame/bumpers of a robot to know if its wheels fully cleared the defense.

[dodar]

Originally Posted by Lil' Lavery

So, your solution to the time delay is to add additional volunteers? With a potentially different interpretation of the rules, and who quite possibly did not see the event first-hand given that they were reviewing another match?

You're essentially adding another "key volunteer" position. Something that many events already struggle to fill.




So, the district system is over then? Because you're not going to find many high school gyms with the scaffolding/catwalk to support an overhead camera like that. Heck, I'd bet more than 50% of regionals would have to find new venues as well, as plenty of college gyms and convention centers wouldn't be able to support that.


I've watched plenty of videos. Mistakes will always be made, but mistakes are part of any sport. Not just FRC. Video replay is not a panacea to fix those officiating errors, which is my point. It's a lot of added cost and complexity, for minimal return. There's still plenty of opportunity for blown calls. Anyone who has followed the NHL this season, and its roll out of expanded video review, will attest to that. More people are complaining about the additional video review than any other officiating issue this season.


That "simple solution" isn't viable in the majority of FRC venues. That simple solution doesn't solve anything regarding interactions underneath bumpers. Specifically, that system cannot answer close calls in terms of CROSSINGS this year, because it cannot see underneath the frame/bumpers of a robot to know if its wheels fully cleared the defense.

Im sorry, but this should be re-worded. Teams that lose good chunks of points and/or get eliminated through sometimes blatant missed calls is not "minimal return".

[Lil' Lavery]

Originally Posted by dodar

Im sorry, but this should be re-worded. Teams that lose good chunks of points and/or get eliminated through sometimes blatant missed calls is not "minimal return".

I don't think video review would fix nearly as many of those cases as some people think they would.

And even if they did, it's minimal return. What happens on the field is such a small part of FIRST's mission. Don't take me wrong, I'm a huge fan of watching and participating in FRC events. But the outcome is not what matters. It really isn't.

[MikLast]

Originally Posted by Lil' Lavery

So, the district system is over then? Because you're not going to find many high school gyms with the scaffolding/catwalk to support an overhead camera like that. Heck, I'd bet more than 50% of regionals would have to find new venues as well, as plenty of college gyms and convention centers wouldn't be able to support that.

Ryan had a good way of combating this about 4 posts before yours:

Originally Posted by Ryan Dognaux

I don't know that I 100% agree with video replay, but people are blowing the level of difficulty WAY out of proportion here. It's 2016, there are some very affordable & simple A/V setups that are possible now.

Here's what it would take to implement a basic level of high quality video review -

- 1 volunteer to man the webcast PC. I get that volunteers are scarce but that's a bad excuse to not do something that will improve events big time.
- 1 GoPro on a tall pole. The one we used at St. Louis this past weekend used a $20 speaker stand, a 7-8 ft. tall PVC pipe, a GoPro and a cell phone charger with a USB cable to give the GoPro power all weekend.
- An HDMI input recorder like the Elegato to allow for recording of the GoPro's view.
- Software to record the stream locally on the PC. We use XSplit because it's so easy to use, but there are other options too. Match files are saved automatically to the PC's hard drive and can be opened immediately after the match ends.

Here's what I envision the process looking like -

1. Each alliance gets one challenge flag during the elimination tournament. The challenge must be issued within 2 minutes of the match ending. Once the match has been challenged, the head referee must watch the match / incident in question.

2. Head referee coordinates with the webcast PC volunteer and pulls up the locally recorded file of the last match. This would literally take a minute to do.

3. Head referee watches the video and based on the evidence shown makes a call to replay the match or let the match stand. Similar to the NFL, the video would need to show overwhelming evidence that the match should be replayed i.e. no close calls.

My opinion - if we want FRC to be represented as a truly competitive sport then we need to present it as most sports are presented. One great example of this is how E-sports have exploded over the past few years. The coverage of online gaming tournaments is incredible and is a model FRC should look to follow. For roughly $1000 in equipment, every event could implement a basic level of coverage that would up the home viewing experience ten fold. There's no reason this same setup couldn't be used for a basic level of video replay. Will it be like the NFL? Of course not. But it has to be better than what we have today - which is nothing. FIRST could easily include the kit I described to travel with the fields from event to event and include a tip sheet on how to set it up. Anyone that can hook up their Xbox to their TV could handle setting it up.

Saying we can't do this because 'it's hard' and 'would take too much effort' is a total cop out. This is FIRST, we're supposed to be doing incredible stuff right? What happened to trying to make it loud - or is that not a thing anymore?

[Lil' Lavery]

Originally Posted by MikLast

Ryan had a good way of combating this about 4 posts before yours:

The post I was responding to specifically said a solution like Ryan's was not what he was talking about. I brought up the Fisheye GoPro in my previous response. He had a specific request for an overhead camera like Orlando's.