Video Review Needs to Happen Now

[Ryan Dognaux]

Originally Posted by jajabinx124

+1

Best way to find out whether it is practical or not.

Agree entirely. We will do this at the off-season event I run here in St. Louis this year. We're already doing the "hard part" - recording 1080P full field video in real-time. The easy part is opening the video file and just watching it again. I don't see a reason not to try this out.

[MrTechCenter]

What if reviews were only allowed for SCORING disputes and not fouls/no-calls? I was quite shocked by how many matches were just simply scored incorrectly this weekend. Most of the time, the outcome of the match wouldn't have changed in these instances, but we shouldn't be giving teams points that they didn't earn. It's just wrong. I saw a few instances of the referees crediting an alliance with three robots on the batter at the end lf the match when the third robot on that alliance was on the complete opposite side of the outer works (I'm not even kidding, this actually happened more than once). I can give tge referees some slack on foul calls because most of those are based on judgement but how can you even mess up the match score like that?

[dodar]

Originally Posted by MrTechCenter

What if reviews were only allowed for SCORING disputes and not fouls/no-calls? I was quite shocked by how many matches were just simply scored incorrectly this weekend. Most of the time, the outcome of the match wouldn't have changed in these instances, but we shouldn't be giving teams points that they didn't earn. It's just wrong. I saw a few instances of the referees crediting an alliance with three robots on the batter at the end lf the match when the third robot on that alliance was on the complete opposite side of the outer works (I'm not even kidding, this actually happened more than once). I can give tge referees some slack on foul calls because most of those are based on judgement but how can you even mess up the match score like that?

But fouls are a part of scores. So they would have to be reviewable too.

[rich2202]

I apologize for not reading 8 pages of posts, but reviewing videos is a bad idea.

The Original Poster brought up a valid point: They potentially lost 25 points and the match.

What about other fouls that were not called against their alliance? What about crossings for the other alliance the Ref's may have missed? Did the robot really start fully in the Neutral Zone and fully cross into the Courtyard?

Is this going to be like the NFL where each team gets to identify a play for review (throw an instant replay flag)? How long will the have to identify the point in the game? What is the penalty if the original ruling is upheld?

Not all fouls are on the field itself. What about cameras in the Castle area to enforce the 6 boulder limit? No touching across the line during Autonomous?

That Team delayed the match 2 minutes 37 seconds, and should have been called for delay. Robot broke contact with the Sally Port for 0.2 seconds and should have been credited with a cross.

The only source of instant replay video is FRC Cameras only, and one field camera from above won't be sufficient for all the rules. You can't allow an opportune video by a bystander, because that would be an unfair advantage. One team could have dozens of cameras, and only show the video when it helps their case.

The Ref's make the call which rule is enforced when. It happens real-time, and that is a major consistency of the enforcement of the rules between matches.

[patar8746]

Originally Posted by MrTechCenter

What if reviews were only allowed for SCORING disputes and not fouls/no-calls? I was quite shocked by how many matches were just simply scored incorrectly this weekend. Most of the time, the outcome of the match wouldn't have changed in these instances, but we shouldn't be giving teams points that they didn't earn. It's just wrong. I saw a few instances of the referees crediting an alliance with three robots on the batter at the end lf the match when the third robot on that alliance was on the complete opposite side of the outer works (I'm not even kidding, this actually happened more than once). I can give tge referees some slack on foul calls because most of those are based on judgement but how can you even mess up the match score like that?

Originally Posted by dodar

But fouls are a part of scores. So they would have to be reviewable too.

I'd say the best way to figure this out is to establish clear communication between Offseason Event Organizers dedicated enough to implement review systems, and intentionally use different rulesets at each, preferably picking one set to be the "control" (not sure what to use, thoughts on this?) changing one aspect we're not sure about between each event, so we'll figure out what works better in each case that way. We're already experimenting with doing video review in general, why not try to learn as much as we can?

[patar8746]

Originally Posted by rich2202

I apologize for not reading 8 pages of posts, but

Stop right there. Please read the posts, as I'm pretty sure your concerns were addressed in previous posts. However, a few of the concerns you're citing are things none of us know how to answer because FRC has never had official video review. These are things that I think we should test at offseasons.

Can't take you offseason organizers for granted but knowing the community, I have faith that there will be people that put in more effort than I can imagine to get this done for the benefit of future students. I know I'm going to do all I can, but this is going to take a community effort.

[MrTechCenter]

Originally Posted by patar8746

I'd say the best way to figure this out is to establish clear communication between Offseason Event Organizers dedicated enough to implement review systems, and intentionally use different rulesets at each, preferably picking one set to be the "control" (not sure what to use, thoughts on this?) changing one aspect we're not sure about between each event, so we'll figure out what works better in each case that way. We're already experimenting with doing video review in general, why not try to learn as much as we can?

As an offseason event organizer (more of a planning comittee member now), I would totally have us be the lab rat for a match review system and I'm sure the rest of the committee would be as well. The only challenge would be finding a head referee willing to do it.

[patar8746]

Originally Posted by MrTechCenter

As an offseason event organizer (more of a planning comittee member now), I would totally have us be the lab rat for a match review system and I'm sure the rest of the committee would be as well. The only challenge would be finding a head referee willing to do it.

^^^ Well we've got one (THANK YOU SIR), any more takers? The more events that are willing to test ideas, the more ideas that get to be tested/the more general examples of using video review the community has to show FIRST. Anybody know people? IRI? Chezy Champs? Suffield Shakedown? Week 0s? R2OC? Between Preseason and Offseason, Blue Alliance shows that for the 2015 season there were 61 events. If less than half of that number of events offered to try video we'd have more than enough data to work with.

[rich2202]

Originally Posted by patar8746

Stop right there. Please read the posts, as I'm pretty sure your concerns were addressed in previous posts.

Ok, I read through all the posts.

First of all, one camera can only solve one problem, and only partially. To be a "fair" Instant Replay (IR) system, it needs to handle most problems. That, one camera cannot solve (i.e. how many boulders are behind the castle wall).

You don't necessarily need one camera on a catwalk. There could be multiple cameras mounted on each castle wall (with at least 2 being really high up). While that would be more feasible for more events, it increases the complexity of the IR system. Don't forget, you also need to game time stamp each frame of the footage so you can know if the potential infraction happened at 21 seconds prior to the end of the game, or at 20 seconds.

That said, I think you might need a camera following each robot for some of the trickier calls where a general field camera would not provide enough detail. In Major League sports, that is easier when you really need to follow only one ball.

While limiting IR to Eliminations, and only potentially game changing calls, in close games, every call is potentially game changing (114 vs 110, when each foul is 5 points). What about Foul Cards? Do you allow IR for Foul Cards (both earned it, and it wasn't given; or it was given, but did they really earn it?). If a team has a Card, it can really affect how they play subsequent matches (more than just a game changer).

IMHO, only a Ref can be the IR initial reviewer. They have to know the game rules in detail (what constitutes a valid crossing).

I was chatting with a Ref at the KC regional this weekend. He said the Outer Works Ref was the most challenging position (the position that watches crossings). IMHO, the genesis of the problem is the Boulder Rules. Not only is the Ref watching crossings of the Robots, but they also have to watch crossings of the Boulders. A lot of Boulders illegally crossed. With the complexity of the Boulder rules, details is everything. That's when I joked on another thread that they will need 18 Ref's, one to watch each Boulder, and remember where it came from.

A few years ago, there were 2 Ref's at each panel (vs. last year when there was a Ref and a Field Reset person to help count totes). IMHO, this year, there should be 2 refs at each panel, at least for the Outer Works Position.

IMHO, increasing the number of Ref's is easier than IR. If you reduce the pressure on an individual Ref, maybe more people would be willing to be a Ref.

Remember: This is High School Competition. How many High School Football games have Instant Replay?

[James1902]

Originally Posted by Ryan Dognaux

I don't know that I 100% agree with video replay, but people are blowing the level of difficulty WAY out of proportion here. It's 2016, there are some very affordable & simple A/V setups that are possible now.

I just wanted to clarify that the "GoPro on a stick" setup wasn't what I was considering cost or labor prohibitive. I was talking about a pure top down shot like John Fogarty was suggesting.

And, I don't want to speak for Sean Lavery, but I believe his point about that particular setup being prone to blind spots on this field still stands.

That being said, let's give it a shot at an offseason! I'm always for more and better webcasts from events and if the side effect is that we may have a way to improve team experiences, i'll take it.

I think the main disagreement we're having here is between people who think a system that will be able to catch most missed calls is too expensive so why bother, and people who think an affordable system would have flaws (blind spots) but is worth trying to improve the experience at least a little.

We should be able to come up with something in between the two that fixes some problems but still bothers everybody equally in it's own way.

Oh and this:

Originally Posted by Ryan Dognaux

if we want FRC to be represented as a truly competitive sport then we need to present it as most sports are presented. One great example of this is how E-sports have exploded over the past few years. The coverage of online gaming tournaments is incredible and is a model FRC should look to follow. For roughly $1000 in equipment, every event could implement a basic level of coverage that would up the home viewing experience ten fold.

is why we started the RoboShow in the first place. I disagree that the GoPro on the stick is the best way to go about it in the long run, but it'd certainly be a start. Would love to talk about getting better coverage to more regionals.

[connoc1]

I've been skimming through these posts (there are a lot), and after being on an alliance that lost out on a ranking point today because of one of these types of errors, I would like to hear from any referee of Stronghold so far.

Do the refs simply have too much on their plates to be watching for defense crossings, fouls, and making sure stuff generally doesn't go sour? It would be good to get input on the people actually making the calls as they will have the best insight towards this issue.

[Ryan Dognaux]

Originally Posted by James1902

And, I don't want to speak for Sean Lavery, but I believe his point about that particular setup being prone to blind spots on this field still stands.

...

I disagree that the GoPro on the stick is the best way to go about it in the long run, but it'd certainly be a start.

Agree that one camera will miss some things, especially with the height of drawbridges and sally ports this year. I would think instant replay would be used for major missed items such as defense crossings, scored boulders and egregious fouls for tipping and such. Maybe the head ref should reserve the right to deny the challenge if it won't affect the outcome over the match overall.

The GoPro definitely isn't on the level of what RoboShow, GameSense, IRI Live, etc. are doing but it's a start. Crawl > walk > run.

[Tristan Lall]

Not much has changed in 11 years. People are still basically arguing that because it is impractical to have a perfect replay system, every replay system is detrimental or infeasible. But the point isn't to be perfect; the point is to be better.

Part of the process of implementing a replay system is establishing the rules for its use. If you're concerned that too many replays will slow down a regional, then limit the number of replays and the circumstances in which replays are available to teams. Maybe every team can have one replay in the qualifying matches, and one in the eliminations, and all the alliance partners have to spend their replays together. I would postulate that much of the benefit of a replay system is to give everyone the certainty that it's there in the rare occasions where it would clearly be beneficial—but making replays artificially scarce conveys the strong implication that they should be used wisely, and would put a tight cap on the likely quantity of delay. Obviously a good technical implementation would also serve to control delay.

Maybe a given replay scheme wouldn't address the rare situation where the referees are so terrible or overworked that nearly every match is questionable. But it would start to address the far likelier scenario that a team feels bad about an entire event (and perhaps the competition in general) because one major, game-changing call was blown. And even if the replay footage was inconclusive, the act of a review will serve to placate the team. Something was done, so it can't be as easily argued that the team is being ignored. And the team and the referee have an explicit common frame of reference to guide the conversation—even if the challenge is unsuccessful, that will cut down on speculation about motive or competence. In this way, the referees appear deferential instead of capricious (without compromising their authority), and it therefore makes the competition appear more credible.

Originally Posted by DonRotolo

Side note: If you have never volunteered as a referee, you got nothing to say.

I can't get behind that. Referees know that they're subject to scrutiny, and should expect commentary substantiated by fact. And competitors should not be made to feel like they're not welcome to contribute their assessments of the problem—because obviously it's they, and not the referees, that are most affected by officiating errors.

Originally Posted by Lil' Lavery

I'm going to leave it at this. Find me one example of a student who changed career paths because of a blown call in a FIRST match.

This is clearly an unreasonable way of looking at the problem. What if the effect was subtle or not yet realized (e.g. the student doesn't rejoin the team the next year, and instead joins the chemistry club, which might, in a couple years, lead them to get a degree in science not engineering)? How would you propose to find such a student, given that this probably isn't a metric that anyone tracks?

[patar8746]

Tristan's second link contained this Gem http://www.chiefdelphi.com/forums/sh...4&postcount=15 but also both threads made me realize there's a problem: People have actually been saying the same things we are on this thread for over a decade. Anyone from FIRST know what's causing the decision to be made against video review?

[PayneTrain]

I love the FIRST Robotics Competition. Some people say "Wil, you complain a lot about FRC, why do you do something you obviously hate?" This is a program that has entirely enraptured me and to the chagrin of other obvious high priorities, I put a lot of time into my team. I know a lot of people put a lot of time into their teams as well. Don't we all want FIRST, an organization that by its very nature must always be sprinting forward instead of dragging its feet, to constantly be improving and tweaking the status quo to make the program better?

Some disagreements lie in both parties wanting to make the organization better in fundamentally different ways (2 champs), some disagreements lie in two parties wanting some change but have differences in methods (Chairman's feedback). The worst ones, and the ones that get me really fired up, are ones where under an almost instinctive impetus to make a change, the controlling party wants to actively fight against the change (Cali districts).

I think piloting a replay system at an event like IRI and other offseasons should lead FRC down the path to pilot it throughout the season in certain regions or globally, and see if it works by 2champs.

Raise your hand if you think missed calls are not a buzzkill and something no one ever feels bad about.

Stand and be counted if you think field disconnects make FRC better.

Shout from the high horse on which you ride, proclaiming "yes, I love to be in a program that says 'No! We cannot come up with any way of improving a problem that has existed for years! There is not even a problem! Scorekeeping errors enable teams, not disable them!"

I know who some of you are, I just want to see how this line of thinking will be swiftly dragged into the mud. FIRST can be all things to all people. It can be a program that transforms people of all ages and backgrounds. It can always be better.