Re: Video Review Needs to Happen Now
 

Honestly I just wanted an excuse to use that emoji :p

Re: Video Review Needs to Happen Now
 

Yes - Technology has changed - but trigonometry, basic game theory math, the number of hours in a day, and human nature haven't.

Few arguments against using video replays rest on technological foundations (some do, but the majority do not).

I look at this as one of those circumstances where "technology" can either be the final puzzle piece that enables great things, or be the shiny trickster that seduces users into great folly. We already know I lean toward the trickster side of the spectrum for this topic.

In one important sense, if technology is truly no longer a problem, it shouldn't appear much (except as minor footnotes) in this thread. Instead our conversation should be focusing on the rest of the strong arguments that have existed since 2005.

Let's hope enough off-season experiments produce enough repeatable, hard-evidence measurements against important criteria (not collections of fuzzy anecdotes from a few inconsistent, small sample-size, experiences) to move the needle in one direction or the other.

Blake
PS: Clear (1976) evidence that imagery doesn't lie. ;)
What are Dave Lavery's rovers really doing up there???
:eek: :rolleyes: :D

Re: Video Review Needs to Happen Now
 

Standing this topic on its head ...

What if inexpensive video (and image processing) became the foundation for both most of the scoring and some of the penalties, in future FRC games?

I think it is technically feasible (I'm being seduced by the "technology trickster") to design interesting FRC games, and to equip FRC fields, to permit/do this.

Cameras would track the state of objects, the location/movement(s) of some objects, and the locations of robots.

Humans would still be needed to enforce some rules (certainly, some penalties), and/or to step in if the field equipment/detectors misfire in some way.

After 2-3 years of behind the scenes R&D, would introducing games designed with this approach in mind, along with introducing fields that attempt to implement those games, be a net improvement or a lead balloon?

Blake
PS: Whatever the answers/opinions, it will be interesting to ask ourselves, "Why?", and then bring the answers to bear on this thread's original topic.

Re: Video Review Needs to Happen Now
 

I'm bowing out of this discussion for now. There's a lot of doubt, negativity and misinformation about what video review could be. These opinions won't change until we see some positive examples of it being used during the off-season. I absolutely cannot wait to implement it and see what happens (hopefully proving the naysayers wrong.)

Remember students - just because someone tells you something won't work doesn't mean you shouldn't go and find that out for yourself, especially if you think you can make it work. Often times the people telling you it won't work have biased opinions from bad experiences of their own or just don't know what they're talking about.

Re: Video Review Needs to Happen Now
 

I can see an adjunct competition where you are given the game and you need to design the scoring system for it. You get to deploy your scoring system with others at the event. Winner is the one that matches the humans the most at an event.

Side benefit is that in case of a question about the score they can go to the computer scores and see what they think.

Ryan, you and the others are on the right track.

This is CD gold. For the last decade I've often been surprised and impressed with what roboteers can do. I never discount a motivated roboteers ability.

I for one welcome my new computer scoring overlords. :rolleyes:

Re: Video Review Needs to Happen Now
 

That is NOT a bad idea at all. Could be an interesting offseason competition: Given game X, design a scoring system with constraints A, B, and C--most likely portability (including ease of assembly into the field), durability (including fault tolerance/"idiot-proofing"), and cost. Must match or beat humans.

Re: Video Review Needs to Happen Now
 

Is anyone in the mood to put to together an accurate enough model of this year's field (or a previous one), then add some similar fidelity robot and game piece models, and then explore the accuracy and obstruction (and misinterpretation of camera imagery) topics connected to using replays to answer questions?

I'm betting someone among us has the models, motivations, domain-knowledge, and free-time necessary.

Any takers?

Blake

Re: Video Review Needs to Happen Now
 

Folks,

This interesting article can probably be used to bolster the positions pn all sides of this topic. It has some info about how sensors and computing are (and aren't) being used to supplement/replace human refereeing in some big league sports.

I enjoyed reading it.

Blake

Re: Video Review Needs to Happen Now
 

Did you forget a link?


That being said, for a number of reasons automated scoring can be very useful. Ball scoring has been almost completely automated on several occasions ('10, '12, '16--'06 was automated but required manual backup due to system design). It's GREAT to not have to worry about scoring balls with the barrage that most alliances can unleash. OTOH, the DOGMA penalty in '10, which was automatic, was kind of a nuisance--and on at least one occasion, a team did exactly what they were supposed to and still got a penalty series from that (ball missed the counter on the way down). '08 had automatic robot lap counting, if that actually worked--I don't recall hearing anything either way on that.

But there are still a number of cases that automation would make particularly "interesting". Here's a couple related ones: Could a sensor detect whether a boulder crossed legally this year, under the case where a robot did not finish (or start) a Crossing but the boulder went across? I think so... but I would argue that that would have to be a very good sensor system to detect that, particularly live on the field. The question is, is the added cost* worth it for a system that would only need to be used some small portion of the time, for a non-profit competition system? And that I would say that it depends.

Many things can be automated. Some things cannot be automated. I happen to like the MLB replay model where it's basically a second set of eyes, detached from the game in question, looking at the various viewpoints, and not everything is actually reviewable.

*Not necessarily money--time, training, etc are all factors in the cost.

Re: Video Review Needs to Happen Now
 

Just as a design thing (not really related, but I'm interested), it'd probably be easier to put a location sensor within each bolder. Location can be accelerometer/gyro combo with a tiny controller. Of course, I'm not a fan of making those boulders even more expensive...

Re: Video Review Needs to Happen Now
 

Doh! Doh! and Double-Doh!

This is the link I forgot to include earlier today.

The missing link

Blake

Re: Video Review Needs to Happen Now
 

Video review made it's debut in Olympics Beach Volleyball this year with specific rules on what may be reviewed, such as line violations and in/out calls, but not judgement calls such as net violations and lifts, and additionally there's only a set amount of time after a play (5 seconds) in which they're allowed to request a challenge.

I think this is a good example of how to implement video review in the context of FRC. Not everything should be allowed to be challenged, only certain black or white calls, such as crossing a zone in 2014. A time limit on requesting a challenge should be implemented.

Re: Video Review Needs to Happen Now
 

And that's a reasonable implementation, too. The game changes year-to-year, but there's almost always some hard-and-fast "do this like this" area that can/should be reviewed.

If I were to implement replay, judgement calls would be out of the question for team-implemented review. (I'd give the head refs the option to use it for judgment calls, if they thought it necessary, and I'd guess that most head refs wouldn't. The bigger the judgement call, the more refs are in the huddle.) To review a call, it'd have to be something that could be seen and quantified from the video, and isn't covered by any automated systems or placement at the end of the match. (End-of-match placement, you can get 2-3 opinions in a matter of seconds.) Sounds like auto line violations, secret passage calls, and defense crossings for 2016; zone entry in 2014; contact around the protected zones in 2012 and 2013.

Re: Video Review Needs to Happen Now
 

Folks,

Now that the summer of 2016 is almost over, does anyone want to report any new (previously unreported) results of experimenting with video replay ("hard" data or anecdotes) during off-season tournaments/scrimmages?

What about any non-tournament / non-scrimmage experiments to determine numbers of cameras needed for good results (for Stronghold), or best camera locations (for Stronghold), or minimum useful frame rates (for fast robots and game pieces), or estimated time needed to accurately review each/any protested ruling (for Stronghold (for each rule and/or scoring event)) or ...

Blake

Re: Video Review Needs to Happen Now
 

Again, as a theoretical - no hard data here - but there are easier ways than cameras. Requiring each robot to carry a small sensor packet that can tell where the robot is at any point in time would have fixed the crossings issue this year. Maybe in the future, it could even be built into the RIO.

Re: Video Review Needs to Happen Now
 

Actually, it wouldn't have totally fixed that issue. You'd have to completely redefine Crossing for that to work automatically--Crossing is determined by bumpers, and making a foolproof system that could tell that the bumpers were clear from a sensor packet (from whom? team/FIRST/other?) that would need to be inside the bumpers would require a lot of complexity. Need to know size + shape + location + orientation of the robot with bumpers, the last two in three dimensions each. It'd be a lot easier for Scales--mount a device or two at the bottom of your bumpers, if they're all above a sensor mounted in the castle wall at the end of a match you're good. (Reasons to deny a Crossing included starting in contact with a defense--see also bad Sally Port crossing--as well as bumpers not finishing clearing and bumpers going over/above the shields.)

Now, they did do something like that in '08, where each robot carried a small IR beacon (or something like that--it was field-supplied with another piece of required equipment) mounted in a known area of the robot--pass under a receiver, get credit. But I don't recall how well that worked.

Can it be done, yes. But to be honest, the second-best tool we have right now is actually cameras, properly pointed, if the rules allow it. The best tool? The 6 pairs of eyes in the heads of the referees. Without those, there is not a good way to tell if the electronics are acting up and not counting stuff properly. And, of course, we sure hope the game is designed so it's a little easier on the refs...

Re: Video Review Needs to Happen Now
 

As I remember, poorly. I do not believe they were used at all events.

Re: Video Review Needs to Happen Now
 

I would ask again in the fall. Most offseasons happen in the spring or fall; relatively few in the summer.

While I think nobody can expect perfect camera coverage of the field, even a single full field view results in some things being reviewable. If someone tries to get a review and there isn't clear video, the outcome doesn't change and the situation is no worse than having no video review at all.

I feel like a "it has to be perfect or it isn't worth doing at all" attitude here will really hold people back from experimenting with this.

Re: Video Review Needs to Happen Now
 

Other than the additional time wasted.

Re: Video Review Needs to Happen Now
 

Nothing's holding off-seasons from experimenting with this. The real problem lies with real competitions, where we should be wary in implementing something that can make them even longer without having a clear and proven positive effect (and a streamlined process for doing so).

Re: Video Review Needs to Happen Now
 

Sure - I'm aware, I thought a few were going to do some trials this summer, so I asked. I or someone should give this thread another poke in late Fall.

I certainly hope not.

Conducting experiments is how you quantify "better, but not perfect". When a discussion reaches that stage, it can leave endless cycles of hand-waving behind.

Imperfect is fine, if the imperfections are known, and well-understood; and if the decision-makers agree that the imperfect system is the one they want (because it's better (in some hopefully well-defined sense) than the system it replaces).

Imperfect, and poorly understood, when/if it were implemented, ... Well, that would be a different kettle of fish.

Well designed experiments are exactly what this topic needs.

Blake

Re: Video Review Needs to Happen Now
 

During the time the video review is happening. Pre match things can be going on. Ie connecting robots, and reseting the field. I can't see very much time wasted at all unless the system to replay matches is not working at all.

Besides I would rather spend 30 minutes longer at ever Frc event then see anyone leave an event how MY team did this year. (go look at the start of the thread my team was on that alliance). If there was video review we would have atleast been semi finalists. But, being our alliance set a higher score then that semi finals alliance there was a good chance if video review existed we could have been finalists. I have thought through every scenario I could have done to make the outcome different. Design choices, driving choices, repair choices, ect. But the bottom line is our season ended because of a bad ref call. I don't fault the refs. This game was ridiculously hard to referee. But, the fact of the situation is still there.

Video review would help the teams, and help the refs. It doesn't cost that much to implement and maybe it can be integrated into an frc dedicated livestream setup. Which would be a huge win.

Re: Video Review Needs to Happen Now
 

In what well-designed experiments has this been qauntified?

Without proof that the net effect (the entire gestalt) will be better (in some well-defined sense) outcomes, both you and I are just announcing our opinions while figuratively waving our hands in the air.

Blake

Re: Video Review Needs to Happen Now
 

I think that one of the biggest things we always need to keep in mind is the goal of FIRST and FRC. What are we really trying to achieve? Inspiration.

The next question we have to ask is "Is it really worth it to implement video review systems, or will the goal of inspiration* be just as attainable even with the few (and far between) errors of the ref?"

*Yes I understand that having teams' hard work recognized plays a part in this. That's why this is a question, not a statement.

Re: Video Review Needs to Happen Now
 

The term inspiration is very subjective and to me having a higher percentage of calls correct is much more inspiring.

Re: Video Review Needs to Happen Now
 

I think the point trying to be made is video replay may not be inspiring, on it's own, but it's a tool to prevent the kids from getting discouraged by bad calls. Thus it's a net positive to inspiration overall.

Re: Video Review Needs to Happen Now
 

I definitly agree with you both. My question is whether this (however amount) increase in inspiration is worth the longer waits, and possible teams leaving to prior awards, which, in my mind, decreases inspiration to those members (trying to quantify an unquantifiable thing :) ).

Re: Video Review Needs to Happen Now
 

The one example of video replay so far at TRI literally happened during the setup for the next match and didn't hold up the match at all.

The argument saying time would be wasted is a bad one because it can be done so quickly that very little time is wasted.

Re: Video Review Needs to Happen Now
 

tl;dr.

I am a full supporter of video review.
If matches/events take longer as a result, then let it go longer.
I can see where subjective calls not being reviewed is fine.
But for missed calls such as breaching defenses in 2016, definitely could have used some video review.

I cant see anyone arguing or being upset, if a call was overturned to get it right, where an alliance ends up losing a match instead of winning. Getting a right call trumps that.

Re: Video Review Needs to Happen Now
 

Fair enough. All I'm saying is that benefits/costs should be weighed carefully. If it takes up little extra time and allows better calls, I'm all for it.

Re: Video Review Needs to Happen Now
 

(First - /agree with the earlier comments about not necessarily causing a delay but...)

This is basically a thought exercise, but if you ignore team sizes for a moment, which is more de-inspirational, potentially ending your season on a bad call, or having to leave the event that's run late a few minutes early?

Another thought here is that wouldn't Gracious Professionalism suggest that a short delay, if needed, to get the call correct, be the right thing to do?

Re: Video Review Needs to Happen Now
 

Follow the rules.

Encourage or discourage FIRST to change the rules from one season to the next, but once they are established, follow them.

Re: Video Review Needs to Happen Now
 

Can you elaborate on what happened? What was the call on the field/what was the challenge?

Re: Video Review Needs to Happen Now
 

More info here: https://www.chiefdelphi.com/forums/s...=146411&page=2

Re: Video Review Needs to Happen Now
 

Given all the opinions (so far) about what Video Review should or shouldn't be, the word "it" in this sentence is very poorly defined.

For that reason, still color me undecided.

YMMV.

Blake

Re: Video Review Needs to Happen Now
 

I like the concept of video reviews being used to make sure the correct call is made, using technology to make the competition better seems like a very FIRSTish thing to do. We have loads of regionals, maybe FIRST could do a trial run at a few and see how it turns out?

One potential downside is that there might be more people in the front rows waving their iPads in the air. If that's a sacrifice we'd have to make, then I don't wanna do it.

Re: Video Review Needs to Happen Now
 

If there's video review at an event, it's gonna be from official cameras or camera stations. I don't see any other way to make it fair to the teams whose entire team + supporters are on drive vs the teams with 5 students and a mentor on video.

Honestly, if I was to do a replay--and this is what I'd be pitching to offseason events--I'd figure out where the most non-judgement calls* were going to be. For 2016, that's the Defenses (and possibly the Secret Passage). It'll vary by year. That's where any cameras get pointed, probably one per area. If whatever is being contested is in view, then it's review time. If not, you're right out of luck, better luck next time.

One other element that I'd be considering would be a case of "what if something else got missed?" My opinion would be that whatever gets caught on video but missed by the refs should be taken into account when determining if the outcome of the match was affected--and unless the outcome is ruled to be affected, the only thing that is reviewed is the original request.


*Defined as a call where, if it's seen, it happened, and there is no room for interpretation. Examples are non-judgement calls; Counter-Examples are judgement calls. Example, defense crossing. Example, two balls in the robot. Example, Secret Passage violation. Counter-Example, intentional tipping. Counter-Example, egregious behavior. Counter-Example, robot with ball bumps another ball out of the way or drives over it. Example, zone entry in 2014. You get the idea.

Re: Video Review Needs to Happen Now
 

My definition of 'it' is what I already posted in this thread on what we will be using at GRC in October. Link in case you need to go read it again -

https://www.chiefdelphi.com/forums/s...2&postcount=49

Re: Video Review Needs to Happen Now
 

This presumes every game has a long reset period and robots take a long time to sync up. Sure, if that's the case, then there could be time for a review. I know I've spent many a reset period minute discussing situations with students in the question box.

But really, shouldn't we be looking to make matches run quicker, with less down time?

Re: Video Review Needs to Happen Now
 

The argument regarding additional time is not centered around the time to conduct the video review (hopefully there is a definite time cap on that), but rather the fact that blown calls lead to replayed matches. A single replay at a single event that did not change the outcome of a match does not change that factor in the least.

Re: Video Review Needs to Happen Now
 

Ryan's point about 200 posts ago was that events could trial video replay to give some HARD data on what they did, how many reviews, outcomes, issues, etc.

One data point at TRI. One coming up in Oct from an event that Ryan is running.

Any other off seasons done or planning to do replays?

Re: Video Review Needs to Happen Now
 

I'm going to try to convince a local offseason we attend and do the Webcast for to do them. Its not my team that hosts the event though so I'm not too sure If it will get the green light.

Re: Video Review Needs to Happen Now
 

Asking is most of the battle, thanks for trying!!

Re: Video Review Needs to Happen Now
 

Has there been a ref offering an opinion on this? I don't want to see a mass exodus of refs over this.

If everyone can call for a match review, would you be expected to review the match for the 6 objections made every match?

The refs all take this very seriously. Let the refs do their job.

On the flip side, perhaps I should start wearing a body camera to work. That will make my day go easier !!!

Re: Video Review Needs to Happen Now
 

There have been refs in this thread.

Here is the way I see it. A ref wants the right call to be made. No one wants their mistake to hurt a team. A safety net stopping some mistakes from hurting teams is something I can't see anyone mad about.

Re: Video Review Needs to Happen Now
 

The concern about 6 objections every match is unjustified. At TRI, only one objection was made the whole event. I think that teams will understand that asking for silly reviews is a waste of time for everyone, and will only ask when needed. Six objections in one match (not to mention, every match) is going to be very very rare.

Re: Video Review Needs to Happen Now
 

Something to add that watching volleyball in the Olympics reminded me of. They have a review system for balls being in/out. And you have as many challenges as you wish as long as your claim isn't proven to be incorrect.

Perhaps a system could be implemented so that all teams get as many vid reviews as they wish until a claim they have made is denied and thus lose the privilege to contest via video review.

To me this is a win win. Discourages claims without certainty, and if a team has two or more valid issues, they aren't punished for it.

Re: Video Review Needs to Happen Now
 

This ref's opinion*: I'm good with it provided the FTA side is on board there are clear bounds to it. Official video only, set situations each year that can be appealed, published mechanisms for doing so, teams cannot mandate reviews of judgment calls (they can question box it and/or refs can decide look ourselves). Maybe we need to cap the challenges per team; I'd be okay experimenting with either plus a blue box warning of possible changes mid-season. It also needs to be piloted more, not because I think I'll change my mind but because there are far too many nuances, procedures, and edge cases left to work out. We do not want to be fumbling over something like this in Week 1 on top of everything else when it could've been honed in off-season. Trust me, you don't want us to be either.

*I feel I can speak with some credibility on both how difficult this is and how much it can suck, but that does make me not a "pure" referee. I am a veteran ref and off-season head ref but also a veteran coach and have been burned by reviewable missed calls before. This includes a district champs playoff loss, though without lingering bitterness (we made Einstein that year). Consume with however much salt you want.

Re: Video Review Needs to Happen Now
 

No salt needed, I'm good

Side no review: Too hard, no tech, refs are volunteers, match play times, adds too much to the competition, we all do the best we can

Side review: Tech isn't that hard, less than 1/12 of a Dean speech to the schedule, we all do the best we can.

As a VEX event producer I've said these words more often than I can count. "The referees rule is final. This is a robot competition that has referees watching each second. The referees rule, and their ruling is final. But in the event they are wrong, and that is a possibility, pull me aside and we will talk it over, as long as you want, but the last words out of my mouth will be 'the referees rule is final'.

But like Siri, I've been on the wrong side of calls and have blown calls DRAT YOU CELL PHONE VIDEO !!!

But this thread is on possibilities. Can we do it, can it work, can it be a positive method, how do we make it work?

Most of the thread has posts of "No", "Umm no", "Don't want to" "Will kill refs" "Will end civilization as we know it" (and worse)

So again, the fall is busy with off season as we build interest, try it out.

As the lone science guy in this thread :rolleyes: hard data beats feelings every day.

Siri asked for some boundary conditions (FTA's, process, etc) and I think thats a key thing. But FIRST isn't going to even test fly without some data points. Lets get the data points. And if the data points are "will end civilization as we know it" then we can close this off.

Re: Video Review Needs to Happen Now
 

Can anyone cite a real world example where this is used and works?

I am struggling to find a situation that this would apply.

Re: Video Review Needs to Happen Now
 

Outside of sports? MLB, NHL, NFL, NBA all have review; MLS doesn't have much if at all.

I'd have to say security video--there's a number of potential uses there. Though often in that sort of case the call is known, but the reasoning isn't.

Re: Video Review Needs to Happen Now
 

I think a reasonable set or rules regarding number of challenges could be:

• For the entire Qualification match schedule, each TEAM gets one unsuccessful CHALLENGE. For an ALLIANCE to issue a CHALLENGE, all 3 TEAMS on the ALLIANCE must have their CHALLENGE remaining, and all must agree to issue a CHALLENGE. If a CHALLENGE is successful, all TEAMS keep their CHALLENGE. If a CHALLENGE is unsuccessful, all TEAMS lose their CHALLENGE for the remainder of the Qualification MATCH schedule. If a TEAM still has their CHALLENGE at the end of the Qualification MATCH schedule, it does not carry over to the Elimination tournament. For a CHALLENGE to be successful, the ranking points awarded at the end of the MATCH must change as a result of review.
• In the Elimination tournament (except for Final MATCHES), each ALLIANCE gets one unsuccessful CHALLENGE. The ALLIANCE CAPTAIN determines when to issue a CHALLENGE. If a CHALLENGE is successful, the ALLIANCE keeps its CHALLENGE. If a CHALLENGE is unsuccessful, the ALLIANCE loses its CHALLENGE for the remainder of the Elimination tournament. For a CHALLENGE to be successful, the winner of the MATCH must change as a result of the review.
• All non-judgement calls in Finals and Einstein MATCHES are reviewed automatically during the ensuing FIELD TIMEOUT.

Re: Video Review Needs to Happen Now
 

I don't know if I'd go this far. This would require teams to calculate in their own heads (or calculators) what the new score would be and if it would swing the game. I think this would just waste more time, and it'd be easier to just say points instead of ranking points.

Re: Video Review Needs to Happen Now
 

If the score is 100-45, the losing alliance challenging to get an extra 5 or 10 points is a waste of time (assuming total score is not part of the ranking system). I think requiring the review have an effect outside of just the match is not unreasonable.

Re: Video Review Needs to Happen Now
 

For quals, the requirement that it change one of the aggregates in the ranking algorithm (though I'm not positive that's what cgmv means) would be pretty loose in most years. This seems like a reasonable standard, since it's easy enough to know beforehand ("that would raise our auto score total") and speaks directly to the impact quals are supposed to have. We could probably keep the same condition for elims for simplicity's sake. Alliance Captains should be smart enough by that point not to risk losing a challenge on a ruling that wouldn't actually benefit them in the bracket.

Not sure how I feel about requiring every member of a qual alliance to still have their challenge coupon. I think I'd be okay with "any" instead of "all". It likely means more challenges, but then normal teams can't be burned by being randomly assigned a trigger-happy alliance partner who wasted their coupon on match 1.

I suggest some kind of time limit rule as well, like the completed coupon must be submitted to the head ref by no later than the starting whistle 3? matches after the match in question (match 4 for a challenge in match 1) or before the next elim level. The latter gets tricky if you're the last QF match to play.

I currently envision the challenge coupons including team name, match number, alliance color, specific challenge (from the list of acceptable ones), approximate time and field location, and a FOUO section for review outcome. Anything else? (Refs would archive the submitted slips; if you're right you'd get another blank one.)

Re: Video Review Needs to Happen Now
 

I agree on that. Anything from missing one breach ranking point on up is reasonable. If total score is part of the ranking system, that could be re-thought, but I'd suggest any score changes less than X amount (determined by the game and ranking, but let's call it one penalty of TBD type) would be unreviewable if the score differential was greater than 2X.

I disagree on the Finals being an automatic review, primarily because that means a minimum of 3 minutes where any refs involved aren't doing their between-match stuff (traffic control, overall monitoring), and also because if there's something tough in the Finals every ref is going to be in the huddle discussing the calls--we want to get the calls right the first time. What I'd do instead would be to reset challenges (I'm in favor of LIMITED challenges, and I'll explain why in a minute) to full for finals regardless of prior usage.


The reason I prefer limited challenges (probably 1/alliance in playoffs, with a second if the first is successful) is that by the second challenge from the same alliance, if the Head Ref hasn't shuffled the crew, he or she probably needs to. And it may be obvious on the reviews that one ref or another needs to be shuffled to break or another field position if possible. For those that aren't refs, the ref crews tend to find their weak links quickly and strengthen them as needed. If there's two challenges, chances are that there's a ref that needs more strength--or it's possible that the alliance is just trying to game the system.

Re: Video Review Needs to Happen Now
 

My intent was that ranking points (or the first order ranking sort in subsequent games) would need to be affected. For Stronghold, this would have meant changing the outcome of the match or awarding a point for a missed breach/capture. I'm reconsidering this now, because you correctly point out that the ranking tiebreakers can also matter. I'm not sure where to draw the line, though, because for Stronghold, the only scores that don't factor into at least one tiebreaker are foul points, and I don't think allowing a challenge to every single Qualification match score is worth it. I'm fine with Eliminations going deep into the evening if it takes that much time to get all of the calls right, but I think Qualifications should be expected to stick to the schedule a bit more.

Yeah, I wasn't sure how fair/unfair it was for teams to be able to jeopardize future alliance partners by losing their challenges early. I do think it's fair to limit challenges in qualification matches to blatantly obvious errors with ranking implications, and the rules I proposed are written with that in mind. They're just a suggestion that I expect the GDC (Hi GDC!) to tweak before placing in the 2017 manual. :wink:

The automatic review wouldn't be done by a referee. It would be done by a "replay official" who would "confirm" most of the calls in the match right away and only call over a ref if they see something questionable.

Re: Video Review Needs to Happen Now
 

Quick thought with regard to the 'must affect ranking points'....

If there is a) a time limit to make the call b) you need all 3 teams on the alliance to agree and c) need the ranking points to change then:

1) Is there enough time as robots are being pulled from the field to coordinate with the other two teams?
2) AND in the event that a 100-90 loss was a missed breech and would have resulted in a match point tie, is there still enough time to find and calculate the tie-breaker while coordinating with two other teams?

End of match is hectic as it is... If a team/alliance is feeling slighted by a call (or non-call), it's even more so. While I agree there should be reasonable limits the challenging team should have sufficient opportunity to avail itself of the rule.

To that end I would change the rule to be that the challenge, if successful, must either change the ranking points awarded, or change the win/loss/tie result of the match, without regard to the tie-breaking formula.

Re: Video Review Needs to Happen Now
 

I'd stick with first-order sort, possibly 2nd-order. Most times the sorting doesn't seem to go beyond 2nd order. 2nd-order sort generally being auto, those would be quick reviews (15 seconds or less).

And this is where you need to remember something: the replay official is going to need to have at least some referee training. Otherwise, how are they going to know a questionable call? I know we're taking judgement out of it as much as possible, but, for example, when we're looking at crossings, there's an awful lot of nuances to crossings (for robot and for boulder) that sometimes people don't quite grasp--I once had a team complain during practice day after I called them for not finishing a Crossing before returning to the NZ from scoring in the high goal. So if they're going to need to have referee training anyways, then you may as well make them a referee. And if they're a referee, you may as well make them work the field for a few matches, particularly if you're short-handed. So that means that several refs may as well have training on the replay system.

I think the best way to handle replay would be to have the "off" ref in the rotation handle it, or have two "off" refs (one assigned to replay at any given time). Now, finding the refs over and above the field crew can sometimes be difficult. But I think with enough effort, someone could be found... And actually, that would speed up replays a bit--if you've got an off-field referee going through them during a match, then the only thing they need to do is to advise the head ref (NOT a regular ref, BTW, that's another thing--this is a call reversal or not) that this-that-and-the-other is the case, or that he needs to take a look as somebody's asking for a judgement call, or what-have-you.

Re: Video Review Needs to Happen Now
 

In Quals, yes, fouls don't factor, but in Playoffs they are the 1st tie-break in a tied score. Cleaner played match wins. [5.4.4]

Re: Video Review Needs to Happen Now
 

We could hedge it and say "changes Nth (likely 1st/2nd) order ranking points or changes rank". Presumably any team looking at a deeper ranking aggregate is on-the-ball enough to determine whether a changed call would move them. It leaves out edge cases where it'd move you closer to jumping someone without actually passing them, but it does have to stop somewhere (I'd argue).

Re: Video Review Needs to Happen Now
 

Caveat -- I'm not trying to reopen the battle about if there should be a video review or not. I just wanted to ask a simple question.

There has been much discussion about the technology needed to do video reviews and some teams have said that they have that level of equipment.

Since we are in the final weeks of off season events, are there any events that are planning to allow for video review?

Thanks!

Re: Video Review Needs to Happen Now
 

Ryan's folks have one coming up sometime in October.

Re: Video Review Needs to Happen Now
 

I asked about doing one at one of the local offseasons, as a pilot. No-go with the planning committee.

On the other hand, they did approve one of my other ideas... If that goes well, I may find myself needing to write a report on it.

Re: Video Review Needs to Happen Now
 

Eric H, if the rest of us can volunteer you for some work, maybe the planning committee will let you place a few cameras at overlapping locations so that you could collect data able to shed light on some of the more technical topics raised earlier in this thread.

Topics like "How many points of view are needed to give an accurate record of things like line-crossings, or of contact between robots, or of contact/positions among/of any other physical parts of a match."

And/or maybe produce some screen shots illustrating:
- How much moving object blur/tearing exists in a single frame of the imagery captured by the cameras you happen to use.
- How far robots and game pieces travel between successive frames of the video at the frame rates you try.
- How much real world area each pixel represents in the images you capture (length of the field, worth of the field), given the camera settings you try (see next bullet)
- How much the resolution varies throughout the cameras' depth of field, given the focus & aperture settings, and lenses you choose to use.
Etc.

You would do a little scurrying around, and maybe pose some robots during down time, but your effect on the event would be hardly noticeable compared to a full replay experiment.

Blake

Re: Video Review Needs to Happen Now
 

Feel free to analyze any of the video we have posted on our channel, we have two events worth from this summer with three different camera angles in every shot.

https://www.youtube.com/playlist?lis...ff6pk35HJXrr9n

Re: Video Review Needs to Happen Now
 

If the event wasn't tomorrow, that might theoretically be possible. But I'd need to clone myself at least once. My effect on the event is rather noticeable in my current role...

However, if you're interested in the typical camera setup, I've put the stream links to TBA (Fall Classic).

Like I said, there's something else I'm up to that has some relation to this topic, but is in a somewhat different vein. If I remember to write it up afterwards, I'll post it.

Re: Video Review Needs to Happen Now
 

Not today, but maybe sometime reasonably soon. Doing that would give me a good reason to recreate a good-enough (bare) field model.

What can you tell us about the cameras? Lenses, F Stop, Virtual shutter speed & other "shutter" settings, frame rate, Model number (video chipset), resolution chosen, and, of course, locations & orientations (plus whatever else I'm forgetting)?

Some results can be evaluated reasonably well without that info, some not so much.

Also, I would have guessed that videos posted on the Web for ordinary consumption would have gone through some lossy compression. Is the raw, 100% uncompressed frame-by-frame imagery available for download from the YouTube page at that link (I'm definitely not a YouTube guru)? Thumbs up if it is. If not, do you have a link to a site where I can download some of it?

Blake

Re: Video Review Needs to Happen Now
 

As it turns out, nobody took advantage... But I'll post the write-up anyways, or something like one.

The offer: Any person who could plausibly be a ref within the next couple of years (high school seniors, parents, mentors) was offered a chance to shadow a referee for a few matches. Now, obviously they wouldn't actually be making any calls, but being with the referees and able to ask questions is big for both the people and the refs, as normally their interaction is one or two kids talking to the head ref in the question box. (I also had an "open box" policy as the head ref for the event: The box is where I am, and the person in it is the person talking to me.)

There were no takers.

And I had a team ask me if they could set up a camera to check the crossings, and at least two teams tried to have me look at a replay. I told them all that I couldn't look at a replay--not that they can't set up a camera, mind you, but that I can't look at a replay.

I did have a team or two ask about crossings being counted--my usual response was to turn to the FMS operator and ask for the appropriate sheet for that match (we used a paper scoring system, and we had scorers working with the refs--HQ, take note, refs could use some scorers just to enter data), then check with the appropriate ref(s) and make corrections as necessary.

Re: Video Review Needs to Happen Now
 

Hey off-season folks,

What news do you have for us?

What are the new anecdotes?

What are the new hard data?

Blake

Re: Video Review Needs to Happen Now
 

Small anecdote here.
I reffed a small, one-day off season event recently. We were pretty short-staffed, so I was doing field reset, counting defense crossings, and keeping tabs on all the stuff behind one alliance wall. In two cases, I know I might've missed a crossing. In one case, it would have made the difference between a Breach or not for a team that was in the top 8 after the Qualification matches. I would have REALLY liked the chance to request a video replay, because I'm honestly not sure if I made the right call in a situation that would have affected a team's ranking somewhat significantly.

Not exactly a new or groundbreaking sentiment, but at some point, I'd like to ref again. I'd feel more confident reffing if I knew I could access video replays to give every team the results they earn.

Re: Video Review Needs to Happen Now
 

Not related exactly to video review but it's worth sharing.

Yesterday I tried my hand at reffing at a fairly large off season event. After my experience in that position I realise how easy it is to miss something like a crossing or a pin or a g43 violation. I'm sure I missed atleast 1 crossing yesterday and that wasn't for lack of trying or knowing the rules. On an Frc field there is a ton of stuff going on and despite 1 ref watching courtyard fouls and 1 watching crossings its extremely easy to miss something.

So why not have video review? If I missed something that caused a team to lose I'd feel horrible and if there was a way for my mistakes to be realized before ruining hundreds of kids competition it would be great.

Video review needs to happen. After my experience as a ref I belive that even more. I've been on both sides of the coin now and on both sides I want video review. I don't care if it means I leave the completion an hour later.

Re: Video Review Needs to Happen Now
 

What if we had video review only for playoffs/eliminations and only the referees can call for video review? This wouldn't slow down the competition too much, and the refs would know exactly what happened. While some argue that "this isn't what FRC is about", I disagree. This allows for teams that should have gone to the next level, which could be the District Championship or World Championship. Those championships allow for much more inspiration than just a regional or district event. Also, the public opinion of teams about certain events would become more positive because they know that there was no error that caused them not to advance to the next level.

Imagine this: there is a finals match like the final Stronghold match we saw this year at an event. The teams are tied, and a referee mistake could allow for a team to advance that maybe shouldn't have.

I believe that this is something we need to experiment with, however the power should not be overused,and would need to be monitored very closely.

Re: Video Review Needs to Happen Now
 

I would make that the Head Referee only, myself--and it's actually easier than you might think to implement, IF you ignore the technological part of the equation. Basically, all the GDC would need to do would be to include video replays in the allowable sources the Head Referee is allowed to consult, for playoffs only, and clarify the "will not review" part of that to only be during qualifications. (The list currently includes GDC members, FIRST personnel, the FTA, and technical staff, per the manual in 5.5.3, just for reference.)

That being said, that's ignoring the technical part--camera placement, clear views, etc. That's been discussed ad nauseum, so I'll just leave the discussion there as solutions exist.

The other half is that by allowing the Head Ref to use it, use is not required. This could lead to variation between events, similar to the classic complaint to the inspectors of "But the Magnolia Regional allowed _______!" Probably some consistent standard would be needed, but I suspect that that would end up in the ref training materials, and thus the teams might not be able to know what it was.

First question I have is, who determines what teams "should have gone" to the next level? Honestly, I would hope that that's determined by the teams on the field of play--even refs don't like swinging the winner by (debatable) penalty, folks!

Second question stems from major sports. They all use replay, to some extent, so the question now becomes why "bad" calls are still making it through the replay system? Paid refs, one game piece (and however many players), $$$$$ replay system, and calls still get through??? Huh, funny how that happens. How are we sure that no error was made?

Now, there is something that would deal with the public opinion BETTER than instant replay, IMO: Transparency about the call, at least with the affected teams if not the entire event. In the playoffs especially, I would say that if the head referee takes the time to go over why the call was made (or missed) with the teams, it's actually better than a replay--and if there's a score correction that needs to be made, then it needs to be made. The teams might not like the call--at least three of them won't!--but I'd be hopeful that they'd understand why the call was made the way it was made.

You'll notice that the only reason for the tie was, in fact, a penalty--and whether that penalty was a good call has been discussed, with the conclusion that there was grounds in the rules for it. Just some food for thought, as that wasn't the only time during the season that a single penalty flipped winner to loser of an elims match. I think I saw that happen two or three times prior to that.


The other part of the problem is that FIRST may just need to do a better job of determining which games need dedicated scorers. 2014 and 2016 didn't have them (at official events--though by the end of 2014 the number of refs had increased to allow some to be scorers). 2015 did. Take a wild guess which games actually needed the scorers? (I'll go on record as saying that having scorers in 2016 really helped at the offseasons I was at, even if they were just recording a referee call.)

Re: Video Review Needs to Happen Now
 

I would like to reiterate match logistics and explain a few reasons why Video review is out of the question, and should never be considered. I am a field staff volunteer, and have work with a few different teams.

Let us start with event logistics from the point of view of field staff, and refs for that matter. A district event typically has around 36 to 40 teams. For that matter lets actually use an event for logistics. How about FIM Southfield event this year.

The event had 39 teams, and 10 hours worth of time scheduled for qualification matches on the public schedule. Each team was set to play 12 matches each giving you 78 matches. That gives you about 7.5 minutes per match.

Now let’s think about game play. What has to happen for a match? The field needs to be configured, robots need to connect, teams need to be announced, the match needs to be played, the scores need to be submitted, then posted, and robots need to clear the field. How long does this all take? About 7 minutes. Exactly the time allotted to each match, and that’s without having any problems with robots or the field that delay the start of a match, match replays (when needed), field repairs, and any number of other factors that may result in a delay of a match on the part of teams or the field. This 7-minute cycle time is just about the limit. it takes 30 seconds or so to prep the field for team connections, 3 to 4 minutes for the teams to setup robots on the field and connect, match time running about 2.5 minutes, then give the refs 30 seconds to confirm the score and for it to be posted. That adds up to about the 7 minutes depending on how long each of these steps takes. Can we make this faster? not without teams setting up robots faster. Even then, with field configuration, robot connection time and the time it takes for matches to run, and scores to be submitted and posted, the fastest time that matches tend to run is 5 and a half minutes, and that only gets hit at most once an event if that. Matches tend to run between 6.5 and 8 minute cycles.

Where in there do you plan to fit in video replay? Replaying a full match would take 2.5 minutes. Not to mention times to analyze what they are seeing, and re watch parts if needed. Maybe add 30 second to adjust and verify and scores that need to be after that? that gives you about 3 minutes to add onto match cycles. so that scores can be posted, so the field can be configured for the next match. Plus, who is to say that there won't me a few additional minutes of discussion about what’s being watched and if those points were counted. How much time do you really want to add to score review? how much is too much? well if you add the three minutes that we gave above, probably the lower end of the range, that gives us 10 minutes per match, and with 78 matches as at Southfield, that gives you 13 hours of game play. You now have to add 3 more hours on to the first day of qualification matches so that you can get done in time. This means keeping students at the venue as late as 10 PM in the case of Southfield. Some people would say that’s reasonable, but some teams traveling more than an hour to events, that’s unreasonable as an hour back to the school and then time to get home, puts students arriving home at 12 PM. That won't fly with many school districts, and do you really want students, mentors and volunteers running on that little sleep? Well think about it. Southfield's event opened at 8 AM the last day. with an hour bus rides, students need to meet before 7, and wake up at 6 to get there by then, and well we all know they don't go to sleep right away.

Now let’s think more about how this entire replay thing would work anyways. Okay so you record the video from the audience screen right? well what’s that? a full field camera? reasonable, but it can't get everything, can't see close details. Okay so you record all the cameras? Well Most setups have 3. Full field, and one for each alliance. Most of the feeds for the walls did not fully catch the defenses, so you don't get to see the exact defense crossed, or even this year, having the portcullis or drawbridge block the view of part of the camera feed. So you add more camera. that works to solve those problems. say one more for each alliance. covering the parts that the others don't see? Now you end up having to allow for video mixing, feed switching, rewind replay, and watching multiple camera screens. That is extra time in each match review that you want to have. maybe say 30 seconds or a minutes? That’s reasonable since the entire point of this is to not miss any detail right? well we have now added at least another hour onto our matches. Where does that fit into the schedule.

Now well you have your 11-minute cycle times, your cameras, your all ready for match review. What’s the cost? well say $500 for a good camera. so $3000 for all of them. Let’s add $500, for wires tripods, and another $500 for the screens and controls so that the video could be watched. You also need a device to record the video. We can use a Tricaster 40 which is $5,000 on B&H. That says our video replay system starts at around $10,000 once you get everything you need. Well that’s not too bad for a video setup, it’s fairly low end, but probably all you need to watch the matches. And remember that needs to be on 20 fields per week. So that is $200,000 for just your video replay system. Let’s also remember that additional time and cost needs to go into training for use of the system, a volunteer spot to possibly man the computer to help the refs, and time and the resources that are needed to prepare the system and vet the options. Still think it’s reasonable?

Okay you want to argue that not every second of every match needs to be replayed? So, you want to play part of a match, you have to seek through the full footage to find those 10 seconds for review, and maybe you still watch it twice? well the time to find the clip, might be 20 seconds and then 20 seconds to watch it twice. so that's 40 seconds. 20 seconds of thinking about how to proceed (that would be super-fast for most people. Please make a decision in 20 seconds for me). and there we have an entire minute added onto game play. Or maybe refs have their own station to watch matches? Okay fork over an additional $20,000 per field for a more advanced streaming device. And we still have the concept of not watching every second of every match. Arguable if you are going to do video review, then it need to happen on every match, since if you only do it when you think you have missed something, or a team asks a question, then that's unfair, since what if a team did not realize the ref did not count a defense crossing they made? Well why should they not get those points and have another team get them since they were being picky? So that means every match gets watched in full. That’s the only way to make it fair for every team.

So you want to argue the time problem? reasonable. Teams can setup for a match while video review is going on. That works. except, it does not change that hard cutoff that it takes the field itself to pre start and run a match. You need to leave 3 minutes for that. and your field staff (FTA, FTAA, CSA, Scorekeeper) won't have any idea about what robots are having connection problems until the field has been pre started, and then fixing any problems could take a minute or two. so you won't save yourself much time here.

You also need to consider the technical aspect of running the equipment. In an ideal world, nothing goes wrong. But that never happens. Cameras will go offline, break, the video streaming boxes won't work, cables will die, balls will fly out of the field and knock over or break video equipment. What happens when something goes wrong. It’s not fair to give a team extra points for crossing a missed defense in one match, but in the next when it happens again, but the opposing alliance hit the camera making it record the ground during those 20 seconds, none because it "was not on camera?" that's not fair now is it. And don't tell me "it won't happen," because you know that it will, and it will be your team that lost points because of it.

All in all, Yes the idea is great. It works for Football, Soccer, Baseball, or other sports that run for 3 hour long games, since 5 minutes don't matter, it’s just a commercial break. but in the fast passed games of FIRST, it’s not the time and place, and in the end you will be doing more bad then good. Feel free to pick this apart, but take a chance to see why it’s really unpractical for this to work.

Re: Video Review Needs to Happen Now
 

Wait. Stop. Lets not do, what will be the third/fourth rehash of this.

A few events said they would offer replay counts for teams. Blake (and I) are looking for the data from those events.

What does the data show?

Commentingonly, you make great points, all of which have been posted before. But I'll give you points for a first post that has that much detail in it. For someone so new to FRC, I think it's very impressive.

Thanks

Re: Video Review Needs to Happen Now
 

Foster, I am not new to FRC I have volunteered in almost every aspect of events, was a student on a team as a critical part of build and drive, ane I help mentior 4 teams. You must understand that i did not wish to rehash the data, and information. I have read a good deal of this thread maybe missing a few posts, so i know its all be said. I (and i would hope that many other field staff members, refs, event coordinates, and gdc/first staff) cant understand how one would expect to achieve this. The reasons i listed are just a few of those that i could list to explain how this would end badly and become more of a stuggle to implement then it would achieve in the long run.

The data that needs to be analized is how long it would take you to walk around the field, seek through a video, find a clip that you want to watch, verify what you expect, play it for two other people so you can be sure, disguss it, and then make the adjustments. Thats process, it would take atleast 2 minutes to complete. and if your talking about recounting scoring, it takes no less then 2.5 minutes to watch a full match, and for something like this years game? thats absolutely required in order to confirm crossings. So it would garentee adding atleast 3 minutes onto any match cycle you have, and thats without debate over any other penaties or other field related problems.

I will say i do agree that every point matters and that missed points suck, but you can't guarantee that a ref won't get distracted. And it can be a disapointment to students and teams. I personally when i was a student drive team memver was on the receiving end of a few bad calls, but never once did we think the wrong call was made after talking with the head ref.

The only feasable solution, is making refs more accountable for their actions. This could be as simple as adding a scorer or additional refs in future years. But providing video replay would more likely then not creat more problems by allowing refs to say "we will just review this later" and stop watching scores or fouls and forget about what they wanted to review at the end of the match.

Re: Video Review Needs to Happen Now
 

We did not offer teams the chance to challenge via video replay at CCC (frankly, we had so much going on during the planning this year that we couldn't even think about it).

However, in the elimination rounds, there were two instances where we believe the match was scored incorrectly. We went back and looked at it on video to see and, if necessary, recount boulder scores and/or defense crossings. It made a significant difference having the video review.

In Finals Match 3, the final score that was initially posted showed that the blue alliance did not get a breach and, thus, did not earn the bonus points. Once the score went up I (Scorekeeper/FTA) immediately knew that it was wrong, the blue alliance was not given credit for the breach and red had won the match and, thus, the tournament initially. Others who were fieldside weren't so sure if they did get the breach or not. We replayed the stream back on one of the monitors and surely enough, the blue alliance did get the breach and not only that but their autonomous wasn't scored correctly either. The scoring adjustments were made and it put them over the red alliance and they ended up winning the tournament.

Not the best implementation, but it made a huge difference.

Re: Video Review Needs to Happen Now
 

While I agree with some of your points, I majorly disagree with some.

First of all with time. You seem to assume every single match will get replayed and frankly for lack of better words that's stupid.
Video replay will probably be used as much as the question box. A box used what? Probably less then 15 times per regional with at least half just being simple questions. Even if we ridiculously assume that every match is replayed in its entirety then lets go to your rebuttal about watching while set up is happening for the next match. You mentioned the match could not be pre started without the score being in and you are correct RIGHT NOW. If FIRST does implement video review I am sure they will make a few tweaks in FMS code and make it so the match can be pre started while video review was happening.

Now for cameras. I think video review could fix 99% of calls with 3 wide angle action cameras(gopros) costing around $900. You also mentioned a tricaster. While I love tricasters it is not needed. Something like a Blackmagic Atem would entirely suffice and you can get a nice one for $1,500. After that, there would need to be a streaming computer which Ill say costs $500 and then another $100 for cables. In the end that is about $3,000 per setup. So to outfit all 20 fields we are talking about $60,000. Now that may seem like a huge number but when you realize that its only 12 teams initial registrations it seems much smaller. Not to mention that setup can FINALLY make it so FIRST can standardize livestreams. Volunteer role is simple Livestream role. Almost all FIRST events already have livestreams and there is someone volunteering already unofficially to run it... many times without even being recognized as a real volunteer.

Now your point about equipment breaking I somewhat understand although your main center camera should be able to see 90% of what is going on, on the field. Not to mention this year is a major outlier. Any other year a single wide angle camera could catch 99.999% of calls.

I like you have done a ton of roles at an FRC competition. This year I have done field reset, been the scorekeeper, reffed, and run regional livestreams. I know video review can happen and I really really hope FIRST wakes up and it makes it happen.

Re: Video Review Needs to Happen Now
 

Thank you very much for your data point with the detailed description on how it worked out.

Question, how long did it take you to do the replay and make a decision?

and again, we've beaten this horse to death with opinions, still looking for the events that did it and the FACTS around it.

Re: Video Review Needs to Happen Now
 

Sam, you make some good points. Ones which I did consider. I have actually done work with video recording systems and have setup a similar camera setup to what your talking about.

First you suggest GoPros and an Atem. The Atem is a good piece of hardware, and would do what you need, for video mixing in a production environment. In this case however, you need something better. It only offers 1 video out which works for the audience video and webs stream, but does not allow you to record all 3 of the camera in your setup.

The second point that I would like to make is the GoPros, they are a good example for a camera for this. I personally have used them in this type of environment, and I believe the Mid-Atlantic had done this as well with their video system last year. The GoPros don't hold up in a streaming environment. They easily over heat when run for a long time. Getting them up and running takes a bit of work. They also did not play nicely with a black magic Atem that I used, and required multiple adapters (costing about $200 for each camera) to get them to work reliably.

Third, your video streaming server is severely under budget. A cheep black magic card, to handle HD video recording on one feed, costs around $200. Something to handle 3 or more recorded feeds starts around $300-$500. There goes your budget for that one, even if you get cheep hardware, you are still looking at around $1000 for that setup. You can get cheaper hardware, but I would not expect it to hold up well during full event.

I should also mention about camera setup. I have tried to place camera around the FRC field to effectively cover the full field. Its hard especially this year, and don't forget last year too. Once your stacks got above 4 high, it blocked robots, chutes, human players, and drivers. There was no way to ensure full video coverage over this, or last years, games with only 3 cameras. Also i would not say that its over yet either. With the increased production value of this years game, and the looks that it will be carried over to next year, I think we will see another low visibility game in our near future.

Second, i do agree that my every event gets video replay is a little crazy, but I don't think its out of the question. And yes, the question box is under used right now, since often there is nothing that can be done to re score matches. In my years of first, I would expect that if you watched match replays, then you could re-score about 50% of matches based on the video recordings of them. It happens that a ref is watching a robot for a penalty while another that they are not watching scores points, yeah this can very easily get over looked. I as a field staff member have caught myself focusing on the movements of a single robot for long parts of a match and ignoring the others. The game is exciting and that's really easy to do when your 3 feet away. You said it your self a few posts ago.

How many matches would you have wanted to replay at that off season event? Are you telling me that you trusted every score you entered and would have only done match review is someone came to the question box? and even then, are you telling me that one team should get their match re-scored just cause they came to the question box? If you do it then, it should happen always in order to make it fair for all teams.

I do agree with you that its a great idea. Video replay would be a handy tool for refs when scoring matches, but i don't think there is a good implementation that would not make more of a hassle for teams and volunteers. There would be countless delays that this would create and i don't think any event could accurately predict its own running time with this as an added factor.

Re: Video Review Needs to Happen Now
 

Any reason you felt the need to make an anonymous account for this thread?

Just curious :)

Video Review Needs to Happen Now
 

I'd say it took us around 5-10 minutes to check everything, decide, and edit the scores. That's only because we didn't really have an actual system in place and we were just playing back the webcast. If we actually planned out and came up with a process, dedicated equipment, etc. it would probably take significantly less time.

Re: Video Review Needs to Happen Now
 

Wow,
This is such a bad idea for so many reason, forget the logistics of it: technology required, time and energy, etc; it puts the emphasis on the wrong part of the event: who won. We need to take the best of the sports model, not the entire thing.

Re: Video Review Needs to Happen Now
 

Another possible data point for you all....

It may not be necessary to re-watch an entire 2 1/2 minute match to determine if an error occurred.

At our off-season event we had a situation where in the last ~15s of our first quarterfinals match our alliance damaged 2 defenses (2 bots nearly simultaneously) and breached before barely making it to the batter. Prior to crossings, the lights on the defenses both indicated one crossing had been completed, but prior to end of match (when they all went off) only one had been scored as damaged. The failure to score the 2nd crossing and breech was the difference between a win and a loss.

We had no replay (or webcast) in place but we sent our student to the ref to object, to no avail. All in all it took almost 3 minutes between quarterfinals matches to have the discussion and return to the queue.

When we reviewed match video later, we only needed to review the final ~30s of the match, enough to see the lights on the defense prior to the crossing and that two crossings occurred, confirming the scoring error.

To those that say we should focus on the positive of the game, and not emphasize winning or losing, I offer this observation. Coming out of the match we lost, the mentors of all three alliance teams attempted to do just that. In fact the closest to success we had was pointing out that we get to play an extra match as a result (knowing we'd win against this alliance). However, until you stand there with ~15-20 students who all know they took a loss due to a clear error, it's difficult to understand the severe lack of "inspiration" that results.

(To be clear, I hold no ill-will or resentment of the refs at the event -- I strongly believe they did their level best within the rules FIRST laid out, and it was a fun competition overall. Also, these options are mine, and I do not speak for my team.)

Re: Video Review Needs to Happen Now
 

Partially because, I don't actually have another account otherwise. (long time reader though.) And partially to abstract away my background, and just focus on the information that I have to provide as a technical professional and my experience as a volunteer.

I think your right that the time could be improved, but for something such as a re-scoring a match, you can't do that in less then 3 minutes. and that does not include the time needed to verify that you entered the right scores and everyone agrees. Let alone if you wish to analyze other aspects of the match on video as well.

I also as a alumni and mentor, have to agree 100% with wilsonmw04. Yes, winning the event is a great experience. As someone who never made it to finals while on a team, I hope that every student has the chance to get there, but FIRST is not about winning, its about learning and spreading STEM, business and professionalism. In college and business, you can't go to your professor, boss, or a client and tell them that they did something wrong and expect them to re evaluate a grade or business decision just cause you have evidence supporting your case. As a free lance software developer, I have lost bids, and when seeing the final product produced by the competition, known that I should have won it, but I can't change the clients mind when they made the decision.

If teams members think that matches are reffed poorly, then those teams should be providing mentors to volunteer as refs so that they have someone that will do a better job, and at that point you might see that its not as easy as you think to make the calls they do.

Re: Video Review Needs to Happen Now
 

I know this is another Opinion / Hot Take post, and it does echo some themes in the thread, but essentially the problem instant replay solves can be solved in other ways - but instant replay is probably the only way that we as event planning volunteers can make an impact.

The "right" way to solve the problem is in game design. While I respect and understand the many constraints the GDC are under in the game design process (and thus this whole paragraph is easier said than done), there are certainly changes to the game rules that would make refereeing more fair. Removing tasks that are scored subjectively by humans in real time is the #1 change that can be made. Assists in 2014 and crossings in 2016 are two perfect examples. To some extent these get better if the human scoring is dedicated solely to the task and focused on a small area of the field, but in both years this task was spread out over several positions and could happen at the same time in multiple areas. Other areas of rules ambiguity could be tightened up and made either more objective or removed entirely. There is a tendency for the GDC to "patch" holes in the game design with specific and excessively subjective rules to cover for a variety of convoluted situations, and that leads to a lot of these problems.

The thing is though, we have no pull on the GDC, and no opportunity to change how the game rules are written whatsoever. So we can't solve these problems the "right" way. We can ask and hope, that's it. If we want to get more calls right, in games where calls are done like this, instant replay is worth exploring. The way it's been done at various offseasons is great - let's keep trying stuff at offseasons to balance the constraints between volunteer requirements, equipment, rules interactions, etc. What we don't need is an essential moratorium on even considering the slightest change to a broken process from people used to the status quo. Let people try it at off-seasons! If it is truly doom and gloom as it is made out to be that will become obvious very quickly, and if it's not, we learned something.

Nobody in the entire thread wants video replay because they believe refs are incompetent, not trying hard enough, or biased. There is universal recognition of the difficulty of the task of refereeing. This is not a matter of people going "oh, now that I know being a ref is Hard, I won't complain anymore" - because what comfort is that to a team that's season is over on a blatant missed call that everyone can see but no one can change? You guys say "it's not about winning", which is really easy to say if you get to the Championship every year anyway, but the fact is in FRC winning is more than symbolic - it creates the future opportunity to compete and be inspired. As long as qualification is merit based, winning, and getting the calls right, will absolutely, tangibly matter. We should try and get it right.

Re: Video Review Needs to Happen Now
 

GRC had one instance of it being used by a team that wanted to make sure all of the high goal shots were correctly counted. As the next match set up, the head referee sat down with me at the webcast table and we pulled up the previous match video. We re-watched the match and the head referee was able to easily count each ball being scored in the high goal. Afterwards they confirmed it with the team and that was it. There was no delay to the event or anything event-related. Overall I thought it was a success.

At least to your first points - technology and energy required are basically the same if you're already webcasting the event. See my post from another thread on the technology aspect - https://www.chiefdelphi.com/forums/s...0&postcount=30 Don't make a mountain out a mole hill on the technology piece.

Re: Video Review Needs to Happen Now
 

Off topic a bit but this doesn't work out in my mind. Someone in college can correct me if I am wrong but if a professor made a mistake grading a test that caused hundreds of kids to fail and you alerted them of it you are telling me they wouldn't fix it?

Re: Video Review Needs to Happen Now
 

Seriously. I would very much like to see this implemented as: 1) improve the webcasts to Make It Louder 2) experiment with replay methods behind the scenes, 3) go from there.

We all agree that the system is not perfect. We're likely to disagree about the proper effort to devote to various improvements. But this desire to shoot down any potentialities before even investigating their manifested difficulties is very aggravating. The biggest obstacle to experimentation is the technological investment, which largely piggybacks off other vast improvements that go directly to public FInspirationRST. Then experiment with how much different cameras help, how to integrate and navigate feeds, how to handle FMS and turnaround issues, etc, behind the scenes. Then handle actual implementation and restrictions thereon.

I also take serious issue with the "it's unfair if it's not X, and you definitely can't do X" strawmen. The status quo is unfair; insisting that an improvement become perfectly fair is unreasonable. Calls will still be missed whether replay is automatic for every second of every match, available throughout quals, available only by challenge in elims, available only to the head ref, what have you. The fact that calls will be missed doesn't mean missing fewer in some systematic way is equivalent. (Though you're of course free to argue it's irrelevant to the goal of FIRST, which is at best going to land on agree-to-disagree again.)

I've been both a grader and a student in this situation, and it's a black-and-white call, it's far rarer that it won't be changed. And I've of course also corrected my own bosses with evidence; I do not understand this argument. If the evidence is ambiguous they may debate it or ignore you, but if it's a cut-and-dry equivalent of "they crossed the defense"--and there's still a way to correct it--every good boss I've had will. And I certainly don't hold the initial call against them if it was reasonable and they correct it responsibly. In fact, the whole thing is very analogous to the way providing video evidence in any situation should work. And I'm saying this as a veteran referee. I have reservations about replay, but that's certainly not one of them.

Re: Video Review Needs to Happen Now
 

You are both right and wrong. Some professors do, but others don't. I have had more then one professor who, refused to regrade assignments due to their own error. It may seem a little silly, in concept, but often times it relates to needing to regrade the entire class or its a project, where the final grade stands as is and they don't want to make the page. In one instance, the professor did offer to, when my grade was in correctly assigned due to their error, but when regrading they ended at the same score finding other reasons to deduct points. So yes, in some cases, but its not as easy as you would think to get grades to be changed or re evaluated.

Re: Video Review Needs to Happen Now
 

Chris,

I broadly agree with your comments about game design, but I agree far less with the assertion about off-season event hosts being left with (probably) only video replay in their bag of tricks for making things better.

Off-season events can and do change rules every year, and not just the rule about using video. Changes other than video are definitely on the table.

One option that doesn't require changing any game rules would be to simply scrounge up 6 volunteers from the crowd and give them the job of watching one robot apiece (there are details to be dealt with, but you get my point).

[/EDIT]
When I wrote the two paragraphs above, my brain was stuck in a thinking-about-off-season-events/experiments rut; and it just dawned on me that Chris was almost certainly thinking about regular-season events. Doh!

That said, the bigger picture point is that if off-season experiments such as getting teams attending events to supply a small handful of students/adults to do a few low-skill scorekeeping tasks (or rule tweaks, or ...), create a dramatic error reduction during off-season events, the GDC would probably notice (notice both the errors and the simple(ish) way to treat the root cause's symptom).
[/EDIT]

I'm not saying in this post that using video evidence it's bad or good, wise or foolish, etc.

I'm also not saying in this post whether or not I think there is a difference between inspiring someone to try something new, and that person later on being excited or depressed by they way an FRC competition unfolds.

I am saying that I don't think video is the only lever that can be pulled.

Now, I'm going back to waiting for the "hard" data posts.

Blake

PS: The one time I got to spend some time with an FRC GDC, they seemed like nice people ;). I think they would welcome well-organized feedback from event organizers; especially if it took the form of a video-based, post-mortem of a game's rules.

I'm thinking about the sort of review and analysis that would use a large number of hours of video from multiple events to identify the sorts of calls/rules that are hardest for humans to make/enforce correctly.

That sort of info could definitely influence future games (and treat a cause instead of a symptom), especially if could be put into a simple checklist of things to avoid, or do.

Maybe a pro-video person reading this thread will contact the GDC in order to volunteer to do that for the next 1-3 seasons?

Re: Video Review Needs to Happen Now
 

Uh, are you basing that just on your experience reffing an offseason, or do you have some other source? I've seen lines at those boxes in the regular season! I'd put it at a minimum of 1 match in 4, somebody is in the box, average over the event (more during playoffs, mind you).

ONLY for equipment operation. If that person makes the call--and makes it wrong--you know that HQ will be swamped with complaints about how instant replay inhales audibly. Any calls resulting from replay need to be made by the ref crew, particularly the Head Ref.


One other data point: I did once review a replay at an offseason--the webcast, to be exact. I won't go into the details, but what I'll term a "integrity of the tournament" question was raised. A quick look at the right place, and the question was answered to the tune of "integrity of tournament not affected". Had to wait a match or so to get the webcast set in playback mode, though.

I'm not against replays, per se. I'm against unnecessary use and excessive use, as well as use without proper equipment (read: without decent video). Basically, what that boils down to is that if the replay is available, it is necessary to use it to confirm a specific non-judgement call, and the team-requested use is kept to a reasonable level, great, provided that it's legal for the event in question.

I also agree with Siri on a very key item: If it is clear that something wasn't done properly, and the mistake is caught, it should be fixed. See also: Question box.

Re: Video Review Needs to Happen Now
 

Great idea in theory. In practice, I see this as only a partial solution. One cannot assume the field hardware works perfectly. Such as in 2014 with the hot goal indicators not flipping during auto once in a while. The head ref made the call whether or not the flippers worked (and if not, called for a replay of match). That's an automated system with a subjective element; head ref's are only human, and I could see one missing a missed flip (especially if it was only one one side) and the resulting questions after the match from the alliances.

When a field scoring fault is suggested, things become subjective by default. Yes, there are places where game design can reduce the number of subjective calls ref's have to make, but subjective calls, at least by the head ref, are unavoidable.

Re: Video Review Needs to Happen Now
 

You know, it's really hard for the game designers--and the refs.

If everything is black and white/objective, everybody complains that they got screwed over because there should have been some leeway/room for judgement (or the ref missed the call), and that everything should be a judgement call.

If everything is a judgement/subjective call, everybody complains that they got screwed over because the refs' interpretation of the game rules was bad, and that everything should be black and white.

And if the two are mixed, everybody's complaining that the objective calls should be subjective, and the subjective should be objective, and the refs still miss/misinterpret everything, so the teams are screwed over!

:p:rolleyes:

Oh, and then someone is bound to bring up instant replay as the only cure-all (instead of what it actually is, one possible tool in the toolbox full of solutions to issues that some teams don't even realize exist). Cue everybody repeating their statements from above paragraphs, followed by debates as to feasibility/fairness/volunteer POV.

Re: Video Review Needs to Happen Now
 

What would hard data for this subject look like to you? Most implementations will only have anecdotal evidence at this point because video review probably only gets used once or twice during an event (so far.)

Re: Video Review Needs to Happen Now
 

That lack of usage is a valid data point.

Re: Video Review Needs to Happen Now
 

What's interesting is I know some argued it would be used so much that it would slow events down, or that would be a risk at least, correct? So far it seems that is not the case for the few events that have tried it out.

Re: Video Review Needs to Happen Now
 

Have any events actually had an overturned call yet? There was one rescoring that was mentioned, but otherwise no event has had something overturned or adjusted. The primary concern is that most video review overturns would result in replaying matches, and thus the added schedule risk. It's not fair to penalize alliances for their strategic behavior based on the score/field conditions that are presented to them in real time, and then go back and adjust those via video review. For instance, if a team crosses a defense 3 or 4 times to damage it, they can't get the extra time they wasted crossing that defense back. Or if a team makes a call to ensure a capture at ~15seconds rather than scoring an extra couple balls based on their real time scoring, then have that scoring adjusted when you give the other alliance a breach that didn't exist previously. Virtually any case where errors are found in video review should mandate a replayed match. Thus the schedule concern.

Re: Video Review Needs to Happen Now
 

TL, DR: A) Ask the right question(s). B) Measure the baseline. C) Understand the relationships. D) Tweak the independent variables. E) Measure the results. F) Decide what changes/(re)allocations, if any, to implement.

- - - - - - - - - - - - - - - - - - - - - -

To prep to answer this, in addition to shooting from the hip ;) , I wanted to refresh my recollection of what has been said so far in the last few months. so, I reviewed this thread, and an adjacent thread. I found these posts I and a few others wrote. There is nothing Earth-shaking in them; but they supply some context.

https://www.chiefdelphi.com/forums/s...0&postcount=64
https://www.chiefdelphi.com/forums/s...0&postcount=72
https://www.chiefdelphi.com/forums/s...&postcount=182
https://www.chiefdelphi.com/forums/s...&postcount=193
https://www.chiefdelphi.com/forums/s...&postcount=207
https://www.chiefdelphi.com/forums/s...&postcount=223
https://www.chiefdelphi.com/forums/s...&postcount=265
https://www.chiefdelphi.com/forums/s...&postcount=268
https://www.chiefdelphi.com/forums/s...5&postcount=11
https://www.chiefdelphi.com/forums/s...2&postcount=13
https://www.chiefdelphi.com/forums/s...2&postcount=17

For me, the outline that follows is the way I would want to approach A) creating a solid understanding of the need (or lack thereof) for adding video to the refs' tools, and B) coming up with a first version of a video system, if developing one is warranted.

The "hard data" would be the results (measurements & statistics) produced by the experiments.

Obviously this is a back of the napkin, discussion-forum-quality sort of an outline - Not even PowerPoint quality yet.

The current system (FIRST) being discussed is a system containing many things, including competition events that contain, at the least, a Playing Field & Game Pieces, the Field Staff (announcers, refs, etc.), the participating Teams, the Match/Game/Robot rules, the Audience, the Matches/Schedule, and the field Computers/Sensors/Software.

We are talking about introducing Video Replays into the FRC (and FTC ...) event part of that FIRST system.

We need to know the pertinent parts of the current baseline system's status/performance, the current system's purpose, and the sensitivity of the system's ability-to-achieve-it's-purpose(s) to changes in the independent variables we are going to adjust.

Some useful metrics might be

• Call accuracy
• Current call challenges outcomes
  • Was a call changed
  • Was the result accurate
• Match outcomes affected by call accuracy and by challenges
• Event outcomes affected by call accuracy and by challenges
• System-purpose outcomes affected by call accuracy and challenges
• Calls (i.e. rules) that could/would/should be affected
  • Perfect video
  • Less-than-perfect video
    • Video usefulness vs equipment performance/placement
    • Equipment purchase costs vs equipment performance
    • Non-equipment-purchase costs: time, labor, maintenance, shipping, training, etc.
  • Video-alternatives (more humans, rule/game changes, anything else?)

In the experiments I would want to

• Compare and contrast off-season events with regular season, and various championships to identify how they are alike, and how they are different; and what effect that has on the results collected in each type of event.
• Compare and contrast multiple locations, times-of-day, levels of human-training, etc. as part of trying each candidate method for accomplishing the event's purposes (and sub-purposes).
  • These locations would include change-nothing "control" events.
• Compare and contrast multiple years' (multiple games') results
• When testing each/any alternative, determine "truth" (the baseline data) by analyzing data (not in real time) collected by a plethora of sensors (these are separate from the sensors/methods that are being tested, and the "truth" is not shared during the event).
• In some circumstances (off-season events?), purposefully stress each alternative by having teams challenge calls in bursts and/or continuously. Do this to stress-test the alternative, not because of actual disagreements.
  • If necessary have teams create difficult-to-assess situations, so that reviewing calls requires more than a trivial glance at the video records or other evidence.

What's above is a quick-and-dirty outline of what I would *want* to do to produce "hard data". After dealing with real-world constraints, thinking a bit more deeply, and getting some preliminary results; I, or whoever, might decide the experiments could be simplified without violating the integrity of the results, or they might add something.

I know there are folks who firmly believe that the need for (or cost of) video reviews is/isn't so obvious, that what I outlined above isn't necessary. I don't disagree that they feel that way. I do say that nothing in this thread so far *proves* that the need does/doesn't exist, and/or that a need would justify the investment (instead of investing in satisfying other needs).

Blake

PS: In the past, I and at least one other person have wished for detailed camera/lens specs and placement info. That would be one example of "hard data", and could be used to answer some important questions; but it's just one part of the bigger picture under the heading of "Video Review Needs to Happen Now".

PPS: Above I have some bullets about identifying which calls could/should/would be affected by reviewing video. Complementing that, I'm not sure whether deciding what the effect of a changed call should be, is part designing each/any experiment (it probably is). Regardless, it is certainly something that would factor into any decisions to introduce (or not) video replays into the system.