Instant Replay Challenge—A Thought Exercise

[EricH]

Every few years or so, someone comes out and says something to the effect of “*Team* got screwed by a ref call, we have video to prove it, but the refs won’t look at it or review the call. We need instant replay.” This is inevitably followed by a chorus of “too much time”, “refs are volunteers”, “deal with it”, “costs too much”, and other objections, which is promptly countered by a “No it doesn’t”, “Refs need to be professional too”, and other counterarguments. Gets everybody nowhere, unless you count semi-heated discussion as somewhere. And it’s getting louder all the time, or seems to be.

In one thread back in 2005, however, a challenge was laid down and not picked up. Quite simply, figure out instant replay for an offseason, and if that could be done, within parameters, implementation would be at least seriously considered. As mentioned, nobody picked it up.

So, I hereby challenge all comers to an Engineering Challenge! Should you choose to accept, your task will be to develop an instant replay system. (Use cannot be guaranteed, particularly at official events, but if organizers of said events are interested, then they will at least know where to look.)

So what’s the catch? Just the following list of requirements.

Must be cost-effective. 
	Low cost is good. Free is better.
	Better value may be a reason to spend more.
Must provide sufficient quality of view to make a conclusive call. 
	Camera quality—should be enough to see whatever a ref would see if he were looking that way.
	Camera locations (and quantity)—should be enough to see whole field at any given time. 
	---Multiple angles preferred but not required.
	Operator training—zoom, what to look for, etc.
Must provide a system for viewing camera recordings promptly when needed.  
	Quality to match camera(s)
	Ability to view multiple cameras if needed.
Must have a System of Use for Challenges
	Limit number of challenges/team or alliance (to limit abuse of system)
		Collateral?
		Fixed number?
	Limit time spent in video challenges
	Should not impact “referee chooses to see video when team does not request it”
	Decide how elim/qual difference should happen—carryover, fresh slate, total by alliance
Must have a method to keep video unbiased
	No audience video
	All questionable calls reviewed can result in calls on anybody, including challenger

Please note that this is merely an exercise in designing a system. Plausible systems to solve a perceived problem is how an engineer ought to be thinking, and many of the people here are looking into engineering, so consider this good practice.

[joelg236]

Very cool idea. My immediate reaction is to say that it should be divided in two sections:

1. Hardware - This includes everything from camera placement to wiring to how video streams get streamlined into one output.
2. Software - How the video can be brought into one central stream for live streams / big screens, while still storing that video in a way that it could be accessed very quickly without hassle.

How could you make such a system easy to use for operators, low cost (ie. not 100 cameras everywhere) and still good enough to be useful? It's an interesting project. Maybe the software side could be open-sourced and if there is enough interest the talented people in FRC could figure something out. I imagine hardware would be the biggest challenge, and I'm not even sure where to start. Every venue has a different layout with different ways to set up, and having to individually plan each one specifically would be a huge hassle. And I'm probably not even thinking of the biggest challenge, which would be making the video "unbiased".

Even if some kind of system was made that worked really well, I have no doubt people would still complain. "The camera just wasn't focused on us when we got hit" type deal.

Just some initial thoughts, I'll have to sleep on it.

[EricH]

Originally Posted by joelg236

And I'm probably not even thinking of the biggest challenge, which would be making the video "unbiased".

I'm thinking that having a totally and completely unbiased system will probably be pretty much impossible, partly because just about everyone has a bias one way or the other, to a greater or lesser degree. However, it's certainly a good goal to work towards; FIRST already applies countermeasures (refs, judges, and inspectors can't ref, judge, or inspect their own teams, or if they do, their lead knows that they're affiliated with that team), so maybe just having those will work. Or maybe not. YMTC* on that.




*You Make The Call

[joelg236]

Originally Posted by EricH

FIRST already applies countermeasures (refs, judges, and inspectors can't ref, judge, or inspect their own teams, or if they do, their lead knows that they're affiliated with that team), so maybe just having those will work. Or maybe not. YMTC* on that.

As far as I'm concerned, the system works close to as good as it can. Others would probably disagree, but I don't see how it could get less unbiased apart from AI (uh, sure...). Rules are a different story (there could always be more specific rules, which I'm a fan of), but in terms of an "unbiased" judging system for calls, even professional sports that have millions of dollars behind them still use refs.

[the programmer]

Here's my thought, have two cameras mounted on opposite ends of the field as shown in this video:



Then you would have a volunteer who would start and stop the cameras every match to create separate videos that would be ready for viewing at the end of the match.

At the tournament, teams would receive in their registration packet a replay slip with their team number printed to use during qualification matches. Teams may ask for a video review within a minute of the final score for their match being displayed. The only things that may be review are calls that change the winner of the match, red cards, yellow cards, and a robot being disabled by rule. If a team wants a review, they hand their slip to the head referee and he reviews it, if the team is correct, the call is reversed and the team receives their slip back; if the team is incorrect, the head referee keeps the replay card.

The same process is used in elimination matches except that each alliance will receive one slip marked with their alliance's seed and "Elimination."

The only downside to this is that there might be instances where a match would have to be replayed because of a missed call but that's still better than the incorrect call remaining.

[CalTran]

Sounds like the setup that 2337, but also replicated on the other side (or each corner?) wouldn't be half bad. The quality and wide angle lense they use on their pole'd gopro is great for watching at home.

[Garrett.d.w]

Originally Posted by the programmer

Here's my thought, have two cameras mounted on opposite ends of the field as shown in this video:



Then you would have a volunteer who would start and stop the cameras every match to create separate videos that would be ready for viewing at the end of the match.

At the tournament, teams would receive in their registration packet a replay slip with their team number printed to use during qualification matches. Teams may ask for a video review within a minute of the final score for their match being displayed. The only things that may be review are calls that change the winner of the match, red cards, yellow cards, and a robot being disabled by rule. If a team wants a review, they hand their slip to the head referee and he reviews it, if the team is correct, the call is reversed and the team receives their slip back; if the team is incorrect, the head referee keeps the replay card.

The same process is used in elimination matches except that each alliance will receive one slip marked with their alliance's seed and "Elimination."

The only downside to this is that there might be instances where a match would have to be replayed because of a missed call but that's still better than the incorrect call remaining.

I like this approach.

I wonder if it would be possible to record the raw video of the stream and use that for your replays.

[Siri]

Originally Posted by CalTran

Sounds like the setup that 2337, but also replicated on the other side (or each corner?) wouldn't be half bad. The quality and wide angle lense they use on their pole'd gopro is great for watching at home.

I wonder what the view with this looks like from the top of the high goals. (Or the top of the pyramids?) That could remove the difficulty of planning different placements for each venue. And while the pyramids aren't perennial, the height of the alliance walls never gets crazily low.

[Tristan Lall]

Although I'm not opposed to a structured implementation operated by the competition, I think it can be done differently, and with most of the (small) costs borne by the teams. As I proposed in that 2005 thread, an off-season event should try giving teams one instant replay coupon for the qualifying rounds and one for the elimination rounds, and then providing a brief window to present video for post-match review. This is practical and equitable.

A student alliance member would go to the challenge box and remit the coupons of all alliance members that showed up for the match (the whole alliance must agree it's replay-worthy), along with whatever video they want (cued to the crucial moment), within 1 min of the end of the match. The head referee will see the challenge is being made, and arrive to look at it (if the referee takes longer, then great, you have more time to cue the video). First, the referee decides if the video (as shown on the playback device) has enough information to show the situation at issue, then decides whether it changes the outcome, and finally, decides whether the evidence is sufficient that the video has not been tampered with or misrepresented.

If it fails the first test, the referee puts an end to it within moments. (For example, if there's not enough detail, or it doesn't show the right part of the field.) If it fails the second test, the team then knows it's not just the referee missing the call, but rather a deliberate decision to call it a particular way. (Rightly or wrongly, at that point it just becomes another futile challenge.) The third test may be a little more complicated—possibly requiring rewinding the video to hear the match being announced, or to verify that the right robots are on the field—but it's a straightforward test of credibility. (And because the burden of proof is on the team, and the timeline is tight, there's really no concern about the wrong video being reviewed, at least in the qualification round. In the elimination round, with the same teams playing, identification is more difficult, but at least the schedule isn't usually as constrained there.)

Why this? Because it strictly limits the number of reviews to minimize the aggregate delay, but still gives teams the opportunity to be heard. (Teams like it better when the officials hear them out, instead of ruling summarily.) It provides the referees with a procedural reason to change a bad call, without appearing indecisive, and gives the head referee a tool to better manage the rest of the referee crew (by understanding which calls are being blown). It also serves the teams notice that they share in the responsibility—if interested in challenging the officials' version of events, plan ahead to ensure that you're making video.

It's not perfect, but it doesn't have to be. I think you'd even see teams collaborating to supply post-match replay video to each other.

[coalhot]

Originally Posted by Tristan Lall

Although I'm not opposed to a structured implementation operated by the competition, I think it can be done differently, and with most of the (small) costs borne by the teams. As I proposed in that 2005 thread, an off-season event should try giving teams one instant replay coupon for the qualifying rounds and one for the elimination rounds, and then providing a brief window to present video for post-match review. This is practical and equitable.

A student alliance member would go to the challenge box and remit the coupons of all alliance members that showed up for the match (the whole alliance must agree it's replay-worthy), along with whatever video they want (cued to the crucial moment), within 1 min of the end of the match. The head referee will see the challenge is being made, and arrive to look at it (if the referee takes longer, then great, you have more time to cue the video). First, the referee decides if the video (as shown on the playback device) has enough information to show the situation at issue, then decides whether it changes the outcome, and finally, decides whether the evidence is sufficient that the video has not been tampered with or misrepresented.

If it fails the first test, the referee puts an end to it within moments. (For example, if there's not enough detail, or it doesn't show the right part of the field.) If it fails the second test, the team then knows it's not just the referee missing the call, but rather a deliberate decision to call it a particular way. (Rightly or wrongly, at that point it just becomes another futile challenge.) The third test may be a little more complicated—possibly requiring rewinding the video to hear the match being announced, or to verify that the right robots are on the field—but it's a straightforward test of credibility. (And because the burden of proof is on the team, and the timeline is tight, there's really no concern about the wrong video being reviewed, at least in the qualification round. In the elimination round, with the same teams playing, identification is more difficult, but at least the schedule isn't usually as constrained there.)

Why this? Because it strictly limits the number of reviews to minimize the aggregate delay, but still gives teams the opportunity to be heard. (Teams like it better when the officials hear them out, instead of ruling summarily.) It provides the referees with a procedural reason to change a bad call, without appearing indecisive, and gives the head referee a tool to better manage the rest of the referee crew (by understanding which calls are being blown). It also serves the teams notice that they share in the responsibility—if interested in challenging the officials' version of events, plan ahead to ensure that you're making video.

It's not perfect, but it doesn't have to be. I think you'd even see teams collaborating to supply post-match replay video to each other.

This is a really good suggestion/setup. I think you should add one more thing though (this has to do with the way the ref calls it though). Do it NFL style. There must be enough video evidence to clearly and without any doubt contest the call on the field. If there isn't, the ref shouldn't bother reviewing the footage. I realize this relies on a lot of interpretation, but still.

A great example of this was at Hatboto-Horsham. A friend of mine took a photo of a robot that made a 10 point hang, but appeared to have a ziptie sticking out of the bottom contacting the floor. At first glance, it seemed to be assisting the climb. However, when we took a few harder looks at the photo, and took a look at the robot (to satisfy our own curiosity), it looked like the ziptie was hanging about an inch above the ground and not actually touching the carpet. It's still hard to tell looking at the photo. We also casually asked a mentor in the pits about that ziptie, who stated "Oh that? I think someone forgot to cut it". This is something that could bog down the system, and is one of the reasons why the NFL has such a strict policy for their instant replay.

I know the NFL has a huge budget for the instant replay system. But, for FIRST, we don't need that. The burden is on the team/s challenging to have clear and decisive footage of their robot and the match. If they don't...well tough luck. This may also increase the number of teams recording/photographing matches at higher resolutions and better angles. A win all around. The only constraint: the whole challenge cycle needs to fit within the ~6 minute field reset time, and not take up all of the head refs time. If that can be satisfied, it would be a viable solution.

[EricH]

The problems I see with team recording are these:

1) The cameras will by nature be focused on one robot. This is not necessarily a bad thing, mind you--but tracking an object at zoom from 40+ feet away isn't easy if it's moving fast. This is where I'd prefer having a trained operator.

2) Time to get the film down to the field after the match. At an offseason, 1 minute is reasonable, if a bit of a stretch when cue time is factored in. At any event IRI or official, add 1-2 minutes for film to arrive. Or have a "Team Filming Area" (which would also address my third concern, below). Review time to determine if you caught something, 1 minute goes out the door.

3) Obstructions and bouncy cameras. Somebody pops up to cheer...right as a challengeable call happens. Maybe it's the camera person. Either way, lousy video quality. A team filming area would limit that.


Brain wave to deal with a few of these things...
What if every team was asked to have someone volunteer to record their robot every time it played? I'm sure there would be no shortage of volunteers for that one. However, each official replay volunteer would get a short training in "what to look for" (staying on the robot, appropriate zoom levels, etc), maybe 20-30 minutes long--this also gives a chance to check equipment for quality settings. Then, six to twelve spots around the arena are designated as "film spots" where the replay volunteers can set up cameras. Start when the MC begins the countdown, stop when the head ref gives the all-clear, head for the field in case of replay challenge.

For timing, it should vary based on venue layout, but say a team has 1 minute to initiate a video challenge, and then has 1-2 minutes to get the video to the head ref, cued up and ready (variable based on whether we're dealing with a high school gym or the Edward Jones Dome). Head ref watches the segment, reaches decision, and it's announced after the next match.


Edit: There is one other thing that I would suggest. If an event were to use instant replay, the head referee should have the ability to call for a video review, whether or not a team requests it. If the head referee calls without the team requesting a video review (say, the team asks about interpretation and the head ref decides that he wants to take a quick look again, or the ref huddle can't agree on how bad a foul is), no penalty to teams' ability to call for a video review later.

[Clinton Bolinger]

Originally Posted by CalTran

Sounds like the setup that 2337, but also replicated on the other side (or each corner?) wouldn't be half bad. The quality and wide angle lense they use on their pole'd gopro is great for watching at home.

Thanks, also in the off season we looked at trying to put multiple cameras (GoPros) on the field. There was one camera at center field, one looking at the blue scoring zone and the third was looking at the red scoring zone. But the issue we ran into was the extra cost and finding an easy way to record all three camera feeds at the same time on one machine.

You can see some of the testing on our YouTube channel from IGVC, the lighting might be pretty bad because we play under a tent.

http://www.youtube.com/playlist?list...-Hkajabm2iK6wK

The corner shots looking down were my favorites after the center field point of view.

-Clinton-

[Racer26]

Originally Posted by EricH

FIRST already applies countermeasures (refs, judges, and inspectors can't ref, judge, or inspect their own teams, or if they do, their lead knows that they're affiliated with that team), so maybe just having those will work. Or maybe not. YMTC* on that.

*You Make The Call

It doesn't really matter too much for this conversation, but I can't recall a single FRC event I've attended where at least one of the refs, judges, or inspectors were not affiliated with a competing team.

Dunno if that's limited to Canada or what, but people affiliated with 1114, 188, 610, 1547 and others form a large percentage of the volunteer base at the Canadian regionals. I strongly suspect this is true of all regions, that the strong teams in the area form a large portion of the volunteer base. Something about being passionate about FIRST.

EDIT: To avoid confusion, I don't think this is a problem, and I think the quality of the events would plummet if we had to get unaffiliated volunteers, if the events could even get enough volunteers to run at all.

[Steven Sigley]

Originally Posted by Clinton Bolinger

You can see some of the testing on our YouTube channel from IGVC, the lighting might be pretty bad because we play under a tent.

http://www.youtube.com/playlist?list...-Hkajabm2iK6wK

The corner shots looking down were my favorites after the center field point of view.

-Clinton-

What if you were to have one in each corner looking down (so you theoretically have 2 angles on any possible penalty) And had 4 computers lined up to show the feed from each one. The potential is there...

[Nawaid Ladak]

I like a lot of the ideas coming out of this thread already. Here is my suggestion.

+ 1) Implement a system where all video streams from the event cameras (not the just the footage that is shown on the big screen, but all the footage that's being shot off the other cameras as well) is recorded and stored. These videos can be deleted prior to the start of the next match. This shouldn't take more than just some software with an external Hard Drive/USB Flash drive.

- 2) FIRST gives out Media badges at the Championship Event. lets make those badges useful. These six individuals' videos (six teams per match) will be the only outside video sources permitted when a challenge is presented by a given team (along with the footage captured by the event cameras). Thus, the Media badge is no longer just a field pass, The person with the badge has a responsibility for his team. Maybe even designating spots for these individuals off the field would also be a useful idea. Implementing this at the various regional and district events could be a challenge.

- 3) I really like Tristan Lall's idea on all three alliance members giving up their challenge cards if they wish to challenge a call on the field. I think this would make them much rarer during qualifications. However I do think there should be a dedicated replay official, similar to how College Football operates, who can determine if a questionable call should have been made or not.

- 4) Use the NFL's example. If you lose a challenge, you lose a timeout. lets apply this for eliminations. Maybe this should be the first thing implemented if FIRST did decide to create a Instant Replay system.

- 5) Feel free to adjust the number of cards a team is given to use during qualifications. I think two is a fair amount. If a team wins their first challenge, they get to keep their card, They lose their card for each subsequent challenge. (Thus, no team is given more than three total challenge opportunities during qualifications.

Even baseball finally adapted instant replay to a cretin degree, why can't FIRST?