Go to Post Learn the theory in the classroom and pick up the practical through clubs (FIRST, mini Baja, SAE)... you really need the scale balanced in the middle - Greg Needel [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
 
Thread Tools Rating: Thread Rating: 4 votes, 5.00 average. Display Modes
  #1   Spotlight this post!  
Unread 13-05-2015, 11:53
GreyingJay GreyingJay is offline
Robonut
AKA: Mr. Lam
FRC #2706 (Merge Robotics)
Team Role: Mentor
 
Join Date: Mar 2015
Rookie Year: 2015
Location: Ottawa, Canada
Posts: 783
GreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond reputeGreyingJay has a reputation beyond repute
Re: "standard error" of OPR values

Quote:
Originally Posted by Oblarg View Post
Unfortunately, this is tenuous - there's no real reason to believe that each team contributes linearly to score by some flat amount per match, and that variance beyond that is a random variable whose distribution does not change match-to-match.
This year may be an anomaly, but it seems to me like, for some teams anyway, this is a reasonable model. Teams have built robots that are very predictable and task-oriented. For example: grab a bin, drive to the feeder station, stack, stack, stack, push, stack, stack, stack, push, etc. Knowing how quickly our human player and stack mechanism are, we can predict with reasonable accuracy how many points we can typically score in a match, with the only real variance coming from when things go wrong.
__________________
"If I'm going to mentor someone, I'm going to be involved in their life as a positive force." -Mechvet
Reply With Quote
  #2   Spotlight this post!  
Unread 13-05-2015, 12:31
Basel A's Avatar
Basel A Basel A is offline
It's pronounced Basl with a soft s
AKA: @BaselThe2nd
FRC #3322 (Eagle Imperium)
Team Role: College Student
 
Join Date: Mar 2009
Rookie Year: 2009
Location: Ann Arbor, Michigan
Posts: 1,934
Basel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond reputeBasel A has a reputation beyond repute
Re: "standard error" of OPR values

I have to strongly agree with what Ed had to say above. Errors in OPR happen when its assumptions go unmet: partner or opponent interaction, team inconsistency (including improvement), etc. If one if these single factors caused significantly more variation than the others, then the standard error might be a reasonable estimate of that factor. However, I don't believe that this is the case.

Another option would be to take this measure in the same way that we take OPR. We know that OPR is not a perfect depiction of a team's robot quality or even a team's contribution to its alliance, but we use OPR anyway. In the same way, we know the standard error is an imperfect depiction of a team's variation in contribution.

People constantly use the same example in discussing consistency in FRC. A low-seeded captain, when considering two similarly contributing teams, is generally better off selecting an inconsistent team over a consistent one. Standard error could be a reasonable measure of this inconsistency (whether due to simple variation or improvement). At a scouting meeting, higher standard error could indicate "teams to watch" (for improvement).

But without having tried it, I suspect a team's standard error will ultimately be mostly unintelligible noise.
__________________
Team 2337 | 2009-2012 | Student
Team 3322 | 2014-Present | College Student
“Be excellent in everything you do and the results will just happen.”
-Paul Copioli
Reply With Quote
  #3   Spotlight this post!  
Unread 13-05-2015, 13:47
Wayne TenBrink's Avatar
Wayne TenBrink Wayne TenBrink is offline
<< (2008 Game Piece)
FRC #1918 (NC Gears)
Team Role: Engineer
 
Join Date: Apr 2007
Rookie Year: 2006
Location: Fremont, MI, USA
Posts: 527
Wayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond reputeWayne TenBrink has a reputation beyond repute
Re: "standard error" of OPR values

Has anyone ever attempted a validation study to compare "actual contribution" (based on scouting data or a review of match video) to OPR values? It seems like this would be fairly easy and accurate for Recycle Rush (and very difficult for Aerial Assist). I did that with our performance at one district event and found the result to be very close (OPR=71 vs "Actual"= 74).

In some ways, OPR is probably more relevant than "actual contribution". For example, a good strategist in Aerial Assist could extract productivity from teams that might otherwise just drive around aimlessly. This sort of contribution would show up in OPR, but a scout wouldn't attribute it to them as an "actual contribution".

It would be interesting to see if OPR error was the same (magnitude and direction) for low, medium, and high OPR teams, etc.
__________________
NC Gears (Newaygo County Geeks Engineering Awesome Robotic Solutions)

FRC 1918 (Competing at St. Joseph and West MI in 2017)
FTC 6043 & 7911
Reply With Quote
  #4   Spotlight this post!  
Unread 13-05-2015, 13:52
IKE's Avatar
IKE IKE is offline
Not so Custom User Title
AKA: Isaac Rife
no team (N/A)
Team Role: Mechanical
 
Join Date: Jan 2008
Rookie Year: 2003
Location: Michigan
Posts: 2,149
IKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond reputeIKE has a reputation beyond repute
Re: "standard error" of OPR values

Quote:
Originally Posted by Wayne TenBrink View Post
Has anyone ever attempted a validation study to compare "actual contribution" (based on scouting data or a review of match video) to OPR values? It seems like this would be fairly easy and accurate for Recycle Rush (and very difficult for Aerial Assist). I did that with our performance at one district event and found the result to be very close (OPR=71 vs "Actual"= 74).

In some ways, OPR is probably more relevant than "actual contribution". For example, a good strategist in Aerial Assist could extract productivity from teams that might otherwise just drive around aimlessly. This sort of contribution would show up in OPR, but a scout wouldn't attribute it to them as an "actual contribution".

It would be interesting to see if OPR error was the same (magnitude and direction) for low, medium, and high OPR teams, etc.
I have done this in the past (2010, 2011, 2012, 2013). 2013 was pretty close. 2010 was pretty close at a given event though interesting strategies could result in interesting scores. 2011 was useful at District level of competition, but not very useful at MSC or Worlds. 2012, semi useful if using some sort of co-op balance partial contribution factor.

Someone did a study for Archimedes this year. I would say it is similar to 2011 where 3 really impressive scorers would put up a really great score, but if you expected 3X, you would instead get more like 2.25 to 2.5....
Reply With Quote
  #5   Spotlight this post!  
Unread 13-05-2015, 13:55
Citrus Dad's Avatar
Citrus Dad Citrus Dad is offline
Business and Scouting Mentor
AKA: Richard McCann
FRC #1678 (Citrus Circuits)
Team Role: Mentor
 
Join Date: May 2012
Rookie Year: 2012
Location: Davis
Posts: 991
Citrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond repute
Re: "standard error" of OPR values

Quote:
Originally Posted by IKE View Post
I have done this in the past (2010, 2011, 2012, 2013). 2013 was pretty close. 2010 was pretty close at a given event though interesting strategies could result in interesting scores. 2011 was useful at District level of competition, but not very useful at MSC or Worlds. 2012, semi useful if using some sort of co-op balance partial contribution factor.

Someone did a study for Archimedes this year. I would say it is similar to 2011 where 3 really impressive scorers would put up a really great score, but if you expected 3X, you would instead get more like 2.25 to 2.5....
I produced predicted scores for Newton using the OPR components to eliminate potential double counting of auto and adjust for coop points. I predicted 118 would average 200 and they averaged 198. I have to check the distribution of OPR vs actual points.
Reply With Quote
  #6   Spotlight this post!  
Unread 13-05-2015, 13:52
Citrus Dad's Avatar
Citrus Dad Citrus Dad is offline
Business and Scouting Mentor
AKA: Richard McCann
FRC #1678 (Citrus Circuits)
Team Role: Mentor
 
Join Date: May 2012
Rookie Year: 2012
Location: Davis
Posts: 991
Citrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond reputeCitrus Dad has a reputation beyond repute
Re: "standard error" of OPR values

Quote:
Originally Posted by Wayne TenBrink View Post
Has anyone ever attempted a validation study to compare "actual contribution" (based on scouting data or a review of match video) to OPR values? It seems like this would be fairly easy and accurate for Recycle Rush (and very difficult for Aerial Assist). I did that with our performance at one district event and found the result to be very close (OPR=71 vs "Actual"= 74).

In some ways, OPR is probably more relevant than "actual contribution". For example, a good strategist in Aerial Assist could extract productivity from teams that might otherwise just drive around aimlessly. This sort of contribution would show up in OPR, but a scout wouldn't attribute it to them as an "actual contribution".

It would be interesting to see if OPR error was the same (magnitude and direction) for low, medium, and high OPR teams, etc.
This is a different question than whether the OPR accurately measures true contribution. (Another benefit of that exercise however is to determine whether the OPR estimate has a bias, e.g., related to relative scoring). There will always be error terms around the OPR parameter, so the question to be answered is what are the statistical properties of those error terms.
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 03:50.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi