Go to Post Steal from the best, and then invent the rest! - dlavery [more]
Home
Go Back   Chief Delphi > FIRST > General Forum
CD-Media   CD-Spy  
portal register members calendar search Today's Posts Mark Forums Read FAQ rules

 
Reply
Thread Tools Rate Thread Display Modes
  #16   Spotlight this post!  
Unread 14-04-2013, 11:59
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 8,097
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Tom Line View Post
What OPR does not clarify, however, is autonomous and climbing.
To be clear: Ed's spreadsheet has separate per-team least-squares estimates of Auto, Climb, and TeleOp.

Quote:
Another place that OPR fails is defense. There were several extremely well-scoring machines at MSC that had low OPR's because they spent half their matches playing defense. Unless you scouted and knew that, you'd assume they were just poor robots.
In general if a "well-scoring" machine is assigned the task of defending, it's often because the other two machines are higher scorers. In the least-squares analysis that is used to compute OPR, the defending machine gets credit for the alliance score.


Reply With Quote
  #17   Spotlight this post!  
Unread 14-04-2013, 12:26
Bongle's Avatar
Bongle Bongle is offline
Registered User
FRC #2702 (REBotics)
Team Role: Mentor
 
Join Date: Feb 2004
Rookie Year: 2002
Location: Waterloo
Posts: 1,069
Bongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond reputeBongle has a reputation beyond repute
Send a message via MSN to Bongle
Re: OPR after Week Seven Events

Quote:
Originally Posted by GearsOfFury View Post
Totally agree with all of your other points except this one. Why do you need to "differentiate your picks" for the sake of differentiation? You need to find the best robots available to complete your alliance. If there were a single, universal system to do this, it wouldn't be any less valuable just because everyone else was using it.
I guess another way of thinking of it is imagine this scene:
A: "How can we make our paper-based scouting better?"
B: "Let's use this OPR thing. It'll make us better than the opposition for sure!"

The problem in this scene is that nearly every other quality team is probably also augmenting their scouting with OPR. So in order to be scouting smarter than the opposition, you need to be thinking further than the numbers you can pull off the internet.


Essentially, the rising tide of more-available analytical tools has lifted everyone's scouting boats: everyone is scouting better because of it, but a given team is probably relatively in the same position vis a vis its competitors as it was before the advent of easy-to-use OPR data. So you need to use it, but to assume that it is the ticket to better-than-your-opposition picking may be a bad assumption.

Another hobby of mine is triathlons, where you see something similar: a new, faster bike will come out, and you'll buy it and use it just to keep up with your competitors. The speedier bike or technology doesn't necessarily move you up the rankings, since everyone else is using it - moving up still requires hard work and getting better.

Last edited by Bongle : 14-04-2013 at 12:32.
Reply With Quote
  #18   Spotlight this post!  
Unread 14-04-2013, 12:30
PayneTrain's Avatar
PayneTrain PayneTrain is offline
Q&A Dartboard Detractor
AKA: Lizard King
FRC #0422 (The Meme Tech Pneumatic Devices)
Team Role: Mascot
 
Join Date: Mar 2010
Rookie Year: 2009
Location: RVA
Posts: 2,259
PayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond reputePayneTrain has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by IndySam View Post
How many thumbs do you have?
Shh, everyone knows that's a special Canadian trade secret.
Reply With Quote
  #19   Spotlight this post!  
Unread 14-04-2013, 12:34
Jaxom Jaxom is offline
Registered User
AKA: Michael Hartwig
FRC #1986 (Team Titanium)
Team Role: Mentor
 
Join Date: Apr 2010
Rookie Year: 2008
Location: Lee's Summit, MO
Posts: 379
Jaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant futureJaxom has a brilliant future
Re: OPR after Week Seven Events

Quote:
Originally Posted by Bongle View Post
Essentially, the rising tide of more-available analytical tools has lifted everyone's scouting boats: everyone is scouting better because of it, but a given team is probably relatively in the same position vis a vis its competitors as it was before the advent of easy-to-use OPR data. So you need to use it, but to assume that it is the ticket to better-than-your-opposition picking may be a bad assumption.
This. No matter how much we improve our automated analyses, humans still have to not only be in the loop, but control it. It might be interesting to find out if building your pick list strictly on OPR is better than just picking the highest-ranked available robot. I'm pretty sure it is, but without some actual, numerical, analysis I won't say that I believe it.

Quote:
Originally Posted by Andrew Schreiber View Post
Somewhere, deep in the bowels of 1114's shop, there exists a room full of nothing but students training to be Karthik's hand doubles. They practice thumbs up and thumbs down for hours a day for many years until they are needed in an occasion such as this.
Preposterous! This is 1114; they're not wasting time practicing with humans to stand in for Karthik's thumbs. They're building something.
__________________


Mentor http://www.teamtitanium.org/
Reply With Quote
  #20   Spotlight this post!  
Unread 14-04-2013, 12:46
Ed Law's Avatar
Ed Law Ed Law is offline
Registered User
no team (formerly with 2834)
 
Join Date: Apr 2008
Rookie Year: 2009
Location: Foster City, CA, USA
Posts: 752
Ed Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Tom Line View Post
Another place that OPR fails is defense. There were several extremely well-scoring machines at MSC that had low OPR's because they spent half their matches playing defense. Unless you scouted and knew that, you'd assume they were just poor robots.

That's the kind of third round sleeper pick that high seeded teams dream of.
Tom, did you see the numbers on 2337. They had a respectable OPR of 44 but ranked 33 out of 64 teams. So they can be considered not the best offensive robot. Perhaps it is because they played a lot of defense. But did you see their CCWM number? It was 17.6 and ranked 11th out of 64 teams. 2054 and 67 made a wise choice. I doubt they did it because of the CCWM number, but you cannot deny that the CCWM number supports their scouts' decision. I had always advocated using CCWM as part of the sanity check for 2nd round pick. It does take defense into consideration.
__________________
Please don't call me Mr. Ed, I am not a talking horse.
Reply With Quote
  #21   Spotlight this post!  
Unread 14-04-2013, 13:00
Woolly's Avatar
Woolly Woolly is offline
Programming Mentor
AKA: Dillon Woollums
FRC #1806 (S.W.A.T.)
Team Role: Mentor
 
Join Date: Feb 2012
Rookie Year: 2012
Location: Springfield, MO
Posts: 512
Woolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond reputeWoolly has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Tom Line View Post
Another place that OPR fails is defense. There were several extremely well-scoring machines at MSC that had low OPR's because they spent half their matches playing defense. Unless you scouted and knew that, you'd assume they were just poor robots.
OPR fails at defense this year because defense in general hasn't been very effective. Those well scoring robots that have low OPRs due to playing defense would have been better off playing offense for all their matches because they are better at offense than defense. For playing defense to be a net gain, you have to stop more points than you would normally score.

Though the fact teams are playing defense when they really shouldn't is why traditional scouting is still important.
__________________


Team 1806 Student: 2012-2013 | Mentor: 2013-Present
Reply With Quote
  #22   Spotlight this post!  
Unread 14-04-2013, 13:06
Ed Law's Avatar
Ed Law Ed Law is offline
Registered User
no team (formerly with 2834)
 
Join Date: Apr 2008
Rookie Year: 2009
Location: Foster City, CA, USA
Posts: 752
Ed Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Chris is me View Post
OPR (specifically CCWM) is better than I expected at separating teams, and seems to be ~85% accurate based on a comparison to a few events of data.

That said, you have to remember that OPR is an approximation of skill that is better than average score but no replacement for a real scouting system. It's a decent "sanity check" to see if you missed an outlier or two but if you have actual data I wouldn't even touch it.

During and after the season, it's a very good approximation of skill that is very useful for Internet debates.
I don't know why there is this constant argument of the shortcomings of OPR. I don't think anybody including myself advocate only using OPR/CCWM as a way to scout, decide on match strategy or alliance selection.

To me, OPR/CCWM is very useful if you were not at the event so you have a general idea what each robot is good at based on Auto OPR, Climb OPR and Tele OPR. Like Chris said, if you have actual data, why do you need OPR? I don't remember who it was, but I was once asked if I have the match data of each robot of every match, will I be able to create a better model to increase the prediction ability of OPR? I thought it was a trick question.

But we need to keep in mind that some teams are very small. One person cannot watch 6 robots at the same time. He/she can try and take some notes but it is very difficult to rank teams based on subjective measures on select matches. In those cases, I think it is better to use OPR/CCWM as a guide rather than selecting the next highest seeded team

Take a look at what teams actually select based on human scouts (I assume) at events and compare the first round picks with OPR and second round picks with combination of OPR/CCWM, it is amazing how good the correlation is and why some teams that seeded high were not selected. There are always exceptions because a team is looking for a very specific attribute in a supporting robot. But in some cases I have to scratch my head where it looked like a poor choice and very often the data confirmed that they ended up as quarterfinalist.
__________________
Please don't call me Mr. Ed, I am not a talking horse.
Reply With Quote
  #23   Spotlight this post!  
Unread 14-04-2013, 13:09
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 8,097
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Ed Law View Post
I don't know why there is this constant argument of the shortcomings of OPR. I don't think anybody including myself advocate only using OPR/CCWM as a way to scout, decide on match strategy or alliance selection.
Quoted for truth.


Reply With Quote
  #24   Spotlight this post!  
Unread 14-04-2013, 13:14
Ether's Avatar
Ether Ether is offline
systems engineer (retired)
no team
 
Join Date: Nov 2009
Rookie Year: 1969
Location: US
Posts: 8,097
Ether has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond reputeEther has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Ed Law View Post
I was once asked if I have the match data of each robot of every match, will I be able to create a better model to increase the prediction ability of OPR? I thought it was a trick question.
Given that data, you could do better than just reporting each team's high/low/mean/median scores etc and using that for prediction.

You could take into account which teams they were playing with, and against, for each of their alliance scores, and so on, recursively for those teams. Computer ranking of football teams do something similar.


Reply With Quote
  #25   Spotlight this post!  
Unread 14-04-2013, 13:17
Andrew Schreiber Andrew Schreiber is offline
Joining the 900 Meme Team
FRC #0079
 
Join Date: Jan 2005
Rookie Year: 2000
Location: Misplaced Michigander
Posts: 4,066
Andrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond reputeAndrew Schreiber has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Ed Law View Post
I don't know why there is this constant argument of the shortcomings of OPR. I don't think anybody including myself advocate only using OPR/CCWM as a way to scout, decide on match strategy or alliance selection.
It's occasionally beneficial to remind people that OPR/CCWM/Metric du jour are not magic bullets. You more than likely know it but freshmandood519 (which I'm hoping is NOT an actual user name on here) probably doesn't.
__________________




.
Reply With Quote
  #26   Spotlight this post!  
Unread 14-04-2013, 13:59
Joe Ross's Avatar Unsung FIRST Hero
Joe Ross Joe Ross is offline
Registered User
FRC #0330 (Beachbots)
Team Role: Engineer
 
Join Date: Jun 2001
Rookie Year: 1997
Location: Los Angeles, CA
Posts: 8,574
Joe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond reputeJoe Ross has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Andrew Schreiber View Post
freshmandood519 (which I'm hoping is NOT an actual user name on here)
That one's safe, but stupidfreshman is taken.
Reply With Quote
  #27   Spotlight this post!  
Unread 14-04-2013, 14:23
Grim Tuesday's Avatar
Grim Tuesday Grim Tuesday is offline
Registered User
AKA: Simon Bohn
FRC #0639 (Code Red)
Team Role: Alumni
 
Join Date: Jan 2010
Rookie Year: 2010
Location: Baltimore MD (JHU)
Posts: 1,606
Grim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond reputeGrim Tuesday has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Ed Law View Post
I don't know why there is this constant argument of the shortcomings of OPR. I don't think anybody including myself advocate only using OPR/CCWM as a way to scout, decide on match strategy or alliance selection.

To me, OPR/CCWM is very useful if you were not at the event so you have a general idea what each robot is good at based on Auto OPR, Climb OPR and Tele OPR. Like Chris said, if you have actual data, why do you need OPR? I don't remember who it was, but I was once asked if I have the match data of each robot of every match, will I be able to create a better model to increase the prediction ability of OPR? I thought it was a trick question.

But we need to keep in mind that some teams are very small. One person cannot watch 6 robots at the same time. He/she can try and take some notes but it is very difficult to rank teams based on subjective measures on select matches. In those cases, I think it is better to use OPR/CCWM as a guide rather than selecting the next highest seeded team

Take a look at what teams actually select based on human scouts (I assume) at events and compare the first round picks with OPR and second round picks with combination of OPR/CCWM, it is amazing how good the correlation is and why some teams that seeded high were not selected. There are always exceptions because a team is looking for a very specific attribute in a supporting robot. But in some cases I have to scratch my head where it looked like a poor choice and very often the data confirmed that they ended up as quarterfinalist.
I think one of the best cases of this OPR/Human scout difference was actually at the Buckeye Regional when we picked you guys. In my opinion, the biggest issue with OPR (and I was talking about this in the week 6 thread) is that it doesn't take into consideration improvement -- it's an average. So if a team has shooter troubles on Friday (like 2834) then gets their game on Saturday morning, their OPR would not be significantly improved. But human scouts can see and snag a second round pick that scores 60+ in a match.

On the other hand, at Buckeye, our first pick, 2252 was ranked 3rd in OPR but in the 20s seeding wise - and they were clearly one of the three best scorers at the event. There often isn't the type of correlation between rank and OPR as I'd like to see and OPR does a very good job of quickly highlighting which teams to watch carefully and look at their schedules to see if it was a particularly hard one.
Reply With Quote
  #28   Spotlight this post!  
Unread 14-04-2013, 14:43
Ed Law's Avatar
Ed Law Ed Law is offline
Registered User
no team (formerly with 2834)
 
Join Date: Apr 2008
Rookie Year: 2009
Location: Foster City, CA, USA
Posts: 752
Ed Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond reputeEd Law has a reputation beyond repute
Re: OPR after Week Seven Events

Quote:
Originally Posted by Grim Tuesday View Post
I think one of the best cases of this OPR/Human scout difference was actually at the Buckeye Regional when we picked you guys. In my opinion, the biggest issue with OPR (and I was talking about this in the week 6 thread) is that it doesn't take into consideration improvement -- it's an average. So if a team has shooter troubles on Friday (like 2834) then gets their game on Saturday morning, their OPR would not be significantly improved. But human scouts can see and snag a second round pick that scores 60+ in a match.

On the other hand, at Buckeye, our first pick, 2252 was ranked 3rd in OPR but in the 20s seeding wise - and they were clearly one of the three best scorers at the event. There often isn't the type of correlation between rank and OPR as I'd like to see and OPR does a very good job of quickly highlighting which teams to watch carefully and look at their schedules to see if it was a particularly hard one.
I am very glad you did not use OPR/CCWM for your second pick at Buckeye. OPR/CCWM does not take the order of the match into consideration so you do not know if a team is improving or the robot is starting to have problems. I use trendlines in Excel to fit a straight line through the data points of all previous matches score based on match scouting. Then I look at the OPR numbers and trendline predictions and decide on what I think each team will likely score in the next match. Using this to look for teams in alliance selection is better than just using OPR/CCWM, and you will not miss a team that showed big improvement on Saturday like at Buckeye.
__________________
Please don't call me Mr. Ed, I am not a talking horse.
Reply With Quote
  #29   Spotlight this post!  
Unread 14-04-2013, 17:32
NXTGeek's Avatar
NXTGeek NXTGeek is offline
Registered User
AKA: Malachi
no team
Team Role: College Student
 
Join Date: Jan 2012
Rookie Year: 2011
Location: Laguna Beach, CA
Posts: 126
NXTGeek is a splendid one to beholdNXTGeek is a splendid one to beholdNXTGeek is a splendid one to beholdNXTGeek is a splendid one to beholdNXTGeek is a splendid one to beholdNXTGeek is a splendid one to beholdNXTGeek is a splendid one to behold
Thumbs up Re: OPR after Week Seven Events

I think OPR is a success, but if there is a way to play on a team's continuous improvement as the season progresses, say through elims and other events, that could give a more accurate preview on a team's performance.
I know that if you make each team's next event substantly more important than the first, the single and double regional teams would get nicked compared to the three regional teams.
The point is, however, that each team is accurately compared in offensive power rating to other teams, conistency included. In this case then, the more accurate average of a team should reflect more of there later regional/s then their first.
OPR is here to stay, but improvement should never relinquish.
__________________
2013 -Team 3476
Champion (Fall Classic)
Champion (Battle at the Border)
Finalist (Indiana Robotics Invitational)
Newton Division CHAMPIONS (Worlds)
Wild Card as Finalist Alliance Captain (Inland Empire -> Worlds)
Innovation in Control Award (Inland Empire)
Industrial Design Award (San Diego)
Semifinalist (San Diego)

2012 -Team 3476
Champion (Battle at the Border)
Champion (San Diego -> Worlds)
Excellence in Engineering Award (San Diego)

2011 -Team 1425
Engineering Inspiration Award (Portland Autodesk -> Worlds)
Finalist (Portland Autodesk)
WFFA (Tim BD, Portland Autodesk)
Finalist (Seattle Olympic)
Entrepreneurship Award (Seattle Olympic)
Reply With Quote
  #30   Spotlight this post!  
Unread 17-04-2013, 16:20
Scott_4140 Scott_4140 is offline
Registered User
FTC #4140 (Fish in the Boat)
Team Role: Coach
 
Join Date: May 2011
Rookie Year: 2005
Location: Lakeville, MN
Posts: 86
Scott_4140 is just really niceScott_4140 is just really niceScott_4140 is just really niceScott_4140 is just really nice
Re: OPR after Week Seven Events

OPR/CCWM look like they would be a fantastic asset to FTC teams at the World Championships.

FIRST doesn't currently maintain a data set of matches for FTC the way they do for FRC. There are a few data sets available for FTC Championship tournaments, but not all. Generating a usable World ranking is probably not practical at this late date. Maybe next season.

It does seem that it would be possible to adapt this spreadsheet to FTC for use at just the FTC World Championships. Division lists are out for FTC.

I tried creating a new tab and entering a set of schedule & match data that I found posted from Alaska. I figured it would be a good test set. I'm not sure if I got the data entered in the propper format. The CTRL-SHIFT-R and CTRL-SHIFT-O both produces an error for a subscript being out of range. Unfortunately my Visual Basic is a bit rusty. It probably requires more than just a new tab. Probably need to tweak the macros for 2-team alliances as well.

What would it take to create an FTC version?
Reply With Quote
Reply


Thread Tools
Display Modes Rate This Thread
Rate This Thread:

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

vB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Forum Jump


All times are GMT -5. The time now is 20:17.

The Chief Delphi Forums are sponsored by Innovation First International, Inc.


Powered by vBulletin® Version 3.6.4
Copyright ©2000 - 2017, Jelsoft Enterprises Ltd.
Copyright © Chief Delphi