08-04-2010 18:22

Karthik

Re: paper: 2010 Team 1114 Championship Scouting Database

Greetings all,

Attached is the 2010 Team 1114 Championship Database. This year's database includes full results for every team who competed in the 2010 season as well. We've based this version on the current divisions posted at usfirst.org. If these change we will update accordingly. (However if it's just a couple of additions, we may not release a new version)

The database includes:
- An interface allow you pull an individual team's record
- Full listing of awards, record & finish
- Team scoring averages
- "Calculated Contribution" which is the same calculation being refered to as OPR on these forums. This calculation usings linear alegbra to determine what a team's average input to their alliance was at each regional. (Only using qualifying match results)
- A master sheet for a sortable comparison of all FIRST teams
- Master sheets for each division and full divisional assigments

The data we have was all mined from the FIRST website. There may be some errors, but I'm confident the data is 97.1114% accurate. That being said, there were some gaps in the standings for WOR and IL. These regionals will have some funky results, as we pulled the standings straight from usfirst.org, but the alliance selection results from actual observation.

We'll upload a version for older Excel shortly. It's a much larger file.

Prior to 2008 we never released any of our regression analysis (Calculated Contribution) that we had been doing since 2004. Since people have become more knowledgeable on the subject we decided to make the change. Please do not take a poor score as a slight or an insult. We simply used the actual scores from matches to perform a calculation. We feel that this tool is the best available metric if you are unable to watch the actual matches. Since none of us can attend every regional, it should be a valuable tool. Regression analysis is much more effective for Breakaway than it was for Lunacy, but still not as good as it was for Overdrive. If you want more details on this, come check out my seminar in Atlanta.

Thanks to Geoff Allan, Ben Bennett and Roberto Rotolo of Team 1114 Stats and Research for creating this year's database.

If you have any questions, please ask.

08-04-2010 18:35

bobrenjc93

Re: paper: 2010 Team 1114 Championship Scouting Database

Very impressive. Thanks a lot!

08-04-2010 18:45

Akash Rastogi

Re: paper: 2010 Team 1114 Championship Scouting Database

Thanks Karthik, amazingly useful as always.

08-04-2010 18:47

Racer26

Re: paper: 2010 Team 1114 Championship Scouting Database

*cringe* Well, that makes us look like crap. At least our hanger is significantly better now, and we should be hanging every match in Atlanta.

08-04-2010 18:52

Chris is me

Re: paper: 2010 Team 1114 Championship Scouting Database

I love how this mathematically demonstrates 1124's dramatic improvement.

Thanks 1114! If you guys are in need of the last few WPI matches, I can get the data for them.

08-04-2010 19:06

waialua359

Re: paper: 2010 Team 1114 Championship Scouting Database

According to the divisions breakdown summary, Archimedes is the most stacked division, including the most Chairman's Award winners.
My bet is that the CA winning team this year comes out from that division.

08-04-2010 19:08

TKM.368

Re: paper: 2010 Team 1114 Championship Scouting Database

Or teams can download the Compatibility Pack to be able to open in an older version.

Thanks 1114!

08-04-2010 19:13

Tom Bottiglieri

Re: paper: 2010 Team 1114 Championship Scouting Database

It is interesting to look at how some teams improved over the season.

What's better? A team who was good but didn't improve much from event to event? Or a team who was not so good at their first event but was much better at their second?

For the latter, do you think the slope can stay steep? Or is this one of those tortoise/hare situations?

08-04-2010 19:19

kjolana1124

Re: paper: 2010 Team 1114 Championship Scouting Database

I see it almost like an exponential curve. Teams that tend to do better over the course of a few events have a habit of leveling off. Whether it be from getting too sure of yourself, falling into a pattern, or just reaching your peak a little too soon, they eventually begin to stabilize. Either that, or in the effort to keep improving they end up becoming weaker.

Continuous improvement =/= continuous work. Sometimes all a team needs is a better mindset between competitions, not a better robot.

08-04-2010 22:38

PaW

Re: paper: 2010 Team 1114 Championship Scouting Database

Annual thanks to you and your team, Karthik! Nice work, as always.

08-04-2010 23:19

Justin Montois

Re: paper: 2010 Team 1114 Championship Scouting Database

wow. They scout like they build robots. Talk about a powerhouse.

08-04-2010 23:22

J Flex 188

Re: paper: 2010 Team 1114 Championship Scouting Database

I simply can't fathom how Roberto Rotolo keeps pumping this stuff out year after year with such a high degree of quality too! He's definitely the backbone to that skilled group of individuals, its nuts, every year!

09-04-2010 00:03

Zach O

Re: paper: 2010 Team 1114 Championship Scouting Database

Wow! What a database! Very impressive. Thank you for posting it.

09-04-2010 09:36

dsmoker

Re: paper: 2010 Team 1114 Championship Scouting Database

For a team like ours, that just qualified for the Championship last weekend and is just barely able to send a skeleton crew (7 kids, 4 mentors) down to Atlanta, this data base is an invaluable resource. Plus, we look pretty good!! Thank you for sharing this information with the FIRST community.

09-04-2010 11:41

Chris Fultz

Re: paper: 2010 Team 1114 Championship Scouting Database

Karthik, is that accuracy in Canadian or American?

Do you know the exchange rate?


Seriously - awesome work and awesome data.
Much appreciated.

09-04-2010 12:04

Drivencrazy

Re: paper: 2010 Team 1114 Championship Scouting Database

FIRST off Karthik thanks alot. This is by far one of the most proffessional scouting databases I have seen.

Can you tell me where the Calculated Contribution comes from or point me to a forum that does?

Once again thanks and good luck at the champs.

09-04-2010 13:19

Karthik

Re: paper: 2010 Team 1114 Championship Scouting Database

This is explained in a few other places on these forums, but it's probably best that I put an explanation in this thread.

Going into an event like the Championship, it's impossible to watch video on all 344 teams. So, how can you get an idea of what each team's scoring potential is? Well, one way to do it would be just to look at their average score per match. Nice and simple, but it only tells you a small part of the story. Since FIRST matches involve alliances, an average score does not isolate the individual performance of a given team. So if Team XYZ repeatedly plays matches with great teams, their average score will not necessarily be an accurate indicator of the team's performance.

So, how do we isolate the impact of a single team on a match? Simple, using good old linear algebra. For those of your familiar with advanced basketball statistics, the method I'm about to describe is very similar to "adjusted +/-". Adjusted +/- has become a very popular tool among NBA franchises to try and figure out just how much each player is contributing on the court. NBA teams have long figured out that just looking at how many points a player scores does not always tell you how much they've impacted a team towards victory. There have been some great papers presented about this at the MIT Sloan Sports Analytics Conference.

So, here's how it works. For each alliance during qualifying matches at a regional, we set up an equation. Say Teams i, j, k were on an alliance together and scored s points. Out equation would be

T_i + T_j + T_k = s

where T_i, T_j & T_k are variables representing those teams. So if there are m teams at a regional, and n matches, we now have equations that give us a m x 2n matrix. We then solve this matrix for our variables, and voila you have each team's Calculated Contribution.

Why is Calculated Contribution so valuable this year? Well, consider a team that plays the midfield and is great at supplying balls to the home zone, but rarely scores them. If your scouts are just tracking goals scored, they might get a big zero. But their Calculated Contribution (if run over a large enough sample size) would should a higher value reflecting the points they helped their alliance score.

There are a lot more subtle details as to why Calculated Contribution is a good tool, and even more that expose some shortcomings. Unfortunately it's a bit much to go into on CD post while I'm eating lunch. I'll talk a lot about this at my Effective FIRST Strategy Seminar in Atlanta.

Hope this helped.

09-04-2010 13:38

hipsterjr

Re: paper: 2010 Team 1114 Championship Scouting Database

This sheet on an ipod touch will be invaluable. Awesome job.
*ekk* (entered my team) looks like the data on us reaffirms our thoughts that we should focus on defense. This program has even aided in scouting my own team and influencing our stratagy! Now that is a testiment to it's effectivness.

09-04-2010 14:09

Tom Bottiglieri

Re: paper: 2010 Team 1114 Championship Scouting Database

I graphed the Calculated Contributions of a few teams over time. I'd like to do this again after champs and compare the results.

http://i44.tinypic.com/1zdryme.jpg

09-04-2010 14:10

Racer26

Re: paper: 2010 Team 1114 Championship Scouting Database

It would be interesting to plot weighted CC over time from year to year and see if teams like 1114 are consistently getting better at FRC.

09-04-2010 15:03

pfreivald

Re: paper: 2010 Team 1114 Championship Scouting Database

I like the calculated contributions bit, especially since we built a midfield robot that is designed to work as a part of an overall alliance, and was not designed as a one-robot scoring machine.

Good stuff!

09-04-2010 15:45

kjolana1124

Re: paper: 2010 Team 1114 Championship Scouting Database

Interesting. Can you please explain what the x and y axis each represent? I'm a little confused.

09-04-2010 16:12

Ian Curtis

Re: paper: 2010 Team 1114 Championship Scouting Database

I presume the y axis is the Calculated Contribution, and the x axis is the event. So for you guys, at x=1 is your performance at WPI, and at x=2 is your performance at CT. Excel probably put in the .5s on its own.

09-04-2010 16:17

kjolana1124

Re: paper: 2010 Team 1114 Championship Scouting Database

Oh, I see. Thanks

10-04-2010 11:38

Drivencrazy

Re: paper: 2010 Team 1114 Championship Scouting Database

"Hope this helped."

Yes indeed it did.

Thanks again and I look forward to your seminar. I'll be there.

10-04-2010 12:33

Lil' Lavery

Re: paper: 2010 Team 1114 Championship Scouting Database

edit: Woops, sorry about that. My general theory still applies.

I agree with Karthik's statements that "Calculated Contributions" (aka "Offensive power rankings" and "Defensive power rankings") worked very well in Overdrive given the more "isolated" nature of the game (still by no means an isolated event, though). I also agree with his statement that they should work much better for Breakaway than they did for Lunacy. However, as awesome as they might be, they still aren't telling the full story and should be taken with a grain of salt.

However, they still can be a useful tool. If you find yourself with a chance to be in a picking position Friday night at an event and don't really have much/any (reliable) scouting data, go ahead and run these types of numbers. Pick out some of the higher ranking teams in whichever attributes you're looking for. Have whatever people you can spare (maybe just your "media person," for example) do whatever they can to watch a match or two of these targeted teams the next day and see if their performance on the field backs up what the numbers say, and base your selection choices off of that.

That being said, this database is still one of the most wonderful resources in FRC. 1114 has done another amazing job on it, just like every year. It's tremendously helpful to find quick information about any team, even those not going to Championship. Kudos again to the Simbots for incredible work.

I do have one request, though. Is there any chance you guys can add in the Dean's list to a teams' profile? You list if a team won a WFFA award, so why not Dean's list as well?

10-04-2010 13:03

Karthik

Re: paper: 2010 Team 1114 Championship Scouting Database

Sean,

"Average Defensive Score" is actually not one of our advanced metrics. It's simply the average score of all the alliances a team faced in qualifying matches. So, a high Average Defensive Score means that the alliances you faced scored a lot of points. This year we ranked that stat in descending order on the theory that in a game with scoring that doesn't reward defense in the qualifying round, a team with a high defensive score was either consistently playing stronger alliances or consistently scoring for their opponent. Regardless, Average Defensive Score has no link to Calculated Contribution or OPR in anyway.

We wanted to do this, unfortunately there were far too many gaps on the FIRST website. Most events displayed the winner of the award, but did not have team associated with the winner. We decided that it we couldn't list all the winners accurately, it would be better to leave the award out. (There were gaps with the WFFA's as well, but not as many and we were able to fill those with some Googling)

10-04-2010 14:41

Vikesrock

Re: paper: 2010 Team 1114 Championship Scouting Database

You just didn't ask the right people

Between myself and EricH we managed to track down all but one of the gaps for Fantasy FIRST scoring purposes.

10-04-2010 16:06

Chris is me

Re: paper: 2010 Team 1114 Championship Scouting Database

Can us FF people supply all that information to 1114?

10-04-2010 16:37

Vikesrock

Re: paper: 2010 Team 1114 Championship Scouting Database

Done.

12-04-2010 14:44

MikeReilly

Re: paper: 2010 Team 1114 Championship Scouting Database

First of all, amazing stuff. Love it, wish I had time to really understand it before the Dome but I will be digesting it and learning it before next year.

Second, is there any consideration to a version with revised scores for Week 1 match winners? I'm guessing that would have an effect with the +5 for winning a match. I'm going to try and play with that myself, but wondering if you did some work on it and saw no major difference before I do.

Thanks!