paper: 2010 Team 1114 Championship Scouting Database

[Karthik]

Thread created automatically to discuss a document in CD-Media.

2010 Team 1114 Championship Scouting Database (http://www.chiefdelphi.com/media/papers/2361?) by Team 1114

[Karthik]

Greetings all,

Attached is the 2010 Team 1114 Championship Database. This year's database includes full results for every team who competed in the 2010 season as well. We've based this version on the current divisions posted at usfirst.org. If these change we will update accordingly. (However if it's just a couple of additions, we may not release a new version)

The database includes:
- An interface allow you pull an individual team's record
- Full listing of awards, record & finish
- Team scoring averages
- "Calculated Contribution" which is the same calculation being refered to as OPR on these forums. This calculation usings linear alegbra to determine what a team's average input to their alliance was at each regional. (Only using qualifying match results)
- A master sheet for a sortable comparison of all FIRST teams
- Master sheets for each division and full divisional assigments

The data we have was all mined from the FIRST website. There may be some errors, but I'm confident the data is 97.1114% accurate. That being said, there were some gaps in the standings for WOR and IL. These regionals will have some funky results, as we pulled the standings straight from usfirst.org, but the alliance selection results from actual observation.

We'll upload a version for older Excel shortly. It's a much larger file.

Prior to 2008 we never released any of our regression analysis (Calculated Contribution) that we had been doing since 2004. Since people have become more knowledgeable on the subject we decided to make the change. Please do not take a poor score as a slight or an insult. We simply used the actual scores from matches to perform a calculation. We feel that this tool is the best available metric if you are unable to watch the actual matches. Since none of us can attend every regional, it should be a valuable tool. Regression analysis is much more effective for Breakaway than it was for Lunacy, but still not as good as it was for Overdrive. If you want more details on this, come check out my seminar in Atlanta (http://www.chiefdelphi.com/forums/showthread.php?t=84958).

Thanks to Geoff Allan, Ben Bennett and Roberto Rotolo of Team 1114 Stats and Research for creating this year's database.

If you have any questions, please ask.

[bobrenjc93]

Very impressive. Thanks a lot!

[Akash Rastogi]

Thanks Karthik, amazingly useful as always.

[Racer26]

*cringe* Well, that makes us look like crap. At least our hanger is significantly better now, and we should be hanging every match in Atlanta.

[Chris is me]

I love how this mathematically demonstrates 1124's dramatic improvement.

Thanks 1114! If you guys are in need of the last few WPI matches, I can get the data for them.

[waialua359]

According to the divisions breakdown summary, Archimedes is the most stacked division, including the most Chairman's Award winners.
My bet is that the CA winning team this year comes out from that division.:D

[TKM.368]

We'll upload a version for older Excel shortly. It's a much larger file.


Or teams can download the Compatibility Pack (http://www.microsoft.com/downloads/details.aspx?familyid=941b3470-3ae9-4aee-8f43-c6bb74cd1466&displaylang=en) to be able to open in an older version.

Thanks 1114!

[Tom Bottiglieri]

I love how this mathematically demonstrates 1124's dramatic improvement.

It is interesting to look at how some teams improved over the season.

What's better? A team who was good but didn't improve much from event to event? Or a team who was not so good at their first event but was much better at their second?

For the latter, do you think the slope can stay steep? Or is this one of those tortoise/hare situations?

[kjolana1124]

What's better? A team who was good but didn't improve much from event to event? Or a team who was not so good at their first event but was much better at their second?

For the latter, do you think the slope can stay steep? Or is this one of those tortoise/hare situations?

I see it almost like an exponential curve. Teams that tend to do better over the course of a few events have a habit of leveling off. Whether it be from getting too sure of yourself, falling into a pattern, or just reaching your peak a little too soon, they eventually begin to stabilize. Either that, or in the effort to keep improving they end up becoming weaker.

Continuous improvement =/= continuous work. Sometimes all a team needs is a better mindset between competitions, not a better robot.

[PaW]

Annual thanks to you and your team, Karthik! Nice work, as always.

[Justin Montois]

wow. They scout like they build robots. Talk about a powerhouse.

[J Flex 188]

I simply can't fathom how Roberto Rotolo keeps pumping this stuff out year after year with such a high degree of quality too! He's definitely the backbone to that skilled group of individuals, its nuts, every year!

[Zach O]

Wow! What a database! Very impressive. Thank you for posting it.

[dsmoker]

For a team like ours, that just qualified for the Championship last weekend and is just barely able to send a skeleton crew (7 kids, 4 mentors) down to Atlanta, this data base is an invaluable resource. Plus, we look pretty good!! Thank you for sharing this information with the FIRST community.

[Chris Fultz]

There may be some errors, but I'm confident the data is 97.1114% accurate.

Karthik, is that accuracy in Canadian or American?

Do you know the exchange rate?


Seriously - awesome work and awesome data.
Much appreciated.

[Drivencrazy]

FIRST off Karthik thanks alot. This is by far one of the most proffessional scouting databases I have seen.

Can you tell me where the Calculated Contribution comes from or point me to a forum that does?

Once again thanks and good luck at the champs.

[Karthik]

FIRST off Karthik thanks alot. This is by far one of the most proffessional scouting databases I have seen.

Can you tell me where the Calculated Contribution comes from or point me to a forum that does?

Once again thanks and good luck at the champs.

This is explained in a few other places on these forums, but it's probably best that I put an explanation in this thread.

Going into an event like the Championship, it's impossible to watch video on all 344 teams. So, how can you get an idea of what each team's scoring potential is? Well, one way to do it would be just to look at their average score per match. Nice and simple, but it only tells you a small part of the story. Since FIRST matches involve alliances, an average score does not isolate the individual performance of a given team. So if Team XYZ repeatedly plays matches with great teams, their average score will not necessarily be an accurate indicator of the team's performance.

So, how do we isolate the impact of a single team on a match? Simple, using good old linear algebra. For those of your familiar with advanced basketball statistics, the method I'm about to describe is very similar to "adjusted +/- (http://www.82games.com/barzilai2.htm)". Adjusted +/- has become a very popular tool among NBA franchises to try and figure out just how much each player is contributing on the court. NBA teams have long figured out that just looking at how many points a player scores does not always tell you how much they've impacted a team towards victory. There have been some great papers presented about this at the MIT Sloan Sports Analytics Conference (http://www.sloansportsconference.com/2010/).

So, here's how it works. For each alliance during qualifying matches at a regional, we set up an equation. Say Teams i, j, k were on an alliance together and scored s points. Out equation would be

T_i + T_j + T_k = s

where T_i, T_j & T_k are variables representing those teams. So if there are m teams at a regional, and n matches, we now have equations that give us a m x 2n matrix. We then solve this matrix for our variables, and voila you have each team's Calculated Contribution.

Why is Calculated Contribution so valuable this year? Well, consider a team that plays the midfield and is great at supplying balls to the home zone, but rarely scores them. If your scouts are just tracking goals scored, they might get a big zero. But their Calculated Contribution (if run over a large enough sample size) would should a higher value reflecting the points they helped their alliance score.

There are a lot more subtle details as to why Calculated Contribution is a good tool, and even more that expose some shortcomings. Unfortunately it's a bit much to go into on CD post while I'm eating lunch. I'll talk a lot about this at my Effective FIRST Strategy Seminar in Atlanta (http://www.chiefdelphi.com/forums/showthread.php?t=84958).

Hope this helped.

[hipsterjr]

This sheet on an ipod touch will be invaluable. Awesome job.
*ekk* (entered my team) looks like the data on us reaffirms our thoughts that we should focus on defense. This program has even aided in scouting my own team and influencing our stratagy! Now that is a testiment to it's effectivness.

[Tom Bottiglieri]

For the latter, do you think the slope can stay steep? Or is this one of those tortoise/hare situations?
I graphed the Calculated Contributions of a few teams over time. I'd like to do this again after champs and compare the results.

http://i44.tinypic.com/1zdryme.jpg

[Racer26]

It would be interesting to plot weighted CC over time from year to year and see if teams like 1114 are consistently getting better at FRC.

[pfreivald]

I like the calculated contributions bit, especially since we built a midfield robot that is designed to work as a part of an overall alliance, and was not designed as a one-robot scoring machine.

Good stuff!

[kjolana1124]

I graphed the Calculated Contributions of a few teams over time. I'd like to do this again after champs and compare the results.

http://i44.tinypic.com/1zdryme.jpg

Interesting. Can you please explain what the x and y axis each represent? I'm a little confused.

[Ian Curtis]

I presume the y axis is the Calculated Contribution, and the x axis is the event. So for you guys, at x=1 is your performance at WPI, and at x=2 is your performance at CT. Excel probably put in the .5s on its own.

[kjolana1124]

Oh, I see. Thanks :)

[Drivencrazy]

"Hope this helped."

Yes indeed it did.

Thanks again and I look forward to your seminar. I'll be there.

[Lil' Lavery]

edit: Woops, sorry about that. My general theory still applies.

I agree with Karthik's statements that "Calculated Contributions" (aka "Offensive power rankings" and "Defensive power rankings") worked very well in Overdrive given the more "isolated" nature of the game (still by no means an isolated event, though). I also agree with his statement that they should work much better for Breakaway than they did for Lunacy. However, as awesome as they might be, they still aren't telling the full story and should be taken with a grain of salt.

However, they still can be a useful tool. If you find yourself with a chance to be in a picking position Friday night at an event and don't really have much/any (reliable) scouting data, go ahead and run these types of numbers. Pick out some of the higher ranking teams in whichever attributes you're looking for. Have whatever people you can spare (maybe just your "media person," for example) do whatever they can to watch a match or two of these targeted teams the next day and see if their performance on the field backs up what the numbers say, and base your selection choices off of that.

That being said, this database is still one of the most wonderful resources in FRC. 1114 has done another amazing job on it, just like every year. It's tremendously helpful to find quick information about any team, even those not going to Championship. Kudos again to the Simbots for incredible work.

I do have one request, though. Is there any chance you guys can add in the Dean's list to a teams' profile? You list if a team won a WFFA award, so why not Dean's list as well?

[Karthik]

Hmmmm... 1712 ranked 3rd in the CDefC in Philadelphia. Interesting, very interesting. :rolleyes:
...
However, as awesome as they might be, they still aren't telling the full story and should be taken with a grain of salt. 1712 never once played defense in our qualification matches in Philadelphia (though we did in the eliminations, but those aren't factored in to 1114's CC). It's still quite possible that perhaps some of our strategies were more defensive oriented than other alliances' (though there were at least three matches where our alliance decided to forgo any defensive efforts). These calculated contributions are still subject to hiccups like this and biasing based on strategies.


Sean,

"Average Defensive Score" is actually not one of our advanced metrics. It's simply the average score of all the alliances a team faced in qualifying matches. So, a high Average Defensive Score means that the alliances you faced scored a lot of points. This year we ranked that stat in descending order on the theory that in a game with scoring that doesn't reward defense in the qualifying round, a team with a high defensive score was either consistently playing stronger alliances or consistently scoring for their opponent. Regardless, Average Defensive Score has no link to Calculated Contribution or OPR in anyway.

I do have one request, though. Is there any chance you guys can add in the Dean's list to a teams' profile? You list if a team won a WFFA award, so why not Dean's list as well?

We wanted to do this, unfortunately there were far too many gaps on the FIRST website. Most events displayed the winner of the award, but did not have team associated with the winner. We decided that it we couldn't list all the winners accurately, it would be better to leave the award out. (There were gaps with the WFFA's as well, but not as many and we were able to fill those with some Googling)

[Vikesrock]

We wanted to do this, unfortunately there were far too many gaps on the FIRST website. Most events displayed the winner of the award, but did not have team associated with the winner. We decided that it we couldn't list all the winners accurately, it would be better to leave the award out. (There were gaps with the WFFA's as well, but not as many and we were able to fill those with some Googling)

You just didn't ask the right people ;)

Between myself and EricH we managed to track down all but one of the gaps for Fantasy FIRST scoring purposes.

[Chris is me]

You just didn't ask the right people ;)

Between myself and EricH we managed to track down all but one of the gaps for Fantasy FIRST scoring purposes.

Can us FF people supply all that information to 1114?

[Vikesrock]

Can us FF people supply all that information to 1114?

Done.

[MikeReilly]

First of all, amazing stuff. Love it, wish I had time to really understand it before the Dome but I will be digesting it and learning it before next year.

Second, is there any consideration to a version with revised scores for Week 1 match winners? I'm guessing that would have an effect with the +5 for winning a match. I'm going to try and play with that myself, but wondering if you did some work on it and saw no major difference before I do.

Thanks!