Top 25 ETCs after Week 3

[mwtidd]

I want to thank everyone for the help in this thread.

[Adam Freeman]

I think anyone that was at West Michigan or saw the webcast would agree that picking 67, even for just a minibot would have been a better choice. Ours was just as reliable and faster.

Our minibot was launched in 15 of our 18 matches...the other three where it did not were from other issues that resulted in failed launches early in the competition or being essentially disabled (last match). Plus the one elim match where it hit first but did not register.

No offense to 2767, but they were not carrying teams with scoring...in fact they were basically just a defender and minibot for most of the competition...so yes that had a very very dependable minibot. But not the "best" at the WM.

67 and 2054 had the fastest and most reliable minibots there. We would have launched them for #1 and #2 in any match we were both fully functioning in (my opinion, don't ask 1918 or 27 ) .

As for having another metric to judge teams on...I think it would be good if it was accurate, but more so for the later picks and only if you were actually watching all the competition matches.

I will have to pull our scouting data for what positions our minibot finished in each match and see what our actual minibot point contribution was.

I know in some matches we just launched late in the count and allowed our partners to finish first. We did not get on the trigger button quickly until late Saturday morning and into the afternoon.

[speedbuggy76]

Here's my calculation file for Peachtree using the dataset from the FIRST website (thus there are no ERC or EMC calculations). Can you take a look at it and see why my values are different than yours? Most of the calculations are done using formulas and VBA, let me know if anything is unclear.

Also, how do you import data from Twitter to do your calculations? For my data, I had to copy/paste the table of values for rankings, match results, and playoff results into a text file, which I then imported into excel as a tab delimited file. Probably not the easiest approach, and I'd like to make it more versatile to handle other regionals.

Link to peachtree calculations: http://dl.dropbox.com/u/11470/peacht...lculations.xls

[mwtidd]

Originally Posted by speedbuggy76

Here's my calculation file for Peachtree using the dataset from the FIRST website (thus there are no ERC or EMC calculations). Can you take a look at it and see why my values are different than yours? Most of the calculations are done using formulas and VBA, let me know if anything is unclear.

Also, how do you import data from Twitter to do your calculations? For my data, I had to copy/paste the table of values for rankings, match results, and playoff results into a text file, which I then imported into excel as a tab delimited file. Probably not the easiest approach, and I'd like to make it more versatile to handle other regionals.

Link to peachtree calculations: http://dl.dropbox.com/u/11470/peacht...lculations.xls

unfortunately I'm on a mac, so it looks like the VB macros aren't working quite right. i'm getting all div 0s

For the twitter feed I use java, to grab each tweet and parse the match data out of each tweet.

[mwtidd]

Originally Posted by Adam Freeman

No offense to 2767, but they were not carrying teams with scoring...in fact they were basically just a defender and minibot for most of the competition...so yes that had a very very dependable minibot. But not the "best" at the WM.

Thanks for the insights!

What's interesting about EMC and ERC is that it does reveal single task teams. 2767 had a 3 for a ERC which is essentially a zero.

Two things probably happened with your EMC, first you probably played with better alliance partners so they probably stole a larger portion of the minibot points than 2767's partners. Also you had a few 0s in there which didn't help either. Team 2767 only had one 0.

[Adam Freeman]

Originally Posted by lineskier

again please disprove these claims but:
2054 and 67 had the 2 best robots or most reliable robots
1918 was very good or very reliable
74 was good and reliable

now these statements may be wrong, I'm looking for insights on them

67's auton was very good (15/18), with twice being knocked off track by partners.
67 scored on average ~6.5 tubes/match.
67's minibot was very reliable and fast, as long as the drive team gave the minibot the respect it deserved.

2054 had an absolutely spot on autonomous in every match.
2054 scored on average ~3.5 tubes/match.
2054's minibot was very dependable, launching almost every match for 1st or 2nd place (have to check the data we have to be certain).

1918 auton was 50/50
1918 scored ~4.5 tubes/match. Capable of putting up 9 if given the chance. Had a couple of cRIO issues during qualifications and one match were a tube got stuck on their robot, so they couldn't score anymore.
1918 had troubles launching their minibot in early matches, seemed to get better as the competition went on. Very fast minibot...probably close to 2054 and 67 in raw speed, but slower in deployment.

74 was basically a clone of 2054....very dependable, very reliable. But, they worked up to that reliability quickly Friday morning.

27 was basically the same as 1918. Very good scorer (very dangerous if left alone), but had some arm issues in qualifications. Auton was spotty. Minibot was really good... I think they did not launch twice the entire weekend.

I will try to pull the data for these teams and calculate the final numbers. By far the top 5 teams at the competition.

226, 1718, and 2000 were the other teams that were pretty strong too.

[mwtidd]

Originally Posted by Adam Freeman

67's auton was very good (15/18), with twice being knocked off track by partners.
67 scored on average ~6.5 tubes/match.
67's minibot was very reliable and fast, as long as the drive team gave the minibot the respect it deserved.

2054 had an absolutely spot on autonomous in every match.
2054 scored on average ~3.5 tubes/match.
2054's minibot was very dependable, launching almost every match for 1st or 2nd place (have to check the data we have to be certain).

1918 auton was 50/50
1918 scored ~4.5 tubes/match. Capable of putting up 9 if given the chance. Had a couple of cRIO issues during qualifications and one match were a tube got stuck on their robot, so they couldn't score anymore.
1918 had troubles launching their minibot in early matches, seemed to get better as the competition went on. Very fast minibot...probably close to 2054 and 67 in raw speed, but slower in deployment.

74 was basically a clone of 2054....very dependable, very reliable. But, they worked up to that reliability quickly Friday morning.

27 was basically the same as 1918. Very good scorer (very dangerous if left alone), but had some arm issues in qualifications. Auton was spotty. Minibot was really good... I think they did not launch twice the entire weekend.

I will try to pull the data for these teams and calculate the final numbers. By far the top 5 teams at the competition.

226, 1718, and 2000 were the other teams that were pretty strong too.

Thanks!

and its pretty cool cause if you look at ETC for those last 3:

226- 15
1718- 15
2000- 14

It seems down matches really kill teams. Teams with a high ETC are essentially reliable every qualifying match or are really really good (having to make up for a zero is not easy).

2054 had a robot score of 25 where you had a robot score of 20. The difference there was probably the autonomous. I tried to get first to include autonomous in the tweet... maybe next year.

I'm going to try to figure out how the race was actually finished. Its entirely possible to figure out which alliance finished in which places, and then use that to predict who finished in which position for each team.

[JesseK]

Oy, I must have missed the thread that said what ETC and EMC are and why they're all the rave right now. I mean, OPR in and of itself isn't a really good metric since minibots more of a binary thing than a linear scoring thing.

IMO, judging from what I watched in Weeks 1-3, Tubes may give a decent gauge for OPR as it relates to tubes if the OPR excludes minibot. Yet minibots themselves can only come from watching the field since the data doesn't give a specific team for each bonus.

[mwtidd]

Originally Posted by JesseK

Oy, I must have missed the thread that said what ETC and EMC are and why they're all the rave right now. I mean, OPR in and of itself isn't a really good metric since minibots more of a binary thing than a linear scoring thing.

IMO, judging from what I watched in Weeks 1-3, Tubes may give a decent gauge for OPR as it relates to tubes if the OPR excludes minibot. Yet minibots themselves can only come from watching the field since the data doesn't give a specific team for each bonus.

Maybe I should specify, ETC, EMC and ERC are a set of metric I came up with. They are still in their very early stages. Where they can tell you some things about some robots, for most robots its turning out to be more of a reliability metric. If you have bad match your ETC drops dramatically.

Also I don't mean to inflate the value of the ETC metric, they are simply something I have been working on and trying to get real feedback on.

I believe that by using my EMC I can estimate the way the race in each match finished, I am working on this right now. Hopefully this will give us a better reading on minibots. I also am working on a filter for the ERC that eliminates essentially non-contributors.

ETC stands for estimated teams contribution
ERC estimated robots contribution
EMC estimated minibot contribution

[JesseK]

I see your point about ranking the teams relative to each other; yet there will still have to be a sort of 'confidence' factored in before we can rely on it any sort of relativistic data like that. For example, the teams ranked 4-12 may be within a Y% of each other, therefore we really can only say with X% confidence that their true rankings are correct. If the confidence levels are low across the board, then the algorithm itself isn't as useful as it may appear.

Since it's a learning exercise, I'd encourage you to lay out your assumptions, figure out which statistical metrics are better (mean versus median, and does std. dev. help explain anything?), as well as the methods of the algorithm (why average minibot over 3 teams instead of 2 poles?). It can only help you more correctly align the algorithm's results with the reality on the field.

Having a predicted match score based upon N-assumptions is much more valuable than having something that says "This team is 10% better than your team".

[mwtidd]

Originally Posted by JesseK

I see your point about ranking the teams relative to each other; yet there will still have to be a sort of 'confidence' factored in before we can rely on it any sort of relativistic data like that. For example, the teams ranked 4-12 may be within a Y% of each other, therefore we really can only say with X% confidence that their true rankings are correct. If the confidence levels are low across the board, then the algorithm itself isn't as useful as it may appear.

Since it's a learning exercise, I'd encourage you to lay out your assumptions, figure out which statistical metrics are better (mean versus median, and does std. dev. help explain anything?), as well as the methods of the algorithm (why average minibot over 3 teams instead of 2 poles?). It can only help you more correctly align the algorithm's results with the reality on the field.

Having a predicted match score based upon N-assumptions is much more valuable than having something that says "This team is 10% better than your team".

Thanks for the feedback. Where the EMC is calculated using percentages there is usually a team that gets a very small share of the overall EMC.

But after actually running through the race options, 30 is the only score that can happen in 2 different ways, and is very easy to check against, basically if the opposing alliance score is 45 you came in second and forth, if not you came in first alone.

Then I will be attributing the minibots score to the appropriate teams based on EMC.

Confidence metrics are not sometimes I have dealt with much, but I am definitely going to research them. Again I am a long ways away from being happy. But there are times where these statistics paired with actual scouting data present some interesting information, that OPR may leave out.

[The Lucas]

Originally Posted by JesseK

Oy, I must have missed the thread that said what ETC and EMC are and why they're all the rave right now. I mean, OPR in and of itself isn't a really good metric since minibots more of a binary thing than a linear scoring thing.

IMO, judging from what I watched in Weeks 1-3, Tubes may give a decent gauge for OPR as it relates to tubes if the OPR excludes minibot. Yet minibots themselves can only come from watching the field since the data doesn't give a specific team for each bonus.

I think Logomotion is a game of diminishing returns which is difficult for all the metrics. So really good alliances are often less than the sum of their parts. It is easiest to visualize this by looking at JVN's scoring matrix. Those Ubertubes you put up high in auton are worth big points (2-4 times more than teleop). Then you form a high logos over those Ubertubes, again big points. Now you are out of High pegs so each additional tube is only worth a fraction of the points. If you are really great, you have to start working on the low pegs, where you have to wonder: At 1pt/tube is this even worth it or should I get ready for the minibot race? Additionally, 3 great hangers would still have to share 2 racks, which limits each individual scoring potential.

Minibot race is the same way. if you have the fastest minibot out there it is worth 30 pts. Your partner's mini bot is a tenth of a sec slower but it is only worth 20 pts. Your 3rd partner my have an equally fast minibot but it is always worth 0 since they cant legally deploy it. Now lets say your opponents have ridiculously slow minibots (like 15 sec). They still can get 25 combined points as long as they deploy them. Chances are in some matches they would get 50 pts with their slow minibots because their opponents dont have any.

Metrics like OPR are a good jumping off point but you still need real scouting to make an educated decision. ETC is still a work in progress but it is interesting to see the minibot scores separated.

[techvikesmom]

all this information is very interesting to see and read. One comment about our kids is that they picked what they saw in the 2 days of qualifying rounds. The kids picked their alliance partners, not the mentors, not the all the statistics!! They did a great job on picking alliance, and playing and they played a great alliance. It was an exciting and awesome weekend for all the robots and their teams. Keep up the great alliances at Niles this weekend.

[mwtidd]

If everyone could take a look at this, and tell me what insights you get this would really help.

This is using my minibot prediction algorithm, and what I see here is HUGE minibot inflation. Basically I think one of these teams is stealing points from other teams...

it looks like to calculate minibots accurately you actually have to look at it starting from the match, rather than doing an initial minibot rating by total points.

values left to right
etc
erc
emc

[Wayne TenBrink]

Originally Posted by Jim Zondag

Me either, and I'm not convinced yet that I need any new acronyms or methods. Everyone keeps obsessing about who is the best, but this is irrelevant until Einstien. At a real tournament, you don't really need a spreadsheet to tell you who is the best. This is obvious; my mom can tell you who is the best with no data at all. A thousand varied analytical methods will be able to give you the top 8 teams in order.
You do, however, need good data and methods to tell you who is 16th-24th. This is where the real value add of competitve analysis is: not for the first pick, but for the second.

Every word is true (as is always the case with Jim). We don't spend much time obsessing over scouting data for our first round pick - the second round is where we focus our efforts.

As for West Michigan:
- 67 was the obvious first pick. They had the best all around package there. Scrap any data from Kettering.
- As #2 seed, 27 was our obvious pick. Another great all-around package.
- There were other good minibots and other decent hangers, but few that could put it all together.
- Our minibot didn't climb nearly as quickly as 27, 67, or 2054 (we'll try to change that by MSC). On Friday we deployed about 2/6 attempts with our backup mini. On Saturday, we made some corrections on the deployment, got the #1 mini back together, and were about 7/8 attempts. EMC data won't pick up on that, but anybody watching from the stands will.
- As with everyone, we had different issues at different times, all of which hurt us on ETC. RUSH also had technical problems Saturday morning. Things came together by eliminations, and standardized rating systems won't reflect that (for us or anybody else). That's why being there and knowing the status of things matters.
- I understand why our EMC is low, but I can't understand an ERC of 16 for 1918. I believe we are competitive with anybody out there for acquiring and hanging tubes.
- Alliances are definitely not the sum of their components. There is a point of diminishing returns in this game, and it is very easy to get in each other's way. If you look at the elimination data at WMD, our #2 alliance outscored the #1 alliance (albeit slightly) and our opponents underscored theirs over the course of the quarters and semi's, despite the fact that we worked through some of the higher seeds.

In general, an observer can tell who is capable of doing what, and they instinctively know when to dismiss certain match data. We tend to rate teams based on their expected capabilities when things are working properly. Recent performance is generally a better indication of future performance than historical average because of all the continuous improvement that goes on in FIRST. I think that ETC's might better reflect true capabilities if they omitted one or two of the lowest matches, or perhaps looked at a 10 match running average, like some golfing handicaps.

Regardless of the imperfection of any system, they have their merits. Many thanks to those who do the work to make them possible.

Edit: Another random observation from West MI - There was one qual match between 27 & 67. Other than that, neither 27 nor 1918 ever played against 67 or 2054 until the finals.