How to use FRC Events data to determine gear ability

[Caleb Sykes]

Considering how important gear scoring will be this year, we should figure out the best way to use FIRST's scoring data to come up with a gear contribution for each team. However, since we aren't actually directly provided with the number of gears scored each match, we will have to get creative with using the data provided to us. Let's try to find the best way to use this info to create one or more useful gear contribution metrics. I want to start this conversation early so that we can have some level of standardization between common data analysis tools.

Here is the data we will have to work with (although this may be updated before competition): http://docs.frcevents2.apiary.io/#re.../score-details

The most relevant categories we have to work with are:
rotor1Auto
rotor2Auto
rotor1Engaged
rotor2Engaged
rotor3Engaged
rotor4Engaged
rotorRankingPointAchieved (Quals only)
autoRotorPoints
teleopRotorPoints
rotorBonusPoints (Playoffs only)

Note that the last 4 categories can be found from the first 6.

The easiest suggestion I have would be to create a "scored gears" contribution which can be found by assigning a number of scored gears to each rotor. Essentially, rotor1Engaged= 1 scored gear, rotor2Engaged = 3, rotor3Engaged = 7, and rotor4Engaged = 13. An alternative to this method would be to subtract 1 scored gear from each of these because every alliance has a free gear which will be scored in the grand majority of matches. Doing this would make rotor1Engaged = 0 scored gears, rotor2Engaged = 2, rotor3Engaged = 6, and rotor4Engaged = 12.

I don't actually have a very strong opinion on which of these to use, as linear regression methods should provide exactly the same results just with an offset of 0.33 for each team. I do however want standardization, so let's try to come to a consensus and stick with it.

I have some other ideas churning in my head that I'll probably post soon, but I'm curious to hear other ideas for how we can best utilize this information.

[CVR]

I don't think there's a good (reliable) way to determine how many gears were scored by a team via FIRST data. Almost every assumption that is made for the reliability of OPR-style data (high number of 'scores', low point value for each 'score', no over-lapping or multiplicative scoring, no pre-requisite to scoring, etc.) is broken with gears this year.

I think the best way to rank teams by "gears-scored" is by good, old-fashioned hand scouting. If you have a 2min 30 second attention span, and can count to ~12, this should be easy enough. It requires a team to have at least 6 people available to watch each match (remember, adults can scout too). On the other hand, scouting fuel this year is almost impossible by traditional methods, and I think will rely heavily on OPR-style scouting. A neat (unintentional?) combo by the GDC.

For people like you (Caleb, Ether, and the other CD stat mavens), I think you're going to have a very rough time publishing an end-of-year spreadsheet ranking all ~3000 teams in terms of gear scoring ability. I wish you luck, but short of significant breakthroughs in how statistics are used in FIRST, I really doubt anything you do will be reliably accurate. I'm hoping that in 2 months, I'll have to admit I was wrong.

[forbes]

I agree that the data we have to work with is not ideal. I personally prefer the option that leaves out the free gear, though.

[Brian Maher]

Originally Posted by Caleb Sykes

The easiest suggestion I have would be to create a "scored gears" contribution which can be found by assigning a number of scored gears to each rotor. Essentially, rotor1Engaged= 1 scored gear, rotor2Engaged = 3, rotor3Engaged = 7, and rotor4Engaged = 13. An alternative to this method would be to subtract 1 scored gear from each of these because every alliance has a free gear which will be scored in the grand majority of matches. Doing this would make rotor1Engaged = 0 scored gears, rotor2Engaged = 2, rotor3Engaged = 6, and rotor4Engaged = 12.

I had imagined this is how the data would be pushed, so after thinking about this all season, this is also the best I've come up with.

Using raw match data, we could come up with estimates for total gears scored and auto gears scored, and then subtract these results to yield a teleop gears scored estimate. From there, Least Squares can be applied to derive a component OPR-like number.

In the interest of standardization, here is some psuedocode of how I think these estimates should be calculated:

function matchTotalGears(match):
     if match.rotor4Engaged: return 12
     if match.rotor3Engaged: return 6
     if match.rotor2Engaged: return 2
     else: return 0

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

[Caleb Sykes]

Originally Posted by Brian Maher

In the interest of standardization, here is some psuedocode of how I think these estimates should be calculated:

function matchTotalGears(match):
     if match.rotor4Engaged: return 12
     if match.rotor3Engaged: return 6
     if match.rotor2Engaged: return 2
     else: return 0

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

What concerns me about doing that method is that some matches could have a negative value for teleop gears, which is nonsensical to me. Consider a match where one gear is scored in auto and only the reserve gear is placed in teleop. By the above method, matchAutoGears would return 1, matchTotalGears would return 0, and matchTeleopGears would return -1.

[Caleb Sykes]

Here is a possible alternative which avoids my above complaint. This would treat the reserve gear just like any other teleop gear. Let's call Brian's original pseudocode Option 1, and this pseudocode Option 2.

function matchTotalGears(match):
     if match.rotor4Engaged: return 13
     if match.rotor3Engaged: return 7
     if match.rotor2Engaged: return 3
     else: return 1

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

[cadandcookies]

Originally Posted by Caleb Sykes

What concerns me about doing that method is that some matches could have a negative value for teleop gears, which is nonsensical to me. Consider a match where one gear is scored in auto and only the reserve gear is placed in teleop. By the above method, matchAutoGears would return 1, matchTotalGears would return 0, and matchTeleopGears would return -1.

Considering it's an extremely narrow set of possibilities, I would think it would be easy to add some edge-case handling code for the rare matches where negative scores are possible. In these cases, teleop would simply be zero. In most other cases, I think Brian's method seems reasonable, but I also haven't thought through the problem terribly much.

[Caleb Sykes]

Originally Posted by cadandcookies

Considering it's an extremely narrow set of possibilities, I would think it would be easy to add some edge-case handling code for the rare matches where negative scores are possible. In these cases, teleop would simply be zero. In most other cases, I think Brian's method seems reasonable, but I also haven't thought through the problem terribly much.

Option 3:

function matchTotalGears(match):
     if match.rotor4Engaged: return 12
     if match.rotor3Engaged: return 6
     if match.rotor2Engaged: return 2
     else: return 0

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     if matchTotalGears(match) - matchAutoGears(match) < 0: return 0
     else: return matchTotalGears(match) - matchAutoGears(match)

My biggest complaint about this method is that we lose the nice linear relationship between these 3 parameters. I tend to like linear relationships, so I would probably prefer one of the first two options over this one.

[Brian Maher]

Originally Posted by Caleb Sykes

Here is a possible alternative which avoids my above complaint. This would treat the reserve gear just like any other teleop gear. Let's call Brian's original pseudocode Option 1, and this pseudocode Option 2.

function matchTotalGears(match):
     if match.rotor4Engaged: return 13
     if match.rotor3Engaged: return 7
     if match.rotor2Engaged: return 3
     else: return 1

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

My one problem with this method is that when the time comes to regress it, the reserve gear will figure into the estimated contributions, which I think makes the contributions less meaningful.

Here is my proposal for Option 4, which maintains the linearity of the properties by incorporating auto into calculating matchTotalGears:

function matchTotalGears(match):
     if match.rotor4Engaged: return 12
     if match.rotor3Engaged: return 6
     if match.rotor2Auto: return 3
     if match.rotor2Engaged: return 2
     if match.rotor1Auto: return 1
     else: return 0

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

[Caleb Sykes]

Originally Posted by Brian Maher

Here is my proposal for Option 4, which maintains the linearity of the properties by incorporating auto into calculating matchTotalGears:

This one is my new favorite. My ranking of the choices is:
Option 4 > Option 2 > Option 1 > Option 3

[Caleb Sykes]

Option 5: Covers the edge case where an alliance scores no gears and also forgets about the reserve gear.

function matchTotalGears(match):
     if match.rotor4Engaged: return 12
     if match.rotor3Engaged: return 6
     if match.rotor2Auto: return 3
     if match.rotor2Engaged: return 2
     if match.rotor1Auto: return 1
     if match.rotor1Engaged: return 0
     else: return -1

function matchAutoGears(match):
     if match.rotor2Auto: return 3 
     if match.rotor1Auto: return 1
     else: return 0

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

Basically this would provide extra punishment for alliances with extremely bad pilots.

[Caleb Sykes]

Here is another metric I am thinking of pursuing this year in addition to the metric described above. While the above metrics looks at the number of "scored" gears for each match, I would like to attempt to track the number of gears placed, regardless of whether they are scored or not. I really want to find the metric that has the most predictive power for future matches, even if it is not defined as clearly as the above metric.

My plan is to use the same general structure as the above methods, but to use an alternative number of gears for each possibility. To do this, I will wait until week 1 events are completed, and then try a bunch of different combinations of gear values for each combination of auto rotors and engaged rotors, and use the mapping which has the most predictive power. For example, I might find that the following mapping has the most predictive power for future matches:

function matchTotalGears(match):
     if match.rotor4Engaged: return 13.6
     if match.rotor3Engaged: return 8.1
     if match.rotor2Auto: return 4.4
     if match.rotor2Engaged: return 3.3
     if match.rotor1Auto: return 0.8
     if match.rotor1Engaged: return 0.2
     else: return -1.5

function matchAutoGears(match):
     if match.rotor2Auto: return 2.8
     if match.rotor1Auto: return 1.7
     else: return -0.1

function matchTeleopGears(match):
     return matchTotalGears(match) - matchAutoGears(match)

I anticipate that I would primarily use this metric for any predictive models I make. I am also planning to ask teams for week 1 scouting data so that I can see if this metric correlates better with scouted gear performance than the "scored gear contribution." I imagine it will.

I am thinking of calling this the "estimated placed gears."

[mac]

If a match has 8 or 9 gears applied 3 rotors will turn. How will you track and or count the gears above 6? God Bless

[Brian Maher]

Originally Posted by mac

If a match has 8 or 9 gears applied 3 rotors will turn. How will you track and or count the gears above 6? God Bless

Given the data we are presented, it is not possible. We are doing our best to approximate based on what we have.