Considering how important gear scoring will be this year, we should figure out the best way to use FIRST’s scoring data to come up with a gear contribution for each team. However, since we aren’t actually directly provided with the number of gears scored each match, we will have to get creative with using the data provided to us. Let’s try to find the best way to use this info to create one or more useful gear contribution metrics. I want to start this conversation early so that we can have some level of standardization between common data analysis tools.
The most relevant categories we have to work with are:
rotor1Auto
rotor2Auto
rotor1Engaged
rotor2Engaged
rotor3Engaged
rotor4Engaged
rotorRankingPointAchieved (Quals only)
autoRotorPoints
teleopRotorPoints
rotorBonusPoints (Playoffs only)
Note that the last 4 categories can be found from the first 6.
The easiest suggestion I have would be to create a “scored gears” contribution which can be found by assigning a number of scored gears to each rotor. Essentially, rotor1Engaged= 1 scored gear, rotor2Engaged = 3, rotor3Engaged = 7, and rotor4Engaged = 13. An alternative to this method would be to subtract 1 scored gear from each of these because every alliance has a free gear which will be scored in the grand majority of matches. Doing this would make rotor1Engaged = 0 scored gears, rotor2Engaged = 2, rotor3Engaged = 6, and rotor4Engaged = 12.
I don’t actually have a very strong opinion on which of these to use, as linear regression methods should provide exactly the same results just with an offset of 0.33 for each team. I do however want standardization, so let’s try to come to a consensus and stick with it.
I have some other ideas churning in my head that I’ll probably post soon, but I’m curious to hear other ideas for how we can best utilize this information.
I don’t think there’s a good (reliable) way to determine how many gears were scored by a team via FIRST data. Almost every assumption that is made for the reliability of OPR-style data (high number of ‘scores’, low point value for each ‘score’, no over-lapping or multiplicative scoring, no pre-requisite to scoring, etc.) is broken with gears this year.
I think the best way to rank teams by “gears-scored” is by good, old-fashioned hand scouting. If you have a 2min 30 second attention span, and can count to ~12, this should be easy enough. It requires a team to have at least 6 people available to watch each match (remember, adults can scout too). On the other hand, scouting fuel this year is almost impossible by traditional methods, and I think will rely heavily on OPR-style scouting. A neat (unintentional?) combo by the GDC.
For people like you (Caleb, Ether, and the other CD stat mavens), I think you’re going to have a very rough time publishing an end-of-year spreadsheet ranking all ~3000 teams in terms of gear scoring ability. I wish you luck, but short of significant breakthroughs in how statistics are used in FIRST, I really doubt anything you do will be reliably accurate. I’m hoping that in 2 months, I’ll have to admit I was wrong.
I had imagined this is how the data would be pushed, so after thinking about this all season, this is also the best I’ve come up with.
Using raw match data, we could come up with estimates for total gears scored and auto gears scored, and then subtract these results to yield a teleop gears scored estimate. From there, Least Squares can be applied to derive a component OPR-like number.
In the interest of standardization, here is some psuedocode of how I think these estimates should be calculated:
function matchTotalGears(match):
if match.rotor4Engaged: return 12
if match.rotor3Engaged: return 6
if match.rotor2Engaged: return 2
else: return 0
function matchAutoGears(match):
if match.rotor2Auto: return 3
if match.rotor1Auto: return 1
else: return 0
function matchTeleopGears(match):
return matchTotalGears(match) - matchAutoGears(match)
What concerns me about doing that method is that some matches could have a negative value for teleop gears, which is nonsensical to me. Consider a match where one gear is scored in auto and only the reserve gear is placed in teleop. By the above method, matchAutoGears would return 1, matchTotalGears would return 0, and matchTeleopGears would return -1.
Here is a possible alternative which avoids my above complaint. This would treat the reserve gear just like any other teleop gear. Let’s call Brian’s original pseudocode Option 1, and this pseudocode Option 2.
function matchTotalGears(match):
if match.rotor4Engaged: return 13
if match.rotor3Engaged: return 7
if match.rotor2Engaged: return 3
else: return 1
function matchAutoGears(match):
if match.rotor2Auto: return 3
if match.rotor1Auto: return 1
else: return 0
function matchTeleopGears(match):
return matchTotalGears(match) - matchAutoGears(match)
Considering it’s an extremely narrow set of possibilities, I would think it would be easy to add some edge-case handling code for the rare matches where negative scores are possible. In these cases, teleop would simply be zero. In most other cases, I think Brian’s method seems reasonable, but I also haven’t thought through the problem terribly much.
function matchTotalGears(match):
if match.rotor4Engaged: return 12
if match.rotor3Engaged: return 6
if match.rotor2Engaged: return 2
else: return 0
function matchAutoGears(match):
if match.rotor2Auto: return 3
if match.rotor1Auto: return 1
else: return 0
function matchTeleopGears(match):
if matchTotalGears(match) - matchAutoGears(match) < 0: return 0
else: return matchTotalGears(match) - matchAutoGears(match)
My biggest complaint about this method is that we lose the nice linear relationship between these 3 parameters. I tend to like linear relationships, so I would probably prefer one of the first two options over this one.
My one problem with this method is that when the time comes to regress it, the reserve gear will figure into the estimated contributions, which I think makes the contributions less meaningful.
Here is my proposal for Option 4, which maintains the linearity of the properties by incorporating auto into calculating matchTotalGears:
function matchTotalGears(match):
if match.rotor4Engaged: return 12
if match.rotor3Engaged: return 6
if match.rotor2Auto: return 3
if match.rotor2Engaged: return 2
if match.rotor1Auto: return 1
else: return 0
function matchAutoGears(match):
if match.rotor2Auto: return 3
if match.rotor1Auto: return 1
else: return 0
function matchTeleopGears(match):
return matchTotalGears(match) - matchAutoGears(match)
Option 5: Covers the edge case where an alliance scores no gears and also forgets about the reserve gear.
function matchTotalGears(match):
if match.rotor4Engaged: return 12
if match.rotor3Engaged: return 6
if match.rotor2Auto: return 3
if match.rotor2Engaged: return 2
if match.rotor1Auto: return 1
if match.rotor1Engaged: return 0
else: return -1
function matchAutoGears(match):
if match.rotor2Auto: return 3
if match.rotor1Auto: return 1
else: return 0
function matchTeleopGears(match):
return matchTotalGears(match) - matchAutoGears(match)
Basically this would provide extra punishment for alliances with extremely bad pilots.
Here is another metric I am thinking of pursuing this year in addition to the metric described above. While the above metrics looks at the number of “scored” gears for each match, I would like to attempt to track the number of gears placed, regardless of whether they are scored or not. I really want to find the metric that has the most predictive power for future matches, even if it is not defined as clearly as the above metric.
My plan is to use the same general structure as the above methods, but to use an alternative number of gears for each possibility. To do this, I will wait until week 1 events are completed, and then try a bunch of different combinations of gear values for each combination of auto rotors and engaged rotors, and use the mapping which has the most predictive power. For example, I might find that the following mapping has the most predictive power for future matches:
function matchTotalGears(match):
if match.rotor4Engaged: return 13.6
if match.rotor3Engaged: return 8.1
if match.rotor2Auto: return 4.4
if match.rotor2Engaged: return 3.3
if match.rotor1Auto: return 0.8
if match.rotor1Engaged: return 0.2
else: return -1.5
function matchAutoGears(match):
if match.rotor2Auto: return 2.8
if match.rotor1Auto: return 1.7
else: return -0.1
function matchTeleopGears(match):
return matchTotalGears(match) - matchAutoGears(match)
I anticipate that I would primarily use this metric for any predictive models I make. I am also planning to ask teams for week 1 scouting data so that I can see if this metric correlates better with scouted gear performance than the “scored gear contribution.” I imagine it will.
I am thinking of calling this the “estimated placed gears.”
Someone just needs to write a program to do vision processing on the scoreboard (streaming graphics) to get the times when the rotors are scored. And use that as a rate (gears/sec) for the alliance. If they score 4 rotors, then you’d have 4 rates (0->1 (1 gear),1->2 (2 gears),2->3 (4 gears), 3->4 (6 gears), so you’d have to consider which one best represents abilities. I’d probably eliminate Rate 1 and then use either the longest period or maximum rate as the rate for comparison. I lean toward the maximum teleop rate, but am not sure which would be better (I’d also consider combining any teleop rotor scores and calculate the rate from that). Without further data I’d put this all into a component OPR to break it out for each team.
This is just an idea between solely using API data and counting gears with scouts. I wonder a little if the FMS system could provide timestamps with rotors scored to facilitate a gear rate calculation.
I like this idea because it is possible to go back and score matches where no good scouting data exists as long as the matches are posted online.
I think it might be possible to get a fuel rating system working as well. I suppose that you’ll be able to tease out when high and low boilers get loaded with balls.
I remember seeing a “how to make a poker playing bot” tutorial that makes me think that this falls into the “just work” category (a lot of work but manageable sub problems).
Down the road, perhaps FIRST would be willing to share more FMS data (e.g. real-time scores with a reason for every change)?
I had this same idea a few weeks back, and sent a request to FIRST HQ. I was told the request was forwarded to the Engineering department, but haven’t heard anything else.
First Name: Caleb
Last Name: Sykes
Email Address: [email protected]
Area of Interest: FIRST Robotics Competition Questions / Comments:
To whom it may concern,
My name is Caleb Sykes, I am a mentor for FRC team 4536. I am planning to use data provided by the FRC Events API (FIRST Events API - v2.0 · Apiary) to create team performance statistics and match predictions this year. This API is an excellent resource, and I really appreciate having it available.
One of the issues I expect to run into this year is accurately determining how many gears are scored in each match. I anticipate that the number of placed gears will not be recorded each match, and hence will not be available from the API, although I would love it if I were wrong. I expect the API to record the number of rotors which are spinning at the end of each match, and this will indirectly provide me some information on the number of gears that were placed. I was wondering if it would be possible for the API to, in addition to providing the number of scored rotors, provide timestamps of when each rotor begins spinning. I think it is possible the FMS already records this information, in which case exposing it to the API would be very plausible. These timestamps could be used in many different interesting ways to determine how good teams are at scoring gears, and would hopefully not require much effort on your part to implement.
I would really appreciate it if the API exposed this information, or something like it, in order that better statistics could be generated.