New scouting metric - help automating it (TBA)?

[nuclearnerd]

Hi All

I came up with a (new?) scouting metric for this weekend's NYC regional. We were concerned that with this year's game being so dependant on assist scores, the "match difficulty" would have a larger than normal influence on seed. I'm really happy with the result, so I thought I might share it here in case there is some interest in standardizing it. The metric was calculated as follows:

For each team:

For each match:

For each team in the match (*other* than the team of interest TOI):

Scrape the current stats from TBA or usfirst.org

Compute the "match difficulty" score(s) from the stats

Display a table showing the difficulty for each match
Sum the difficulty for each match to get a "quals difficulty" score

The result table looked something like this:

Team 9999:
Qual#    DiffScore (negative is harder)
9          -5
19         -10
29         2
39         10
TOTAL      -3

The heart of the table is the "match difficulty" score. This weekend we tried the following two scores:

1) Sum the QS score of the TOI's alliance mates, and subtract the QS score of the TOI's opponents. So if the mates were sitting with with QS Scores of 10 and 12, and the opponents were 8, 8 and 5, the DiffScore would be 1.

2) One of our other mentors suggested that the difficulty of a match in this game is not linear with the abilities of the other robots on the field. Having two poorly performing mate is *much* more damaging than having two just-average mates. So we made a second score which was 1 or 0 if both mates were less than some QS threshold (we used 3 or 4 at this regional) and the opponents had at least two bots above the threshold

So how did it work out?

• The DifficultyScores for *our own team's* matches were really useful for deciding how to allot pit resources. We spent a lot of Day 2 helping repair future alliance mates for matches we expected to be really difficult.
• The QualsDifficulty score (sum of the match DifficultyScores) was really useful during scouting to decide if a team was getting a "free ride", or a "rough ride". This helped us prepare our selection list.
• For the most part, the QualsDifficulty score was pretty even amongst teams after all matches in our regional (the linear metric anyway - good job NYFirst!). *But* the running totals or the DifficultyScores swung a lot along the way. A number of teams got all their "easy" or "hard" matches on the morning of Day 3. We were able to predict, to some extent, how the seed ranks would swing on Day 3 before Alliance Selection. This helped us make scouting decisions on the night of Day 2, before all quals have been played.

I think a lot of other teams would be interested in having kind of metric during regionals. Unfortunately I'm not much of a coder - my spreadsheet was hand entered by copy-pasting from TBA and drag-and-dropping columns and formulas as necessary. If this were to get widespread use, it would need to be automated, preferably in real time on a server somewhere (TBA?). Is anyone out there willing to take this on?

Secondly, the formula(s) for the DifficultyScore of each match is up for debate. Perhaps the CD community has a different idea? I know there are Apps like FRC Spyder that make match predictions using the "OPR" of each team on the field, but I think I'm more comfortable with QS (or rank) - it seems more game-independant, even if its a bit circular to use current rank to predict future rank. Also our metric is orthoganal to the stats of the TOI - we just want to know about their alliance mates & opponents.

Lastly, I'm not in love with the DifficultyScore name. We could use (checks thesaurus): Handicap, Disparity, Equality.... or yet another acronym Any other suggestions?

[nuclearnerd]

Huh... nothing? I guess it's not as widely interesting an idea as I thought. We'll keep using it ourselves though!

[Steven Smith]

I'm by no means an expert scout, and don't take this opinion as a final yes/no on the usefulness...

The DifficultyScores for *our own team's* matches were really useful for deciding how to allot pit resources. We spent a lot of Day 2 helping repair future alliance mates for matches we expected to be really difficult.

I think this metric is less useful for me to use in this way, because I (and I assume many teams) can already generate this via intuition. I typically look at other teams at a event as "top tier", "average", and "bottom tier". If I'm up against two historically top tier teams with two rookies working on their kit bots, I know the match will be difficult. If I'm against three mid tier teams, I might do a little poking around to make sure once isn't a hidden top tier for the event, etc. So generally, between pre-scouting and an hour of walking the pits and just looking around, I can (and do) designate which of the matches will be more or less difficult for me.

The QualsDifficulty score (sum of the match DifficultyScores) was really useful during scouting to decide if a team was getting a "free ride", or a "rough ride". This helped us prepare our selection list.

Without rigorously working through this, my confusion is how you can use qualification seed alone, to generate a metric that tells you if someone is artificially seeded too high (high seed, but low difficulty matches). This "weak team" is going to play out in your calculations as a "difficult team" because they are seeded high. I think the reason a lot of your data will converge to say the field was balanced is because you have a single input (QS of each team) going through a level of abstraction, that generates an output (difficulty score). I'm just stretching a bit here, but that is my gut feel?

Personally, I feel like the standard metrics taken on a team to measure their individual contribution to a match, provide a more clear picture anyway that current seeding. I'm more interested in whether the team is making auton points (consistently), making their shots, not causing fouls, etc. A goal of scouting is to generate a list of who are really the best robots at the competition, which may vary widely from the official seeding. From there, you can generate your picklist which should match your first list, with perhaps skipping robots that do not compliment your strategy well.

Kudos for trying to think outside of the box, and hopefully you can get some more discussion both for/against your idea, but it just didn't jump out at me as something I'd be interested in.

[jlmcmchl]

Maybe a more effective would be the relative ranks of a team's OPR scores (fouls and assists removed), to their alliance partners. For example:

Team x: 12 aOPR
Team y: 20 aOPR
Team z: 16 aOPR

Team x would get 3 points, team y would get 1 point and team z would get 2 points on this alliance. The higher this value is, the more the team got carried. The lower, the more they earned their rank.

My rationale behind removing assist points is because the higher the assist points, the more cooperative the team is. This is not reflective of their robot's individual skills and abilities, but is directly affected by their partner's abilities.

Unfortunately not all events have twitter data, so this isn't always the most effective. I'll get back to this later tonight with some results.

[nuclearnerd]

Originally Posted by Steven Smith

my confusion is how you can use qualification seed alone, to generate a metric that tells you if someone is artificially seeded too high (high seed, but low difficulty matches). This "weak team" is going to play out in your calculations as a "difficult team" because they are seeded high.

You're right, it is a bit circular to use past QS to evaluate past QS (it's slightly more valid to use it to predict future QS). My intuition is there is still some signal in the analysis, and that as the regional progresses, the results will converge to something reliable. That said, another score could be used just as well. I haven't taken the time to do an analysis (we just came up with the idea during the pick list meeting on Saturday night).