Introducing Pairwise - Easy, Season Independent, Qualitative Scouting

[Skyehawk]

Greetings all, some of you may recognize me, others may not.

Before the season starts, I wanted to introduce a side project myself and other prominent members of the community have been inputting to/working on. Pairwise:

https://www.pairwisetool.com/

What is Pairwise?

There are a lot of scouting apps out there; this *kind-of* is one kind of not - think of it as more of an analysis tool. Most of these apps (if not all) are focused around two things:

• Quantitative data (i.e., number of game pieces)
• Specific to the season

This is well and good, but there are some instances where all those data (yes, “data” is plural) are vastly overkill or it is too big of an ask for a small crew to collect. Furthermore, there are metrics that are very difficult to quantify. How does one evaluate driver skill? How does one evaluate driver skill consistently from match to match? For this reason, a lot of teams avoid qualitative metrics, gut feels, and vibe checks in their data collection. This is understandable; it is difficult to analyze. Pairwise is a tool to analyze qualitative observations.

Pairwise offers a method to analyze, for the small team, for the larger team, for the rookie, for the old hand. Pairwise is suited for:

• Teams that may not have enough members to scout efficiently
• Teams that may not have time to create their own data collection methods
• Teams that rely on community statistical tools
• Teams that would like a starting point based on team preference to begin scouting meetings
• Developing team comparison/analytical skills to expedite scouting meetings
• Scouting on vibes

This is a web-based application, chosen for accessibility and compatibility. It is continually undergoing improvement and tweaks for usability.

Pairwise includes:

• A Data Input/Formatter tool (think tier maker) for an alliance (use the included one or your own)
• A File processing tool - to generate ranks and pairwise ranking scores (online or offline)

How does it work?

A user generates a bunch of pairwise comparisons (with the included tool or on their own). These are then processed, resulting in a ranking.

The backend of generating rankings works with Singular Value Decomposition (SVD), which can be illustrated with the transitive property, asserting that if a > b and b > c, then it follows that a > c. If we have a whole bunch of observations of which teams performed better in a given category (say holistically for the moment), then we can slowly build which teams are better than others via common alliance partners. Now in reality, the backend is a bit more complex, but you get the idea; we are able to build ranks based on common alliance partners (this can be extended further, say to all teams on the field or back-to-back matches, but we will not cover that now).

Is Pairwise useful to me?

The most valuable aspect of Pairwise in my eyes is not the data itself, but rather the time those scouting take after the match to discuss the good and the bad that occurred on that alliance. *Thinking critically*, the resultant list and data are a bonus at this point. I believe all teams should encourage good scouting culture by not treating folks as data input monkeys but rather as actual *scouts* and evaluating what is in front of them and if it is a good fit.

Pairwise was designed to fill a gap in scouting, analysis of qualitative results. It is relatively straightforward to access, test and use, which in theory means you could get a rookie team up and running the day before competition. There are still a lot of tweaks and optimizations to usability that have yet to be implemented (Touch support is on the way for the data input tool, I promise!).

As far as I know, no publicly accessible tool like this exists for FRC. Pairwise isn’t reliant on season-specific updates or on the use of the included data generation tools. It can be used for your custom off-season events, to encourage critical thinking in the stands, as a supplementary data collection tool, or just to look at who is best in a qualitative metric. The versatility should be there for your use-case.

Results from the file processing will look like this:

Testing

Please scroll though the Pairwise help pages for more info:

I know a lot of you are interested in testing tools, but with no data that is quite difficult.
Here is a .zip with some .csv files in it you can tweak and play with the results.

• One is a very basic set of observations for a few teams just indication who is better than who and who is equal
• Another is NHL teams (completely made up “who is better than who and by how much” not from a season)
• Finally there is data from 2023 Roboteer rumble, both in the binary “Better than or equal” and in the “Better-by-how-much” format

FRC_pairwise_test_data.zip (1.6 KB)

Edit, now fixed
Note: you need to be careful about not including any leading spaces or characters in the CSV. I will work to address this further, but if you get an invalid file error make sure your data look like this:

101,201,0

…and not like this (those spaces are sneaky):

101, 201, 0

[MyraJ]

I believe all teams should encourage good scouting culture by not treating folks as data input monkeys but rather as actual scouts and evaluating what is in front of them and if it is a good fit.

I’m getting a badge that says “chief data input monkey”
In other news super excited to try blending this with into scouting meetings to make our meetings run smoother with the data side of stuff

[Skyehawk]

Touch support should be working on the basic data input/collection tool.

This is a huge relief to me.

This means there should be no worry about having to cache anything on mobile devices - as long as there is mobile data you can refresh the page whenever and use TBA integration.

I still need to get a “copy to clipboard” button on there, but that’s not a huge deal.

Some other minor html details still need fixing up.

[Karthik]

@Skyehawk

How does it work?

A user generates a bunch of pairwise comparisons (with the included tool or on their own). These are then processed, resulting in a ranking.

The backend of generating rankings works with Singular Value Decomposition (SVD), which can be illustrated with the transitive property, asserting that if a > b and b > c, then it follows that a > c. If we have a whole bunch of observations of which teams performed better in a given category (say holistically for the moment), then we can slowly build which teams are better than others via common alliance partners. Now in reality, the backend is a bit more complex, but you get the idea; we are able to build ranks based on common alliance partners (this can be extended further, say to all teams on the field or back-to-back matches, but we will not cover that now).

This is fantastic! We played around with something like this on 1114 back in the day for VRC scouting, where getting accurate scouting data was a challenge due to the small size of the robots and low resolution cameras. Being able to build a list based on repeated answers to the question of “Who’s better X or Y”, is incredibly useful for teams who don’t have the resources to scout 6 robots per match for every match. In essence it’s an intelligent way of turning qualitative data into something quantitative.

I’m really excited to see if this ends up being a tool that teams use and seeing how well the data teams generate correlates with both empirically scouted data, as well advanced metrics like OPR and EPA.

[Skyehawk]

Thanks Karthik,

Really hoping this proves useful to folks without the person-power in the stands when used in conjunction with things like EPA from Statbotics.

By the same token I feel like it is straight forward enough (with a added “quick start” tutorial; on the to do list) that a rookie team that looks like they “got it right” can be up and running early in the event.

Excited to see what people do with it.

[Skyehawk]

Hey all,

The last useability puzzle piece to use Pairwise with your phone, a notes app, and the included data collection tool is in place.

There is touch support (see above) and now a easy copy button to get the comparisons onto your clipboard to make storing the data easier (Pairwise will not be handling individual user data storage, at least for the 2024 official season)

There will most certainly be some user interface/ user experience (UI/UX) tweaks along the way, but all the ease of use components planned for the season are in place.

More updates to come in this thread. Next up: user tutorials.

Happy ranking!

[Skyehawk]

In the works:

Some helpful tools for the user for big and small events including:

Observation trimming: removing the first n observations. You would use this for examining the last few matches a teams have completed, treating those recent matches as more representative of the current level of performance rather than matches at the beginning of the event this may also be helpful for very large events (worlds divisions). I debated on going by match number and including that info in the .csv. I may still include match number because that is a very useful piece of info to have in there for subsetting.

Weighted averages: when multiple comparisons exist between teams an option to average the comparisons similar to ELO where the last observation between the two teams accounts for 70% of the average and the previous observation (s) account for 30%. The user sets the percentages for how aggressive they want the most recent matches to contribute. this will be most useful for smaller events were you see the same team on the field a lot, especially if you include additional observations beyond alliance partners comparisons. (the FMS match scheduler prioritizes different alliance partners so this is mostly applicable to small events (game manual section 10.5.2) )

Matrix Factorization for resolving inconsistencies: teams improve over time, teams have bad matches, data input isn’t always perfect - this tool will hopefully help resolve inconsistencies or errors in the input data - this is a balancing act and will be highly experimental.

[Skyehawk]

One more quick update:

I just stress tested with a rather large file (~9K observations) of real world data (760 observations duplicated a few times, so performance may be a little better than if they were all unique, but still…)

Ran in about 2 seconds.

[Skyehawk]

Match filtering on the data input tool now works, i.e. you do not need to subset the observations yourself.

if you want to compute over matches 20-60 you can do that now with a slider.

This requires match number as the 4th column in the .csv.

[Richard_McCann]

@Skyehawk If you need coding capacity support, our software scouting subteam will have extra availability after next weekend.

[Skyehawk]

There is now support for tagging a match as unrepresentative.

This is for both the data collection tool and the processing tool.

There are a few bugs with the drag and drop still- changelog and todo here. Admittedly there mobile variant is a little clunky to toggle the flag checkbox, you have to touch and hold and sometimes it is annoying to toggle. This is just an HTML UI/UX issue and I will be fiddling with it to make it a bit more smooth. The desktop experience is very smooth.

The point of including this is to allow for direct reporting of who the best and worst teams were on an alliance (for some user defined metric) without having to add in your own bias through guessing on “what could have been”. If last year’s world champion tipped over in auto and some mid range bots outperformed the tipped bot - then the mid range bots were the best on the alliance regardless of a tipped powerhouse that was flagged as incapacitated. If the powerhouse team sorts out their tipping issues later in the event you can process the data with those comparisons removed to make the results more representative.

Moving forward I may have the ability to filter by certain flags (i.e. exclude matches where heavy defense was played). We’ll see.

… and the processing tool with the ability to exclude flagged (incapacitated) comparisons.

[Dale]

This is going to be huge for 6665, C-Otters a tiny girls-only team we are rebooting in Monterey. We don’t have enough scouts to be able to do conventional quantitive scouting but should be able to scare up two, one for red and one for blue. Using your tool at least we won’t be completely blind should we end up as an alliance captain. Thanks a million!

[Skyehawk]

I am getting together a “Basic usage from start to finish” style tutorial, it probably will not be ready today, but relatively soon. The goal is to emphasize some of the finer points that will result in a smoother overall experience for basic use, as using a new system often comes with growing pains.

EDIT: There is now some info up in tutorial format. Not complete, but a start. Available HERE.

[Skyehawk]

I am very much in support only mode right now. No new features for the rest of the competition season.

Work and school are a substantial commitment at the moment and I have other human things to do as well. This may change in the upcoming weeks, but I made a commitment to everyone not to add features in-season. Stability is important.

I will do my best to support what is here and offer advice to the handful of teams giving Pairwise a go. (I did see some activity of teams testing over the last week or three).

Pairwise isn’t going anywhere, just running into typical dev side project stuff.

Edit:

Mention my username or DM me for support if you need

[Skyehawk]

There is now support for locally saving your comparisons to your device!

(This save persists through refreshing the page).

You can play with the data input tool here.

The local data can be cleared with the button or if you happen to clear the browser cache. The only information that is saved is the comparisons themselves in human readable format. I will try to get this integrated into the file processing at a later date.

I will still play around with the UI to make it a bit more usable, and to bug test.

As an aside, I am a scientific programmer, not a front end or back end developer/engineer. If anyone wants to help me on the dev side that would be 1) appreciated and 2) a good opportunity to add to a portfolio/ do some collaborative work.

[Skyehawk]

Small quality of life UI/UX improvements:

Placed items in tiers now get a colored border. Larger more usable incapacitated checkboxes, more efficient use of space for the data input tool.