2018 scouting is over (officially anyway)


#1

Okay everyone, I’d thought I would get this up here now that we are done with official play, and before everyone decides it is finally nap time.

What was the MOST difficult aspect of scouting this year?

Please discuss.


#2

Cycle times


#3

Detroit was a bit hard to get a great view of the whole field for 6 people. We made it work, but the birds-eye view of 7 rivers & CIR made things a bit easier.

We ultimately struggled to find a way to collect all the time-based data we wanted, and to make it representative of actual robot performance. End answer was to track action counts and average time to complete actions. The hope was to see who at least was capable of doing certain things quickly. Combine that with drive team feedback to see who we could work well with, and we assumed fast robots who we work well with will be able to place cubes at the right times, and lead to success.

One thing our scouts want to add for next year is a way to describe what general role the robot played in the match, so you can first sort data by “who has performed this role?” and then later by “who was best at this role?”.

Also, as usual, I tend to not associate team numbers with names with mental images of the robot. Remembering who was who was a bigger struggle for me this year.


#4

The number of different cycles of varying difficulties was hard to scout because depending on a match, a team could put 12 in the switch and 1 in the scale or 6 in the scale and 0 in the switch. Averages were skewed as a result. This is unlike past years where you could just average shots made and know who the best shooters were.


#5

um

What’s a “Scouting”?


#6

After much brainstorming debate, 1902 took a similar approach to scouting Power-Up as with previous years, mostly collecting mostly action-based data. However, in our qualitative data we had scouts record when/where the actions were taken and about how long they took; so I guess you could say that the time-based scoring increased the volume of data we collected? It worked pretty well for us but took a while to iron out.

How did y’all manage 2018 Scouting, and what lessons did you learn from it?


#7

What we were looking for in a third/fourth robot was not often demonstrated in qualification match play.


#8

+1


#9

Would you be willing to clarify that?


#10

None of the above.

The most difficult part was that teams played different roles each match, and these roles had them scoring in different goals. Ex. a solid scale robot could have to play the switch for a few matches because of strategy, which makes the easy solution of just averaging action count inaccurate.

Ultimately, my team assigned different weights to switch, scale, and exchange scoring, and used the sum of those weighted values to determine an expected contribution, which we used as a primary sorting metric in our pick list. Autonomous capability was used to prioritize teams within certain tiers, which were determined by the scouts.

Ultimately, I liked the added challenge of having to dig deeper to see what you can expect a team to do in any particular strategy. The difficulty just made it more reawarding to figure out.

edited to add below

What we wanted to show as 3rd/4th robot capability was never a winning strategy in our matches at champs.


#11

I know as a switch captain at NE DCMP, one of our primary sorts was by number of cubes in auto - we decided it was important to try to get ahead early, and teleop scale speed could be sacrificed for better scale autos.

An approach we took to get proper averages of how many cubes a robot had a possibility of doing in a match was inputting both number of cubes scored on a game element and approximate amount of time spent working on that element. This helped us understand whether a certain robot wasn’t doing an element because they had problems with it, or if they weren’t doing it due to strategy (e.g. 1 and done on the scale). We used this to get a sort of “possible number of cubes per match” statistic.


#12

Sure. Obviously some of this is specific to 254, but a lot of it applies to any team in a picking position.

Our priorities were (in rough descending order), defense/intelligence/robustness, autonomous, climb/forklift compatibility, switch play, exchange. (We knew that us + 1st pick would focus on the scale, so we didn’t really scout for that).

Defense is extremely hard to assess in qualification matches because it was hardly ever played.

Autonomous was observable, but what teams actually tried was a function of their partners and field state. “Unlucky” plate assignments, or very good partners, could mean that a robot didn’t have the opportunity to strut their stuff in auto.

Compatibility with our forklift is mainly a function of robot geometry and weight distribution, so we tended to scout it in the pits instead.

Switch and exchange play were often delegated to the weakest robot on a qualification alliance rather than the robot that was best at it (usually the best robot took the scale). There were several matches where a robot that is phenomenal on the switch instead spent the entire match struggling on the scale. This was absolutely the right thing for them to do to win the match, but it mean’t we couldn’t scout them effectively for the role they would play on our alliance.


#13

This is a topic that I would enjoy talking about in more detail but I will keep this at a high level (…for now):

2018 was a unique year for FRC in that it changed from object based scoring (ball in hole = x points) to time based scoring (scale/switch in favor accumulates points for each second). In object based scoring games, the teams who can score faster tend to score more than teams that score slower. The strategy for this year’s game boiled down to putting enough cubes on scale/switch to keep ownership. Sometimes this meant a full-blown scale war while other matches were battles of the switches depending on the combination of robots playing.

So why is this all important to answer the original question?

Scoring more cubes does not mean scoring more points. There were many matches where switches and/or scales were held by a single cube. Adding more cubes did not translate to more points if the switch/scale was in favor of your alliance. This skews data.

I would argue that the hardest part about this year’s scouting game is the identification and collection of KPIs to evaluate teams effectively. Then it is about matching up with the best partners who complement your skills and building an alliance around a strategy. Collecting the scouting data also leads beyond alliance selection and can be leveraged for match planning purposes.

Bottom line is that I think this year the biggest challenge was identifying a reasonable set of data to collect that is easy for scouts to input and that translates into useful conclusions for match planning and alliance selection purposes.


#14

We didn’t just want the average cumulative cubes scored or the median cubes placed in the scale. We wanted data to determine adaptable, reliable partners we could count on and compliment in abilities.

We didn’t realize its importance early on in the season, but for the Detroit Championship we really wanted a partner who could stack cubes on the scale really well with us. There were many robots who placed in the scale faster than us, but we really shined in our ability to score precise, tall stacks on a losing scale (and use torque to our advantage when we were matching another robot cube for cube). Fast cycles to the scale don’t matter if your cubes just fall off a second later. We didn’t want a team who would knock off our stacks. We wanted a team who could stack high and stack well with us.

We recorded cubes scored in the scale and cubes dropped. Cycles from platform zone cubes were much different than those from the portal (especially with defenders like 708), so timed data seemed too situational to be of much use. Recording scale placement ability is a bit odd, being qualitative data that you can’t directly compare like quantitative data. We recorded general notes on how many layers high a team stacked in the match and how much room they took in the null territory. We needed to make sure we could stack at the same time high up without knocking each other’s cubes off. Our more experienced scouts made sure we knew everything we needed to know beyond simple counting.

To look for adaptability and strategic competence, we didn’t look at total cubes scored in a match like many teams. We looked at cubes scored in the vault, scale, switch, and opponent switch separately. For ownership, we approximated how many more cubes an alliance needed in each switch and the scale to neutralize it (including negatives if they controlled it). This allowed us to see if teams placed cubes where they were needed and distinguish when a team only scored three cubes in the scale because they could or because they only needed that many to secure it the entire match. If a team had cubes well above what they needed to own the scale but allowed their switch to be lost, for example, we’d know they weren’t the best at switching tasks when needed.

An important thing we wanted were partners we would work well with together. Our drive team gave teams 1-5 ratings with explanations noting their experiences working with other teams. The database automatically distinguished teams with a significantly low rating that we took into consideration when picking. However, this rating system was more often used to note very positive experiences with other teams. While this isn’t always the case, every team we picked on Tesla we had played a qual match with. The best partners are those you don’t need to note you worked well with, they’re the ones you just remember. They’re the ones who make every match fun and every match a success no matter what.

We mainly collected data using a custom website that auto-populates a Google Spreadsheet with raw data as needed. The Spreadsheet consolidates this with data from pit scouting (mostly just for pictures), drive team ratings, TBA imports, and other sources. Data is useless if not interpreted and applied well. One sheet was used to look at a specific team using graphs, pictures, statistics, and other notes. We used color coded graphs of cubes scored in each of the four aforementioned locations by qual match that included markings for how many more/less cubes the alliance needed to neutralize each of the switches and scale. A graph also showed climb successes/fails. This lets us see what they did every match. You expect a team to score more in the scale if they didn’t use time to score in the switch or climb, for instance. Another sheet allowed us to compare entire alliances of three robots each for qual alliances, hypothetical playoff alliances, and actual playoff alliances. A third was a large chart of statistics ranging from means, maxes, z-scores, and proportions that we could sort teams by automatically.

We generated separate lists for “scale” and “switch/exchange” categories with a cutoff on the scale list that showed where we would take the best switch bot over the next available scale bot. Some teams were placed on both lists as seemed appropriate. Our lists (on Google Sheets) automatically filled in important info next to each team such as rank (to tell if they were available to be picked by us) and general statistics for quick adjustments if needed (especially at regionals where there are still qual matches right before alliance selections), though the original lists were made by our lead scout directly comparing graphs and other information.

We built Tesla’s alliance 3 around the idea of placing high and placing well, picking 2590 to scale with us. We knew we weren’t the fastest alliance at the scale. Our alliance couldn’t guarantee a second auto scale cube - let alone a third. We went against the popular idea of win the scale early and win the match. We knew we would start off without the scale out of autonomous, but we just took it back in teleop every time. Most alliances could take the scale early, but couldn’t win back a losing scale; we were the opposite. It worked out pretty well. We’re most proud of finals 1 on Tesla, where we (alliance 3 of 2614-2590-708-1720) owned the scale for 113 seconds in teleop while alliance 1 (2056-1241-2869-6090) owned it for 8 seconds in auto and 24 seconds in teleop. We even had the time to neutralize their switch for a while.

In our second pick we prioritized smart drivers who’d be able to adapt to the match as needed. One team stood out on Tesla from both watching their matches and playing a qual match together - 708, Hatters Robotics. We never expected they’d be available for our second pick. From protecting our switch, cube sniping our opponents, filling our vault, being an amazing defender, and doing multiple at the same time, 708 did it all. We loved working with them and were happy to be behind the glass with them again for playoffs. (:smiley: They even gave us cool hats.)

For our third pick we wanted another good scale bot, if available. Luckily, the rest of the alliances somehow didn’t pick up 1720. If needed they were ready to come in at any time to triple scale with us and 2590. This allowed for more flexible strategies if we wanted to change out teams at any time. They were great to work with in quals and did everything from consistent center switch 2 autos to quick scale scoring to double climbing.

The ideas of adaptability, reliability, and redundancy were apparent through our entire alliance. From switch autos to scale autos, we had different combinations we could utilize among our alliance when something didn’t work in a match. In teleop, both our drivers and 2590’s were comfortable with scaling from the near or far side of the field. Being able to send the robot who was already scoring on the scale from autonomous to the far side (instead of determining near / far side ahead of time like in most quals) sped up early scale cycles and prevented congestion in the null territory while we took back the scale. While we triple climbed with 2590 nearly every playoff match, we knew that we could choose either robot to double climb with 2590, we could climb side by side with 708, or we could have 1720 lift either remaining robot when needed. Triple climbing allowed the near side scale robot to use pyramid cubes usually used for levitate instead of going against defense and congestion on the far side of the field. Needless to say, our alliance had plenty of strategic options. We made the alliance we dreamed of a reality by teaming up with 2590, 708, and 1720 on Tesla. (Thanks to all of our partners who have made 2018 such a successful season for us!)

MARS will be writing up full scouting documentation over the summer and will then do a full database release during the off-season, including our custom scouting forms and specific recorded metrics, the systems’ technical details and code, and notes on our application of collected data by our strategists and drive team to all of our matches and alliance selections. Our scouts have done a truly fantastic job this season. A big thank you goes to 2393 for scouting with us at Palmetto and Smoky Mountains and 291 for scouting with us at Buckeye.

TL;DR: Collecting more qualitative data, such as stacking quality on the scale and strategic driving, and using that data effectively along with quantitative performance data to determine alliance partners and match strategies were the most difficult parts of scouting FIRST Power Up. As much as I love statistics and numbers, they aren’t everything to making successful alliances and successful strategies.


#15

Perfectly sums up my thoughts about scouting this year. +1

On the plus side, the game was really fun to watch at all levels of play, so the whole motivating scouters issue wasn’t as much of a thing this year on our team.


#16

Quality of scale cube placement.


#17

+1. Quality of scale placement, driving ability, and strategic capabilities would be the hardest part of scouting Power Up, I’d say.

We built Tesla’s alliance 3 around the idea of placing high and placing well, picking 2590 to scale with us. We went against the popular idea of win the scale early to win the match (prioritizing quality of placement over available teams with working 3 scale autos). We knew we would start off without the scale out of autonomous, but we just took it back in teleop every time. Most alliances could take the scale early, but couldn’t win back a losing scale; we were the opposite. It worked out pretty well. We’re most proud of finals 1 on Tesla.

One team stood out in driving ability on Tesla - 708, Hatters Robotics. From protecting our switch, cube sniping our opponents, filling our vault, being an amazing defender, and doing multiple at the same time, 708 did it all insanely well. Finding good driving is, in my opinion, just something you have to see for yourself. Numbers don’t tell you who can make crazy maneuvers when needed.

We measured strategic ability by comparing cubes placed in each switch and the scale with the approximate number of cubes needed to neutralize it. This allowed us to see if teams placed cubes where they were needed and distinguish when a team only scored three cubes in the scale because they could or because they only needed that many to secure it the entire match. If a team had cubes well above what they needed to own the scale but allowed their switch to be lost, for example, we’d know they weren’t the best at switching tasks when needed.


#18

+1. Quality of scale placement, driving ability, and strategic capabilities are the hardest part of scouting Power Up, I’d say.

We built Tesla’s alliance 3 around the idea of placing high and placing well, picking 2590 to scale with us. We went against the popular idea of win the scale early to win the match (prioritizing quality of placement over available teams with working 3 scale autos). We knew we would start off without the scale out of autonomous, but most alliances couldn’t win back a losing scale. Once we got it, we held it for the match. We’re most proud of finals 1 on Tesla.

One team stood out in driving ability on Tesla - 708, Hatters Robotics. From protecting our switch, cube sniping our opponents, filling our vault, being an amazing defender, and doing multiple at the same time, 708 did it all insanely well. Finding good driving is, in my opinion, just something you have to see for yourself.

We measured strategic ability by comparing cubes placed in each switch and the scale with the approximate number of cubes needed to neutralize it. This allowed us to see if teams placed cubes where they were needed and distinguish when a team only scored three cubes in the scale because they could or because they only needed that many to secure it the entire match.


#19

+1. Scale placement, good driving, and strategic decisions on actions


#20

Scouting this game, I’m sure we can all agree, was not really a matter of numbers. A bot could score many cubes on the scale, but if another robot contests them in a pushing war for defense or if they dropped too many cubes they were not a viable bot.

Performance was really what mattered this year, if a bot could perform a task effectively then they were viable. They may have been slower than another bot, but if they were smooth they were effective. Smooth when it came to the scale was important, and pushing power plus quick cycles were required for a good switch robot.