I’ve seen my fair share of scouting apps, forms, and scouts and this year 1817 embarked on their most ambitious scouting endeavor yet. We scout every match and in those matches, we scout every robot’s individual performance. We then compile that data into a csv and run it through a few scripts to look at several things. We were originally going to have a script generate a picklist for us based on that data but ran into some problems with it. Our head scout then forms strategies based on the teams’ performances and eventually forms our official picklist.
My question is; What do you all do with all the data you collect throughout the competition(s)?
We have a printer in our pit to print off our data for alliance selection and our drive team. I am our lead strategy person, I go into the regional with things that I want from a team for an alliance.
I print off multiple copies of our data sorted by metric, and simply cross them off during alliance selection. While I haven’t had to pick a team yet, I would pick a team that has a good scale bot, then I’d look on my sheet for the highest ranking robot that does exchange or switch well. Something like that.
We worked on getting this on a 4G-LTE connected iPad, so I can upload our data onto the iPad and our drive team can go in the pits and strategize. While I am still working on this, I actually got a 70% match prediction accuracy (granted the complete data set for a whole regional).
Really, it just comes down to alliance selection and strategy meetings with drive teams before matches. If I am talking with two other teams before our next match, we could see how many cubes we as an alliance are averaging vs how many the other alliance is averaging. Then we can adjust our strategy to match.
We print off scouting sheets (no electronic scouting for us yet) and scout every robot in every match individually. We then enter all of that data into a spreadsheet, talk about things alliance partners need to do, and what we want them to do. Then we make lists of team in order of how good they are at each game piece. This year we had three lists; Scale, Switch, Vault. Then you prioritize them down based on auto and face the boss. That way our representative can decide what we need based on what robots we already have on our alliance.
We scout every robot in every matching and compile it into a csv, and we load this into Tableau. From there we have two main outputs. The first is a match breakdown packet that includes everything we want to know about both our own alliance and the opposing alliance for our next match. This is output as a pdf and given to our drive team to build strategy for that match. The second output is a general pick list which is further refined based on feedback from our scout team. A benefit of Tableau is you can sort by a calculated metric, and have the picked robots be a filter, so we can just filter out teams as they are picked.
I would recommend using Tableau. One interesting benefit is what I call data discovery. We don’t always know exactly how we would like to view the data, and in the process of building visualizations we end up finding an interesting way to view the data and we discover trends or other interesting things that we were not initially looking for.
Generally, scouting data is used for a few basic things; match planning, picklisting, and having data to back up everything you’re trying to say to higher ranked teams.
Match planning with scouting data is super helpful, because it allows the drive team to know what a team is really capable of (despite what they say they’re capable of in most cases). It makes organizing an alliance easier, because you could see that despite a team wanting to do something they’re solid at, it could prove more advantageous to play them in a different role and have your 3rd alliance partner take on that. (Something like, your robot is worth 7 cubes on the scale or 5 on the switch, your alliance partner does 5 on the switch but 9 in the vault, and the third does 5 in the vault or 3 on the switch would give you exact data on where to play each robot to have a better shot at winning.)
By the end of the first day of competition, the scouting data should give you enough information to compile a picklist and a watchlist. Both teams I’ve been involved with for strategy have a meeting on the night of day 1 of the competition to discuss what’s going to happen to the upper level seeds the next day, and where our team could potentially end up. From there, we decide who to watch the next day, or who to talk to to try and get picked.
On top of all that, having scouting data gives you the unique position of knowing exactly where you fall at the event. Knowing that you are the “Second best switch robot at the event” can be a truly powerful statement when talking to upper ranked teams. In addition to that, you have a very good idea of what the best teams are outputting, which can give a solid goal to your drivers for upcoming matches.
Second trying Tableau at minimum for scouting data in tabular format. One thing is with match data you can have it summarize the data. And you could summarize as the average, max, or some other way.
Once you make some simple scatter charts, you can add filters to make things interesting. You can mess with color, sizes, etc to get more on one chart. It is drag and drop, so it is easy. I make sure to add team numbers as labels.
For the charts, I like to put auto scale vs teleop scale, or things like that. You can really pretty quickly see the top teams and other good info. I cross validate with scouts notes and observations and you can quickly make a pick list.
I’m sure you could add model parameters and generate a pick list using a multiparameter model if so inclined. The main multiparameter method I used was kmeans clustering, but it was for grouping teams by ability rather than ranking them.
Another thing to do is join other data, like rank with the match data. That can easily be done and might help to know the current RP and ranking situation while looking at the charts.
We have workshop presentations, and a couple scouting system whitepapers posted at www.citruscircuits.org. We developed some additional comparative visualization tools this year that we picked up from 118. Our scouting app has visualization of match by match data on our phone viewer app. We use that for match strategy. We download the data in a spreadsheet template for use on draft night.
Looks like I’ll have some good reading for the trip to Houston!
Thank you all for your awesome responses. I’m glad to see that we are on the right track with what we are doing but it is obvious we have much to learn and test. Have any teams experimented with automated picklists based purely on data?
We have first pick and second pick equations that take in scouted datapoints and spit out a number. The higher this number, the better they are ranked on our first pick or second pick lists. After every event, we adjust the equation’s weights to find what works best. These equations are all detailed in the whitepapers Richard mentioned above.
However, we don’t purely use these automatically calculated lists. At our draft night meetings, we consider other robot factors that may not be clearly displayed in the data. The calculated lists provide a starting point for our discussions, but are not the final product.
Since then, we’ve gotten better about having one of the lead scouts meet with the drive coach in preparation for each match, and sometimes join in the strategy discussions. We also use it to identify potential alliance partners and make sure we’re on each other’s radar.
We scout several key metrics (those that win the game) and take notes on our partners and foes in every game leading up to when we are on the same field. Helps with in game strategy. Take same notes on notable performers and/or likely captains. After round 1, we know them from past performance and more rounds then we have better knowledge on them.
Verify with stats the first night to make sure our top 28 list is solid and look for any outliers , sometimes add 1 or 2 that for some reason the stats say we should evaluate them too.
Then second day we watch the above plus a potential captain list of top 20.
Lots of pen, highlights, notes and excel… the scouts know their bots and share that with the drive team. We cover solidly 2/3 of the field with detailed notes the other third don’t usually don’t contribute in eliminations in regionals as they never hit our scouting radar.
Problem with posted data and data in general… too small of sample size, does not tell the story or tendencies in a detailed enough fashion as much of it based on alliance metrics and not per bot. Also data overload means potentially less reliable insights. We only have one bot per scout per match (on our watch list and a sheet for every watched bot for their portfolio of performance) we choose not to track every bot, just the ones we might face or may partner with. We had 8 games in SD …seriously where is the data? 12 games is more like it for data or rank to really be solid.
Notes/metrics from previous events in same season events flows forward for teams we face later. We scout by video too.
Things we don’t care about: General pit scouting/rank
Things we do care about : Who helps us form a potentially strong enough elimination allince and how to beat the others we may face in other elimination alliances.
We go though this , regardless of where we ourselves rank . This process helps us pick or help any alliance we are selected by in eliminations. One part to a successful season: Scouting/driving/engineering/strategy.
My team uses an app that records the data of a match that’s easy to use once you set it up. We do have a version up in the Google Play store if anyone would like to take a look. You can PM me with any questions on how to use it, the set-up is a bit confusing
But for me personally when I don’t have tech to aid me I use a basic principal when looking at a team: What are their weaknesses and what can we do to prevent them from using their strengths?
This is a system that one of my former mentors taught me. Just watch them, see what they do well and if there are any glaring weaknesses. We can strategize around preventing them from using their strengths and exploiting any weaknesses. It’s also not that hard to simply write down what they did in a match, and that will make it very clear what they can do well. Weaknesses are generally things you observe overtime, since these can be fixed in between matches.
When doing it this way, me and the other Scouts compare notes and make a strategy for the matches against those teams based on that data. With the app it’s basically the same just with overtime data.
4103 takes the data we collect and head to a quiet place (ex. cafeteria where we eat lunch) and discuss alliance selection and strategy. This is typically after our last match or at least 5 matches before the end. It is normally strategy leads, 1-2 mentors, and occasionally drive team if they aren’t busy. We’re trying to get more input from our drive team since they work/don’t work with specific teams and we need to know that.
1775 scouts each individual robot in every match then inputs it into a master excel file. In this excel file, summation and ranking pages are created for every category of play (i.e. Cubes on Scale, Climb, Auto Cube Placement, etc.).
Then on Friday night, the whole team gets together and decides where we are and what we do well. With data we look for teams higher than us to pitch ourselves to if we do something they do not. Then we also look at teams below us that do things we can’t to create a pick list.
Saturday we scout again but don’t input, but compare the new data to what we have. If the data has changed, then we add adjust our pick rankings depending on if a team has improved or gotten worse on Saturday.
PWNAGE 2451 has started a whole new scouting program this year. Our focus was capturing the data and simplifying the analytics. After 3 regionals, I think we have reached our goal.
We use the App ROBLU. It is currently available for the Andriod operating system. We have a student that has developed a WEB interface for ROBLU that we will be testing this weekend and hoping to deploy for Detroit Champs.
We scout every match and team during quals. Our scouts with iPhones use paper and Driods use the app. The Droid users then enter the paper data between matches. We also have a group that just collects qualitative data about each match.
Our ROBLU data is hosted on our own Amazin Web Server. This allows us to export the data on the fly to a TABLEAU database. We deploy a TABLEAU server so the drive team and match strategy team have live data during the event.
Our match strategy team meets with each alliance partner before our next match to plan the match gameplan. They use the live data to evaluate our teammate’s abilities and the likey strategy of our upcoming match opposition.
Alliance selection lists are created using the Tableau reporting on Friday evening. This season teams are categorized by ability and efficiency. Once the lists are complete, we evaluate the play of the teams on the list and make manual adjustments during the last quals matches on Saturday. We have shortened the scouting meeting from a 5-hour meeting consisting of “What ifs”, “I think, I feel, I sense”, to a 1.5 to 2 hour, “I know and I observed” meeting. (I also use a Blah, Blah button when the rabbit trail takes off. Focus is important)
We take the data and input into our server via cellular data on a phone. This is compiled with data pulled from the Blue Alliance, along with data we gathered at other events by going through them meticulously via video.
At the end of the first night we go through and make our picklist based on what we would need in alliance partners, who compliments us not necessarily the best robots, but the robots that would work best with us.
We assign weights to individual metrics - and we differ them from the first pick to the second pick.
Then we clean up both lists so there are no duplicates and merge them. Then we watch video on bots we question. Any more questions are left to our follow up watchers the next day prior to elims.
For instance - you may want a scale bot as your first pick. For your second, you may want exchange / switch. While a fast scale bot may do well as a second pick, a superfast exchange / switch bot may be even better. So your criteria change between first and second pick. Or, if there aren’t any GOOD scale bots left (let’s say you’re in the 6/7/8 alliance) you may go with the two switch strategy and abandon scale bots all together.
Good scouting goes along with understanding what you need and what will help you the most, not just who the ‘best’ robot is.