Hello everyone!
My team is planning to use this unusual free time to redesign its scouting system. We have a lot of questions unanswered yet and we wanted to get a feel of what makes a good scouting system.
Does your team have a scouting system?
Is there a reason not to have a scouting system?
If you have a scouting system, what platform (android tablets, paper) do you use and why?
How do you manage your data?
How do you manage pit scouting?
Do you think scouting with the students’ phones is a viable idea?
What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
Thank you very much if you answered the questions, we will come back with the final result before the season and share it with the community.
Ill fight this. Short of not driving during any match (you got bigger problems either way at this point), having at least some data going into every qualification match can be a big help, even if youre only there to do defense. Having info on who succumbs easily to defense and who scores well means knowing who to defend to deny the most points, which can squeak out a win.
Arguably more viable than buying an maintaining a set of phones/tablets, if money is an issue.
Each paper form is kept inside of a sheet protector, so that way we can reuse them from match to match. The scouts write using whiteboard markers, and once they’re done somebody enters the data into Excel, which does the data analysis/management.
Quick note if you do end up doing something similar: for match scouting, we have 12 forms - that way we can hand out 6 clean sheets to the scouts after they turn in the other 6, which gives the data entry person more time to enter in the data.
As I recall, 3674 (the Cloverbots) had some scouting courses on their website - 1678 (Citrus Circuits) and 3663 (CPR) also have some cool resources on their respective sites. If you want information on how to utilize Tableau for scouting purposes, there’s also a neat series of videos on Youtube by a member of 1983 (Skunkworks).
@MrForbes is right, there are definitely a lot of threads out there that talk about scouting. We can just add this thread to that list
1678, imo, is the gold standard. They release a whitepaper on their system each year, I would recommend you read all of them. You can find them all here.
@Katie_UPS and @Brian_Maher have some really insightful threads on how to improve data collection and how to approach alliance selections, which you can find below.
Past that, I can answer your questions on specifically how we approach it.
Does your team have a scouting system?
Yes!
Is there a reason not to have a scouting system?
No! Even if you have an incredibly small team, I think it is worth it to at least have a notebook with some qualitative notes to help aid in strategy and picking.
If you have a scouting system, what platform (android tablets, paper) do you use and why?
We use a web based system. This allows our scouts to access the website, whether that be on a phone, tablet, or computer. It allows us not to be locked to one system and allows for real time information flow between the scouts and people in the pits.
We build this system custom each year. The stack is traditionally HTML / CSS / JS / PHP / MySQL.
How do you manage your data?
We store all our data in MySQL tables on our web server. We have also run local servers as well as looked into Firebase to store data in the past.
How do you manage pit scouting?
We only ask gather what we deem important and is hard to ascertain from watching matches. This generally includes a picture, robot weight (in years that we had cheesecake), programming language (in years that moving in auto was worth points), and other information we can passively gather to help identify how ready a team might be for eliminations.
Do you think scouting with the students’ phones is a viable idea?
Yes! That is what we currently do.
What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
We are happy with our system, but we would like more checking of our data to make sure it is accurate. We plan on doing this with TBA match information, but have never gotten around to programming this.
Thanks Saikiran for the plug. A lot of my thoughts are outlined in the posts he linked, but I’ll take a stab at these questions in particular. Answers on behalf of both 2791 and 333 unless otherwise specified, both teams use similar systems.
Does your team have a scouting system?
Yes
Is there a reason not to have a scouting system?
N/A
If you have a scouting system, what platform (android tablets, paper) do you use and why?
Paper worksheets that are entered into a Google Sheets spreadsheet. I really like this system because it is simple to create and maintain, easy for scouts to use, and incredibly fail safe (even if you lose the spreadsheet data, you can still lean on paper sheets). While the data entry step may be seen by some as unnecessary labor compared to an electronic system, I strongly prefer having another set of eyes sanity checking each data point before entering it into our system.
I think a paper system with thoughtfully chosen metrics and thorough training for scouts before the event are much more important steps for collecting useful scouting data than putting together an electronic system (and you need to do those things to have a good electronic system too). I also would not be comfortable with an electronic system unless it was very thoroughly tested, which is a time consuming endeavor.
How do you manage your data?
The aforementioned Google Sheets spreadsheet, which is also home to all our analytics and visualization tools.
How do you manage pit scouting?
A Google Form that populates directly to a page in the spreadsheet. We can then easily reference this data elsewhere in the spreadsheet.
Do you think scouting with the students’ phones is a viable idea?
For pit scouting, yes, we do it. For matches, it is viable I suppose, but I’m not a big fan of it for a couple reasons:
I don’t want to have to worry about whether critical devices are charged or not, especially when the screen is going to be used pretty heavily which eats battery
Google Forms are terrible for entering data during a match. Whatever interface you choose to use needs to be easily to fill out quickly and accurately, which usually leads teams to building a custom app, which I don’t think is worth it for 95% of teams. An app is only really helpful if you want to collect detailed realtime timing or location data that is impossible to collect on paper, but I think this data is going to be useless (compared to simpler metrics) or inaccurate for anyone who isn’t an elite team with an elite scouting culture in a highly competitive situation.
What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
333: The team had a massive uptick in freshman this year (yay, this a good thing!), but many of these students don’t yet have an appreciation for the importance of scouting since they have yet to go to a competition. I’m a bit worried about the impact on scouting culture on the team next year if we can’t make it to any events this year since >50% of the team will have never been to an event, but there are certainly bigger COVID-19 related issues to worry about.
2791: the team was going to attempt an 18-person data collection operation this year. While I have a ton of confidence in @Connor_H’s ability to execute it, I was a bit concerned about issues cropping up at the event since it is such a massive new undertaking. I do not recommend this approach to most teams, 2791 is in a rare position of having a ton of students (>100) and a lot of students who are willing and eager to help with scouting. Frankly, I’m not sure if I recommend it to any.
I’d be happy to answer any follow up questions or elaborate on any of these points.
Dang, I was gonna post here but it seems Brian beat me to it.
Just to build off of our 18 person scouting system, we had a ton of failsafes built in. If we were to see any cracks in any part of the collection flow, we were going to default to a 6 person method comparable to what Brian described above. I’m going to have a topic up in the near future with all of our documentation and files we had planned to use for those interested. I wouldn’t suggest attempting this kind of system unless you have a lot of excessive manpower, and a solid scouting culture on your team. We had around 14 students in the scouting/strategy subteam and 48 trained scouts this year and we agreed it would be a fun undertaking.
Is there a reason not to have a scouting system?
Teams I’ve been on have not scouted at events like off-seasons. Also, a team only “needs” as much of a system as they have the man-power for. A system could be someone in the stands with a notebook as mentioned earlier.
If you have a scouting system, what platform (android tablets, paper) do you use and why?
I like paper because it doesn’t have a data limit or a battery but it comes with its own limitations. 1296 tried tablets and QR codes and from what I’ve heard, it was a huge headache so I avoid anything that hasn’t been proven prior to the event. 253 uses phones + cell phone data
How do you manage your data?
In a pinch, a binder + notecards is the easiest low barrier method. I assume your team is going for something a little fancier.
1296 used SPAM’s poor man’s scouting database which is a resource I can’t recommend enough. It runs in Excel and is easy to use out-of-the-box and not terribly difficult to modify once you figure out how excel macros work.
In 2019, 253 was using a google form + google sheets. Easy to set up and free. Learning query + javascript means it can be modified to kingdom come… the biggest limitation here is google forms is not an ideal input method.
In 2020, 253 was working with other bay area teams to make a website scouting system that was looking really promising but then COVID happened - I imagine it ran very similar to what Saikiran described.
How do you manage pit scouting?
Typically very poorly. I’ve never done a good job of getting that information into a useable format other than making sure a picture of every robot makes it into a shared location. In high school, I used pit scouting as a learning tool for younger team members. 253 student leadership does something similar - each subteam does its own pit-scouting related to subteam stuff.
Do you think scouting with the students’ phones is a viable idea?
I’m not a huge fan of it, however I know that text data being sent back and forth is not what chews through data limits (ahem, youtube). Every team knows their situation best to make that call - but there are some who note that not every event has good reception AND wireless noise is not good for robots playing on time.
What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
Visualization and bad data. Bad data is fixable with things like checksums and redundancy, but often those are great ideas that there is never the time/manpower to implement.
Finding good ways to make data useable and human-readable is a constant struggle - I could write a whole post on this alone but to summarize: averages are a good start but a terrible snapshot of a team’s potential/performance, graphs are great but hard to compare across multiple teams, capturing what a team could do vs what they did is probably my biggest weakness/blindspot
Our team does have a scouting system. I think teams should have a scouting system, because it can impact strategy and how well you do at competition.
We use paper with six students (one per robot), and then one of our mentors enters the data into a Google Sheet. It’s efficient because all students have to do is write what they see, and the mentor has plenty of time to enter data afterwards and during the next match. Before we go to the competition, the scouting team sits down and makes a scouting sheet so everyone is looking for the same information. After the first day of scouting, strategy gets together, makes sense of the data, and makes a rough priority/pick list.
We prefer not to use students’ phones. Some students might not have phones, batteries might die, and we find that students focus better if they don’t take their phones out at all.
There are always going to be small mistakes in data, such as one too many tallies or they circle the wrong thing. They shouldn’t generally be disastrous data-ruining mistakes, though.
We have scouted with teams in the past and shared data with teams that didn’t have the resources to help out. I think it is important for students to feel ownership over their work and data, and helping other teams out can help with this.
If another team asks for our data we’re more than happy to share it. Another team’s data might also be a form of data validation to check consistency in your own team’s scouting depending on how much you trust the other team’s scouting ability. We gave a few of our scouts to 1533 at DCMP last year which meant we also had data from their scouting system and I was happy to see mostly similar stats recorded between both of our teams systems.
Does your team have a scouting system?
Yes…every year since our rookie year
Is there a reason not to have a scouting system?
Hmm… like most things its best to do what you need done right.
If you have a scouting system, what platform (android tablets, paper) do you use and why? Paper, pens, highlighters and Excell watchlists
How do you manage your data?
We continue to build a partner list and in doing so find the teams that work best if we were to play with them. The data is to that goal, also finding ways to mitigate the threats to success of those we face.
How do you manage pit scouting?
As necessary, pits never tell the whole story. If something is up that we are unsure of, we will send scouts to find out what’s up in a pit visit. We try not to be surprised.
Do you think scouting with the students’ phones is a viable idea?
Many have tried over the years and all have failed… the phone is too limiting IMO
What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
If I had a magic wand, it would be an app with easy accurate input that tracks more accurately the items that matter to us. For now, to get all the detail paper and pen works. I tend to promote game instances so later anyone can look at the notes and see how active a team was and how long it took and how it happened.
In week 2 competition this season we had an app ready to go …by the second day everyone was using paper. Hard to change what works for …new
