Designing a Scouting System

Correct me if I’m wrong, but the latest thread I’ve seen about scouting systems specifically is from 2015.

What do you mean when you say scouting system? Do you mean an app, or are you also including paper scouting?

Did you sort for “most recent”, or the default sort of “relevance” in your search?

Anyways, here’s a big, recent thread…enjoy!

Of course you should fight it, because every team should scout.

But in real life, there are some teams who really don’t need any scouting data…my point is, make your team not be one of those teams!

1 Like

I meant everything, paper, tablet, phone, mind! Thanks for asking!

This thread is a pretty good read for help about pit scouting.

Are there any specific areas you wanted to address when redesigning your own system? What was it previously?

1 Like

If I had to answer question 7, I would say it is definitely the data management that wasn’t perfect. How does your team do it?

We use a combination of paper and Excel.

Each paper form is kept inside of a sheet protector, so that way we can reuse them from match to match. The scouts write using whiteboard markers, and once they’re done somebody enters the data into Excel, which does the data analysis/management.

Quick note if you do end up doing something similar: for match scouting, we have 12 forms - that way we can hand out 6 clean sheets to the scouts after they turn in the other 6, which gives the data entry person more time to enter in the data.

As I recall, 3674 (the Cloverbots) had some scouting courses on their website - 1678 (Citrus Circuits) and 3663 (CPR) also have some cool resources on their respective sites. If you want information on how to utilize Tableau for scouting purposes, there’s also a neat series of videos on Youtube by a member of 1983 (Skunkworks).

1 Like

@MrForbes is right, there are definitely a lot of threads out there that talk about scouting. We can just add this thread to that list :slight_smile:

1678, imo, is the gold standard. They release a whitepaper on their system each year, I would recommend you read all of them. You can find them all here.

@Katie_UPS and @Brian_Maher have some really insightful threads on how to improve data collection and how to approach alliance selections, which you can find below.




Katie’s Your scouts hate scouting and your data is bad: Here’s why thread is incredibly important to take into account when you design a system. I would have it open at all times through any development.

Past that, I can answer your questions on specifically how we approach it.

  1. Does your team have a scouting system?

Yes!

  1. Is there a reason not to have a scouting system?

No! Even if you have an incredibly small team, I think it is worth it to at least have a notebook with some qualitative notes to help aid in strategy and picking.

  1. If you have a scouting system, what platform (android tablets, paper) do you use and why?

We use a web based system. This allows our scouts to access the website, whether that be on a phone, tablet, or computer. It allows us not to be locked to one system and allows for real time information flow between the scouts and people in the pits.

We build this system custom each year. The stack is traditionally HTML / CSS / JS / PHP / MySQL.

  1. How do you manage your data?

We store all our data in MySQL tables on our web server. We have also run local servers as well as looked into Firebase to store data in the past.

  1. How do you manage pit scouting?

We only ask gather what we deem important and is hard to ascertain from watching matches. This generally includes a picture, robot weight (in years that we had cheesecake), programming language (in years that moving in auto was worth points), and other information we can passively gather to help identify how ready a team might be for eliminations.

  1. Do you think scouting with the students’ phones is a viable idea?

Yes! That is what we currently do.

  1. What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?

We are happy with our system, but we would like more checking of our data to make sure it is accurate. We plan on doing this with TBA match information, but have never gotten around to programming this.

6 Likes

Thanks Saikiran for the plug. A lot of my thoughts are outlined in the posts he linked, but I’ll take a stab at these questions in particular. Answers on behalf of both 2791 and 333 unless otherwise specified, both teams use similar systems.

  1. Does your team have a scouting system?

Yes

  1. Is there a reason not to have a scouting system?

N/A

  1. If you have a scouting system, what platform (android tablets, paper) do you use and why?

Paper worksheets that are entered into a Google Sheets spreadsheet. I really like this system because it is simple to create and maintain, easy for scouts to use, and incredibly fail safe (even if you lose the spreadsheet data, you can still lean on paper sheets). While the data entry step may be seen by some as unnecessary labor compared to an electronic system, I strongly prefer having another set of eyes sanity checking each data point before entering it into our system.

I think a paper system with thoughtfully chosen metrics and thorough training for scouts before the event are much more important steps for collecting useful scouting data than putting together an electronic system (and you need to do those things to have a good electronic system too). I also would not be comfortable with an electronic system unless it was very thoroughly tested, which is a time consuming endeavor.

  1. How do you manage your data?

The aforementioned Google Sheets spreadsheet, which is also home to all our analytics and visualization tools.

  1. How do you manage pit scouting?

A Google Form that populates directly to a page in the spreadsheet. We can then easily reference this data elsewhere in the spreadsheet.

  1. Do you think scouting with the students’ phones is a viable idea?

For pit scouting, yes, we do it. For matches, it is viable I suppose, but I’m not a big fan of it for a couple reasons:

  • I don’t want to have to worry about whether critical devices are charged or not, especially when the screen is going to be used pretty heavily which eats battery
  • Google Forms are terrible for entering data during a match. Whatever interface you choose to use needs to be easily to fill out quickly and accurately, which usually leads teams to building a custom app, which I don’t think is worth it for 95% of teams. An app is only really helpful if you want to collect detailed realtime timing or location data that is impossible to collect on paper, but I think this data is going to be useless (compared to simpler metrics) or inaccurate for anyone who isn’t an elite team with an elite scouting culture in a highly competitive situation.
  1. What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?

333: The team had a massive uptick in freshman this year (yay, this a good thing!), but many of these students don’t yet have an appreciation for the importance of scouting since they have yet to go to a competition. I’m a bit worried about the impact on scouting culture on the team next year if we can’t make it to any events this year since >50% of the team will have never been to an event, but there are certainly bigger COVID-19 related issues to worry about.

2791: the team was going to attempt an 18-person data collection operation this year. While I have a ton of confidence in @Connor_H’s ability to execute it, I was a bit concerned about issues cropping up at the event since it is such a massive new undertaking. I do not recommend this approach to most teams, 2791 is in a rare position of having a ton of students (>100) and a lot of students who are willing and eager to help with scouting. Frankly, I’m not sure if I recommend it to any.

I’d be happy to answer any follow up questions or elaborate on any of these points.

8 Likes

Dang, I was gonna post here but it seems Brian beat me to it.

Just to build off of our 18 person scouting system, we had a ton of failsafes built in. If we were to see any cracks in any part of the collection flow, we were going to default to a 6 person method comparable to what Brian described above. I’m going to have a topic up in the near future with all of our documentation and files we had planned to use for those interested. I wouldn’t suggest attempting this kind of system unless you have a lot of excessive manpower, and a solid scouting culture on your team. We had around 14 students in the scouting/strategy subteam and 48 trained scouts this year and we agreed it would be a fun undertaking.

5 Likes
  • Does your team have a scouting system?
    Yep!

  • Is there a reason not to have a scouting system?
    Teams I’ve been on have not scouted at events like off-seasons. Also, a team only “needs” as much of a system as they have the man-power for. A system could be someone in the stands with a notebook as mentioned earlier.

  • If you have a scouting system, what platform (android tablets, paper) do you use and why?
    I like paper because it doesn’t have a data limit or a battery but it comes with its own limitations. 1296 tried tablets and QR codes and from what I’ve heard, it was a huge headache so I avoid anything that hasn’t been proven prior to the event. 253 uses phones + cell phone data

  • How do you manage your data?
    In a pinch, a binder + notecards is the easiest low barrier method. I assume your team is going for something a little fancier.
    1296 used SPAM’s poor man’s scouting database which is a resource I can’t recommend enough. It runs in Excel and is easy to use out-of-the-box and not terribly difficult to modify once you figure out how excel macros work.
    In 2019, 253 was using a google form + google sheets. Easy to set up and free. Learning query + javascript means it can be modified to kingdom come… the biggest limitation here is google forms is not an ideal input method.
    In 2020, 253 was working with other bay area teams to make a website scouting system that was looking really promising but then COVID happened - I imagine it ran very similar to what Saikiran described.

  • How do you manage pit scouting?
    Typically very poorly. I’ve never done a good job of getting that information into a useable format other than making sure a picture of every robot makes it into a shared location. In high school, I used pit scouting as a learning tool for younger team members. 253 student leadership does something similar - each subteam does its own pit-scouting related to subteam stuff.

  • Do you think scouting with the students’ phones is a viable idea?
    I’m not a huge fan of it, however I know that text data being sent back and forth is not what chews through data limits (ahem, youtube). Every team knows their situation best to make that call - but there are some who note that not every event has good reception AND wireless noise is not good for robots playing on time.

  • What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
    Visualization and bad data. Bad data is fixable with things like checksums and redundancy, but often those are great ideas that there is never the time/manpower to implement.

Finding good ways to make data useable and human-readable is a constant struggle - I could write a whole post on this alone but to summarize: averages are a good start but a terrible snapshot of a team’s potential/performance, graphs are great but hard to compare across multiple teams, capturing what a team could do vs what they did is probably my biggest weakness/blindspot

2 Likes

How do y’all feel about sharing data between teams?

3 Likes

Our team does have a scouting system. I think teams should have a scouting system, because it can impact strategy and how well you do at competition.

We use paper with six students (one per robot), and then one of our mentors enters the data into a Google Sheet. It’s efficient because all students have to do is write what they see, and the mentor has plenty of time to enter data afterwards and during the next match. Before we go to the competition, the scouting team sits down and makes a scouting sheet so everyone is looking for the same information. After the first day of scouting, strategy gets together, makes sense of the data, and makes a rough priority/pick list.

We prefer not to use students’ phones. Some students might not have phones, batteries might die, and we find that students focus better if they don’t take their phones out at all.

There are always going to be small mistakes in data, such as one too many tallies or they circle the wrong thing. They shouldn’t generally be disastrous data-ruining mistakes, though.

1 Like

We have scouted with teams in the past and shared data with teams that didn’t have the resources to help out. I think it is important for students to feel ownership over their work and data, and helping other teams out can help with this.

3 Likes

If another team asks for our data we’re more than happy to share it. Another team’s data might also be a form of data validation to check consistency in your own team’s scouting depending on how much you trust the other team’s scouting ability. We gave a few of our scouts to 1533 at DCMP last year which meant we also had data from their scouting system and I was happy to see mostly similar stats recorded between both of our teams systems.

3 Likes
  • Does your team have a scouting system?
    Yes…every year since our rookie year

  • Is there a reason not to have a scouting system?
    Hmm… like most things its best to do what you need done right.

  • If you have a scouting system, what platform (android tablets, paper) do you use and why? Paper, pens, highlighters and Excell watchlists

  • How do you manage your data?
    We continue to build a partner list and in doing so find the teams that work best if we were to play with them. The data is to that goal, also finding ways to mitigate the threats to success of those we face.

  • How do you manage pit scouting?
    As necessary, pits never tell the whole story. If something is up that we are unsure of, we will send scouts to find out what’s up in a pit visit. We try not to be surprised.

  • Do you think scouting with the students’ phones is a viable idea?
    Many have tried over the years and all have failed… the phone is too limiting IMO

  • What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?

If I had a magic wand, it would be an app with easy accurate input that tracks more accurately the items that matter to us. For now, to get all the detail paper and pen works. I tend to promote game instances so later anyone can look at the notes and see how active a team was and how long it took and how it happened.

In week 2 competition this season we had an app ready to go …by the second day everyone was using paper. Hard to change what works for …new

Here’s an example of what I mean by tracking instances:

ISR2 Final 2

1 Like

1983 Skunkworks Scouting Team. (mentor Weeks - since 2009)
1. Does your team have a scouting system?
We’ve had one since 2009 and its been modified every year as our scouting became more sophisticated. We even have a process for choosing alliance partners which now has been streamlined with statistical analysis techniques that we can apply in just over an hour (used to be 4+ hours).
2. Is there a reason not to have a scouting system?
Only when scouting is not an option … like if you end up on Einstein.
3. If you have a scouting system, what platform (android tablets, paper) do you use and why?
We use Google Nexus 7s (the team owns 8 - 6 for scouting, one for the drive team and a spare) running Kiosk (as our web browser because we get a full screen) and web apps made in HTML, CSS and Javascript. We run our server on a scout’s home Raspberry Pi in Linux with scripts running Python. Our current database is MySQL but we’ve used PostGreSQL in the past. We run PgAdmin4 to design the database. The kids (with MS systems) use Atom to design the web site (not public at the moment) and the web apps using GitHub for source control. We’ve used Tableau in the past for analysis (after all, we pioneered the use of it for First) but we’ve moved on. Our analysis is now a web app using free online packages to do more sophisticated analysis of data that Tableau is a level below. We also have a portable printer (and 2 portable power supplies) so we can print out charts if hardcopy is needed. Usually, the drive team pulls up the latest analysis (with a web app we’ve written) to get all the strategy we can provide (without ever coming to the stands or having somebody run a chart to them).
4. How do you manage your data?
Its on a database on the Raspberry Pi. Because its ours and there’s no limit to space, all the year’s data is there so we can look up any data from previous competitions if there are opponents or allies that have played before. In fact, our strategy app takes in that data. Its backed up after every competition.
5. How do you manage pit scouting?
We’ve ALWAYS had pit scouting. It doesn’t change much from year to year and lately, we’ve had the drive team do it. I often insist on pit scouting because: A. These are robot geeks and you all can share in the culture. &B. If you’ve pit scouted, and you got their names and you want to ally with them, you’ve met at least one of them and you can talk to that kid and find out if they’re willing to play defense or change their autonomous or whatever.
6. Do you think scouting with the students’ phones is a viable idea?
N/A
7. What is the biggest flaw or the thing you wished you could improve but can’t in your current scouting system?
Right off the bat, RETURN WIFI to competitions!! Our printer has WiFi. We don’t always get signal in these enclosed gyms (fortunately, our web apps cache all data). The only other thing is Statistics. I’m a high tech engineer (retired) but my statistics background is only goes so far. I’m always looking for a new metric (or better one) that provides some significance (like OPR, ELO, CCTWM, aWAR, etc.)

3 Likes

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.