What is everyone thinking as far as how to successfully paper scout this year
In my years in first we worked a lot with paper scouting and with applications.
From my experience I think that working with papers is a bit easier and more fun for scouters while filling the scouting form in most cases and have less chance to fail but when you want to analyze the data, using papers is much harder and takes much more time.
With a good app you can overcome the cons of an app comparing to using papers and getting the advantages of using an application so this is what I recommend.
Btw we are using websites in the last few years so we dont have to worry about students using android or IOS.
Talking specific about this year, I think using papers is better than in most years because we have less data to analyze, but will still recommend a good application.
The biggest thing about successful paper scouting is quickly and accurately getting that data transferred to a digital platform. That is usually where paper falls apart past the normal scouting pitfalls. Make sure you have someone trained and ready for this if you want to use your data in any timely manner, and make sure the scouts understand how your paper sheets work along with giving them some time to practice, if possible.
7 Likes
You can paper scout effectively with highly experienced students and mentors who collect qualitative observations, but for quantitative scouting that involves a broader base of students, paper is not effective unless you have a highly regimented paper to digital transfer system (e.g…, 971 has such a system). It is much easier to create an effective system in Google Forms that students can use on smart phones. We have done this several times for off season events when we don’t want to get our more complex electronic system up and running. Reach out to me if you want me to connect you with the students who have created this simpler system.
1 Like
I’d be extremely weary of endorsing Gforms for use in scouting, namely due to the fact that an active internet connection is needed. We tried Gforms in 2019, and our first event the HS had no cell service or wifi, completely crippling our scouting system.
Be sure that you can have an active internet connection (without breaking the rules, aka no hotspots) and that all devices can connect to the internet if you choose this route.
6 Likes
Yes that can be problematic. We even had such a problem with Bluetooth in the arenas that we went to QR scans to transmit data in our standard system. However, compared to the technical and organizational complexity of paper systems, if you aren’t able to make the leap up to a robust electronic system, GForms is the next best option that we’ve used successfully in urban areas. (I suspect much of your problem was from spotty cell coverage in more rural areas.)
1 Like
Or just spotty coverage in the concrete monoliths that are stadiums…
4 Likes
Paper scouting can easily track what you need to know, you have to ask yourself "What do you need to know “besides Blue Alliance aggregate tracking” per alliance scoring.
Once again BA tracks per alliance scores (diluted fidelity), so where Paper Scouting comes in is higher fidelity scouting.
It also requires training and many scouts that are NOT overloaded in tracking what you want to track (as a team). As in concentrate on ONE bot per match per scout to get better data.
So, what do you want to track? Then Paper scout that. I like to determine a shorthand method that can replay any game a team has that I want to track. I use the clock and seconds left, I use Field position, I use cycle times and cycle repeats, I use teams that give that team problems etc. The advantage of paper scouting is fidelity to be confirmed by BA stats. Hope that helps. No surprises is my goal when scouting mentor.
With say 35-50 teams in an event, it is possible to high fidelity scout those you will face including eliminations. Maybe in the Championships I would rely more heavily on apps to track it all.
Apps require cognitive load and are subject to bad data entries and being abandoned . Paper is fluid and a circle, highlight or X plus arrows can tell the story. In the end you want to find parners and know how to defend those that are not partners.
2 Likes
Any scouting system, especially a paper scouting one, should leverage the data FMS/TBA provides. It’s likely you’ll be able to get auto movement and climbing data there. ScoutLess looks like a good option.
What issues did you have with Bluetooth?
1 Like
My favorite scouting system for any year is paper scouting sheets that are entered into Google Sheets (or Excel) after the match for analysis+visualization. Combining the two, you get the best of both worlds: a reliable system that is easy to use for both data collection and analysis, as well as being quick to implement.
As for what data points to collect this year, I think 90%+ of teams would be perfectly fine collecting just counts of balls scored low+high in auto+teleop, missed shots and whether different types of climbs were attempted. You can probably grab climb successes from TBA, and you can encourage scouts to write down more open ended notes about things like shooting locations.
Like with robots, I think it’s almost always better to do the most important things reliably than to try to do everything and end up with something you can’t depend on. Regardless of what additional data I collect I always end up spending 95% of my time looking at # of game pieces scored and endgame success rate.
9 Likes
Radio band interference. Between 2013 and 2018 the amount of Bluetooth bandwidth used casually by smart phones seemed to increase immensely.
Our experience in 2012 with paper scouting and the switch to electronic in 2013 was that the cognitive load and poor data entries declined markedly. When we use simple Google Form scouting in the off season, this gets even easier. Note that is it impossible to get individual bot scoring, especially by component, from TBA.
The paper systems I have seen well executed are incredibly labor intensive and require significant mentor involvement. The electronic systems largely can be entirely student operated, a critical criterion from out standpoint for focusing on student participation.
Can you elaborate on what aspects of running a paper scouting system you see as requiring more mentor support than an electronic one? In my experience (which admittedly is limited on the electronic side), aside from requiring one more person at the competition to do data entry, I can’t think of any aspect of paper scouting that would be more logistically challenging to run than an electronic system.
1 Like
I utilized paper scouting throughout the entirety of my FIRST career as a lead scout. There are some drawbacks when compared to app-based scouting, but if done correctly it can be equally as powerful and can have more benefits. One such benefit I found the most useful was the ability to directly see the data as it is entered into the database, which allows for bad data to be taken care of. It is difficult to deal with incorrect data with an app, as the data entry is automatic. In my opinion, it is better to have a smaller volume of completely correct data than a larger volume of data that includes errors. Paper scouting allowed me to throw out or fix the data that was obviously incorrect – remember, people make mistakes and their eyes can fool them.
However, the key thing that you need for paper scouting is someone who is capable of inputting data in a short span of time (6 teams in a ~7 minute match cycle). I lagged behind at my first events as a lead scout, and I had immense backlogs that took a long while after the event day had finished to input. Eventually, with enough practice I became quick enough to where I could usually scout the match itself and input the other scouts’ sheets during field reset. Even still, extraneous circumstances can affect this, such as at PNW DCMP 2019, which many of you will remember as having some of the worst stands at any event period. There was simply no room for me to input data in the stands, so I had to stay up in a hotel lobby until the wee hours of the morning inputting. An app-based scouting system would have avoided this, but like I said prior there are pros and cons to both. Apps take time to develop and may require a basic programming knowledge (which I did not have at the time). Many of them also require internet access, especially methods such as Google Sheets. Meanwhile, paper scouting requires you to create an efficient sheet as well as a good spreadsheet to input on to. I would recommend talking to those that will be scouting with you to get their opinion on which method they prefer, and compare that to your own personal abilities.
As for this season specifically, I am not going to be too active, so I don’t have any specific recommendations for this year’s game. Good luck this season!
I’ve been able to get paper sheets recorded before the next match started (idk 90% of the time?) but the key requirements were:
- sheet and the data entry are in the same order
- most of your fields are numbers/single character
- Two people do data entry - one to read and one to type.
If you have two people who are used to doing it/working together, you can get into a really quick flow. We found that the person entering needed to be higher trained but teaching a reader was a quick “here’s how I want you to read this to me.” One could just say a string of numbers and the data entry person could quickly put it all in.
My additional plug for paper scouting is that at an off season, 1296’s database went wonky. We were able to manage for match-to-match strategy but it was unusable for picklists. We were able to sort all the scouting sheets by team and create index card stats that we would then sort.
It had an unexpected upside: we were looking for a team who had a unicorn match (we were seeded low) and it was much easier to have multiple people look through paper slips than it would have been to dig through our excel sheet. That said, if we had a publicly accessible google sheet then it would have been pretty equal (if not better) 
5 Likes
This works wonders. Sadly I never had that help, small team problem I suppose. That’s another thing about scouting, it can seem kind of boring at first and as such new members will be less drawn to doing it, leaving one with a small subteam. Effectively advertising the role as being immensely important to the team’s success remedies this problem.
I don’t see the value of transcribing or inputting the paper notes to a spreadsheet/computer program. If you measure the metrics, you want it’s easy to start ranking teams and come up with strategies to play with or against these teams.
Breaking it down in say at a big 54-team event, 1/2 of the teams will not need to be tracked unless you play with them in Quals (They will not be in eliminations). That leaves the other group that may not play with you in Quals and those with obvious positive attributes in game play. That means we need to track twenty-eight teams, twenty-four of which end up in playing in Eliminations. As the rank continues of course you track the top 16, as they could be captains too
This can easily be done without spreadsheets/Tableau, as the eyes do not deceive.
It can be as simple as you go , to have the lead scout student write down teams they see that are good with input from match scouts. That makes an enormous difference.
Data analysis is best on huge datasets, 54 is not huge. Having an ordered list of notes helps any alliance you may be on.
So you break it down into at least 6 scouts per match to do match scouting (One scout per watched robot so some scouts can relax unless all 6 are tracked) and then have a student do the high level tracking of those we think will make it to eliminations and teams that bolster what we do, to potentially form a strong alliance. I also build my list then get with the lead scout and compare notes, verify stat outliers with BA and rewatch those the final day.
This is a really good list, there are two things I’d add:
- Difficulty getting all your data entered during a match cycle is a pretty reliable indicator you’re collecting too much data. I say this from firsthand experience being the one collecting too much data.
- Doing a few dry runs of data entry to identify and address potential pain points before the competition can go a long way
5 Likes
Agree. I designed paper scouting systems that fed into a spreadsheet as a student in 2012 and 2013. I built an electronic system in 2014. The paper systems were superior.
In 2018 and 2019, I helped my students design paper systems.
Aforementioned paper scouting systems were exclusively run by students. Mentors did provide general guidance in the scouting system design (e.g., schema/what to track) but were never involved with actually collecting data.
Agree with points 1 and 2, disagree with point 3.
IMO you should design the paper and the spreadsheet so you rarely need to look at the computer. If the columns are in the same order as your sheets and you have a tactile numpad, you just need to get into a rhythm.
Is the second person for QA? I do like having spot checks for accuracy. This is where paper becomes really nice: you have an auditable record. Mistakes do happen and multiple sets of eyes help.
IMO, being able to consistently input the data during field reset (not during match play) is a scouting system design constraint.
The FRC teams I’ve been on collect too many metrics (i.e., the schema had more fields/columns than needed). See section 3 of the paper for more details.
Agree. If you believe that scouting is as important as driving, then the scouting team should practice and drill like the drive team.
2 Likes
That’s a good point - I could never quite manage it and it was annoying when switching from one paper to the next, but it is definitely doable by one person with that in mind! I would say the second set of eyes were only QAing if they were paying attention - knowing how teams tend to perform and seeing a glaring difference or something, otherwise it was just a way to keep the typist’s eyes on the screen.
As for QA, I dream that one day I will actually use the final scores (sans fouls) as a check-sum for data integrity.