This past season FRC 238 used a Scantron style scouting system originally developed by FRC 2067 called PiScout. We have assumed development of the system after making numerous changes during the season (and now offseason) and wanted to share it with the CD community in case other teams might find it useful.
We feel like the Scantron style system provides many (though certainly not all) of the benefits of all-digital phone or tablet systems while avoiding many of the hardware requirements. All that’s required is a single scanner and laptop, which can be located in the team’s pit!
Through the use of AWS free tier, teams can even set up a remote server to host the data on the web (documentation on the wiki). Our system can be accessed at http://scout.frc238.org
Though it is fairly inexpensive for us to keep the snapshot on AWS, we would love to know if others are getting use out of it to make sure we are justifying even that minimal investment.
Thanks for sharing! I had an idea for a similar system this year but never got around to really making it or starting it. Happy to see that someone did it successfully. If you don’t mind, what did you like about it? What didn’t you? Was it easy to adjust to?
Not 238 but I used the system as drive coach/head of scouting for 2067 for 2 years. Here are some pros and cons off the top of my head from my experiences.
Pros:
No one has to do data entry
The system works both online and offline so you are not reliant on a constant internet connection at the venue (you can tether the laptop to a phone and upload in batches or you can take the scanning laptop to lunch and connect to a public wifi while you are out).
Creates a paper backup that prevents technical glitches from ruining your tournament. This past season we had several glitches with our laptop that would have been very problematic if we didn’t have paper backups.
Unlike typical paper systems you don’t need to be as organized when giving out the sheets (each match is processed individually with the team).
Because of the use of scantrons for tests in school and for standardized tests such as the SAT it is easy for many students to learn.
Scouts can write notes in the margins for themselves as the watch the match (some scouts made tallies sometimes scouts made notes to tell the drive team later).
The system is great for collaboration. You can split the forms between several teams and have it all fed into the scanner with both teams getting the data from the same website. On 2067 we used this as a tool during eliminations/playoffs (giving teams access to our scouting website) and during quals (238 originally started using the system when we scouted together at a tournament).
Cons:
It’s not quite as quick as an app for entering data. Data has to be delivered and fed into the scanner.
It’s power reliant. The system relies on both a scanner and a computer so while it isn’t quite as power reliant as an app based system you still need to figure out power (most of the time we had it in the pits but occasionally we had a small setup going in an out of the way hallway).
Uses a lot of paper with a certain quality of printing. The number of matches you can fit on a sheet depends on the data you are collecting but overall the system takes more paper than a typical
The familiar format of a Scantron format can make the system feel like a test to students so you have to find ways to keep them engaged.
If I think of more later I will probably edit this post. if you have anymore questions or want more details feel free to ask in this thread or over pm.
[/li]
If I’m understanding correctly, not having a system that’s constantly online and having to submit data in batches or during breaks also means that the data’s not instantly accessible for pre-match strategizing, which puts a damper on the entire premise for scouting.
(Nothing really against this scantron format, but regarding digital scouting in general)
Based on their documentation, it seems as though the data can be accessed on the laptop directly but can be shared online for other teams to access as well.
Team 4909 uses a similar approach for our tablet scouting program as we can see our data anytime, while we can upload to a server to share data with other teams we collected data with.
Sure, I can elaborate on this a bit! I pretty much agree with the points my brother posted above. In 2016, 238 used an alll paper, 1 team per page style system. Compared to this, we saw the following benefits when switching to PiScout:
Better access to near real-time data. While we had this with the old system, you couldn’t take the sheet with you (as it was needed in the stands for scouting). With PiScout we were able to print off the Alliance Compare page and use it for pre-match strategy
Better visibility for students of where the data goes. By having all of our data available on the web (and even more so now that it’s mobile friendly) it is easy for students to see where the scouting data goes and better understand how it’s being used.
Less coordination needed in the stands. With the type of paper system we used, you needed to collect and file 6 sheets per match and hand out 6 new sheets and it was difficult to prepare more than a few matches in advance. With the PiScout system, you can handle this more flexibly to what your team prefers. You could have clipboards full of blank sheets, hand out sheets every 3 matches that are pre-filled with team/match data, or have a clipboard with all the sheets pre-filled for the whole day. You can collect and process sheets every 3 matches, every 6, etc. (and you can change this through the event as your needs may change).
Better able to sort and visualize data for picklist creation. Our experience with picklist creation with the paper system was a bit of a mess and relied on a lot of gut feel of which couple teams to be comparing to avoid a bunch of iteration after “discovering” teams that should have been higher. It’s hard to visualize 40 pieces of paper at once.
Issues we had in 2017 (we have made some changes in the system to try to address these):
Too much time spent validating data. I didn’t realize this was an issue until after the season, but apparently our students were spending quite a bit of time validating the data as it was entered. They were checking all of the text representations of the data against the sheet in the GUI that pops up. We have since tweaked the system to highlight the detected boxes so data can be validated with a much quicker visual scan of the sheet on the screen.
Issues with Team/Match info being incorrect. In spot checks of data from the first event, this data was the most often incorrect. We improved this by having a small number of people pre-fill the team and match info (and write them out on the form as well) after the match schedule is published and also added the auto-flag if the server is on the internet (or when they are submitted to the remote server if not)
Significant limitations we have not addressed:
With the way the system is currently setup you get data every 3 matches at best. This doesn’t quite meet what you can get from full digital systems.
The type of data you can capture is more restrictive than either all digital or all paper systems. If you want to capture text comments, auto paths, shooting locations, etc. it is very difficult to do with this type of system.
Overall we feel there are significant benefits over our previous paper based system. From a pure data perspective I feel it is inferior to all digital systems, but when logistics, expenses, etc. are considered we feel this system is a good compromise.
In terms of getting used to the system, it did take a little bit to get a few students used to how the scanner worked and how the program would display while it was scanning, but ultimately there wasn’t too much trouble with the transition.
Correct the website is constantly online. The laptop that is uploading the data may not always be online depending on the event connectivity. We often used the website to share data with other teams (like 238) and when the venue had good enough signal I would use the website as drive coach for match strategy.
To reply to his batch comment. The frequency of the batches can as often as you want. Both 238 and 2067 are in the NE district so for district events the batches would be at a pretty high frequency to accommodate the fast turnaround time. For DCMP and CMP the batch frequency was far lower. If the logistics are handled correctly your data should be as up to date as if you used a non digital system (possibly more so as more data can be uploaded while you are in queue finalizing your strategy).
This is correct. All data is available from the local server (the laptop in the pit) as soon as it has been scanned. We typically print out the Alliance Compare page using the printer/scanner for upcoming matches to use for pre-match strategizing.
We run the remote server to:
Help with visibility of data from previous events for pre-scouting
Provide data to folks not at the event
Allow multiple people looking at the data for picklisting
Rankings predictions
Occasional mobile access to supplement the printed Alliance Compare
For most of these purposes, occasional syncs are sufficient. If you have constant internet access available (venue internet or tethered device), the system will happily sync after every scan and keep the remote server info completely up to date for access from mobile devices.
There is a little bit of “lag” in data availability from the fact that we have 3 matches per page on the scouting forms. Early in the event, we will try to process the forms every 3 matches as they get completed because new data is very important. As the event goes on we may consolidate into 6 or 9 match batches as new data becomes less critical (this also depends on distance to the pit, student availability, etc.). If you were willing to use the paper and felt like you needed data faster than every 3 matches, it shouldn’t be too tough to modify the program to scout a single match per sheet (and I can help point someone in the right direction on this if desired)
Thanks a bunch for the detailed response, I appreciate it. This problem did come across my mind when I was thinking about such a project, and my first idea for a solution was having a dedicated text field in one section of the scantron, and simply cropping that text field in the photo to display next to any particular match data. You could OCR it to get a real string out of it, but sometimes it may not work, so taking an image of it might just be easier. What are your thoughts for this? (I’m super on-board with the idea of this scouting system)
We only paid for the scanner this past season. For 2018 we will likely end up paying for AWS to avoid having to move it off my account. I can list out what other requirements there are that we didn’t pay for this season
Laptop - We had a programming laptop that was starting to get a bit sluggish with Eclipse that we dedicated to this. A slightly faster system might be nice, but overall this wasn’t a big deal for the $0 cost.
Scanner - ~$150. We bought this and have been pretty happy with it so far. The system should work with most scanners, though I would recommend one with Auto Document Feed based on 2067’s experience (NeatDesk with no ADF) vs. ours (with ADF)
Copies - We have use of the school’s copier for this. For a district event you’ll have ~40 teams * 12 matches per team / 3 team-matches per page = ~160 copies. With DCMP and CMP it’s probably about 1,000 pages for a season?
Remote server - We used AWS free tier which is free for 1 year. If you don’t want to move to someone else’s AWS account after a year it should be $20-$150 dollars to run the remote server, depending on whether you leave it up. I have a bit more detail on this on the AWS page on the Wiki
I don’t have much experience with OCR, so I’m not sure how well this would work. I have seen many of our student’s handwriting (and my own for that matter) and would be a little worried about this.
Ultimately, we didn’t really seem to miss comments all that much. We have 1 or 2 lead scouts that take notes in a notebook which may or may not get entered into PiScout using the comments feature.
If the 2018 game has any elements we feel we desperately need non-bubble representation for, we may see what we can do with some sort of snapshotting like you’ve described or some other digitizing solution, but this adds significant complexity to scouter capture, program capture (scanning), storage, and visualization. For those reasons, we will likely only venture there if we feel there is extreme game value to having this info (we did not see this value in capturing auto paths, shooting locations, or gear running paths in 2017).
I’ve been trying to do something similar to this as Scout Lead for my FTC Team.
Note: I’m not a programmer, but a mech so anything more than the basic APCS code goes right over my head.
My (very) cursory look at this suggests that it is getting some data from TBA - what data is it getting and how could I circumnavigate it?
Hi! The program fetches 3 things from TBA if available: team pictures, match schedules, and rankings data.
The absolute easiest way to bypass it is to use the system without an internet connection. The whole system will work without the TBA connection and this would prevent it from grabbing FRC team pictures by accident. the match schedule and ranking pages would not work, but the home page, team pages and compare pages would all work fine.
A little more complicated would be removing the image code (this wouldn’t be too bad) and coming up with a way to feed your FTC match schedule in (and making any necessary modifications for 2v2). This would allow for using the Match and Ranking pages
The system is designed to make it fairly easy (and hopefully at least somewhat documented) to update for different teams and games, but I won’t pretend it is 100% trivial. I am happy to provide a pointer or two if someone is trying it and gets stuck though!