Citrus Circuits 2019 Scouting System Whitepaper

Team 1678 is proud to present our 2019 Scouting System Whitepaper!

The Scouting Whitepaper is a technical document that describes FRC Team 1678: Citrus Circuits’ electronic scouting system. To scout, we use electronic devices to collect, process, and visualize data. These devices are operated by trained team members, whom we call scouts, to record statistics about robots at the competitions we attend.

The whitepaper includes: a technical overview of our system and each of its components (including application screenshots and data computations), the logistics and processes our subteam uses, the lessons we’ve learned and our plans for future improvements, a brief analysis of our data accuracy, and links to additional resources.

All of our scouting whitepapers are available at citruscircuits.org/scouting.
The code for our 2015-2019 scouting systems is available at github.com/frc1678.

If you have any questions about the whitepaper, our scouting system, how to start your own scouting system, or anything else, please don’t hesitate to send us an email at softwarescouting@citruscircuits.org. Also, if you’d like to give us feedback about the whitepaper (we really appreciate it), tell us how your team was impacted by the whitepaper, or share your own system with us, please send us an email. We are interested in helping to develop the FRC scouting community and opening the power of an electronic scouting system up to other teams, and we are interested in working with other teams to make innovations in FRC scouting. We’d love to hear from you!

We’ll also be actively responding to any questions in this thread for the next few weeks. The best way to contact us is at softwarescouting@citruscircuits.org.

38 Likes

@AJohnson342 not sure what all this contains. thought it might be of interest

1 Like

Thank you all so much for releasing these each year! I don’t think it’s a secret that scouting is a huge part of why 1678 has been on Einstien the last seven years and I’m very happy that you all choose to share it with the community.

9 Likes

Awesome stuff! Couple of thoughts:

Having 3 scouts per robot allows for some cool redundancy checks. The SPR values for scouts are interesting to me. If you have the data available and would be willing to share, I’d love to see the spread of SPR values. You can anonymize the users if you want, but I just am curious what the average score is and what the spread of values looks like for you guys. At least if you could say what the average and stdev values are that would be intriguing.

Elo pushing ability looks sweet and seems like a solid solution to an otherwise challenging problem. How did you arrive at your values for K, C, and winValue? Do these values stay the same every year or do you re-tune each season? Have you looked at increasing K early on in the event to get quicker stabilization of ratings?

The graphs in chapter 5 are cool, but if you have any raw data on predictions you’d be willing to share publicly or privately I’d love to look more into it.

I’ll be looking more into the code later.

3 Likes

Thanks a bunch for sharing!

I have a few questions:

  • How many programmers are dedicated to working on the scouting system during the season?
  • Do you have any parts complete before the season starts?
  • How do you prevent your scouts from being overloaded by the amount of data that needs to be input? I’ve had a lot of trouble balancing how much is too much in terms of ergonomics.

Thanks!

1 Like

Here are anonymized SPRs from the Carver Division of Houston Champs 2019: scout_precision_rankings.csv

Since we had no prior experience using an Elo system, we arrived at our values for K, C, and winValue by tweaking them until we got numbers that matched with our observations.

This was the first year we used an Elo system.

We have not, but we’ll look into it in the future. Thanks for the idea!

We’d be happy to share!
The following files are all from the Carver Division of Houston Champs 2019.
RP and seed predictions based on prescouting data: prescouting_predicted_rp_accuracy.csv
Match score predictions based on prescouting data: prescouting_predicted_score_accuracy.csv
Match score predictions based on competition data (in this file, each prediction is made after the previous match): competition_predicted_score_accuracy.csv

4 Likes

From Section 6.2 of the whitepaper: In 2019, we had 17 developers work on the scouting system during the season full-time (our scouting developers do not write software for the robot).

Yes, we re-use software from the previous year and/or we develop new software before build season begins. We’re working on changing our system to be more adaptable year-to-year by abstracting our software to be as game-agnostic as possible (the possible type of data collected for each game doesn’t vary significantly from a software point of view).

Collect as little data as possible. Limit yourself to the absolute necessities, or you’ll sacrifice the accuracy of your data. This is a problem we’ve struggled with, and we discuss it in Section 4.3.1 of the whitepaper.

In the past, we’ve focused on collecting as much data as possible, without considering the impact that had on our data accuracy. Our approach moving forward is to limit the number of data fields we collect as much as possible, and to keep in mind that any data field collected decreases the accuracy of our data. Since our system has 18 objective match scouts, we’re able to collect a larger number of data fields accurately since we only need 2 out of the 3 scouts for a robot in a match to get the correct data.

5 Likes

Thanks!

It is very generous of you guys to release this information. After reading your whitepaper, you guys seem to have a good grasp on the positive team culture surrounding scouting. I was wondering if you could share some information on how you guys go about that. What roles do mentors and members of leadership play?

Also the SPR information is very interesting, do you guys have some sort of reward system for this? Or is getting the highest overall SPR/matches scouted a competitive thing on your team?

We use something similar to your “Pick Ability”, we call it our “Selection Sauce”. How do you guys feel about the system?

I took the pre-event total RP predictions, pre-event score predictions, and in-event score predictions from my event simulator and compared them to 1678’s. You can see the full results in this workbook: 1678 prediction comparison.xlsx (51.1 KB) .

Summary:
Pre-event total RP Predictions:
For these I compared to my predicted average RPs. 1678 has an RMSE of 4.9 and I have an RMSE of 4.1. I chose not to directly compare seed predictions because it wouldn’t be an apples-to-apples comparison.

Pre-event score predictions:
For each match, my simulator takes the max pre-event OPR of each alliance member and sums them together for a predicted match score (assuming no matches played). 1678 had an RMSE of 15.3 for their pre-event score predictions, and I had an RMSE of 13.8.

In-event score predictions:
For these, my simulator uses predicted contributions, which are a hybrid of pre-event max OPR and in-event results. I chose to only compare matches 25 and higher in this comparison, as some teams have no data for 1678 to use for the first 12 matches, and only one match worth of data for matches 12-24. I thought it would be fairer to only compare matches where they have at least two prior data points for their predictions. For in-event predictions, 1678 has an RMSE of 13.2, and I have an RMSE of 13.8.

I’d caution against reading too much into the results when we only have one event’s worth of data points. Also of note is that there is clearly far more to 1678’s system than the simple averages spit out from the prior data. Thanks to 1678 for posting these!

4 Likes

Here’s a few things we do:
Before competition, at the beginning of scout training, we show scouts how their data is used and give historical examples of how our scouting system provided us a huge competitive advantage.
During competition, after alliance selections, we congratulate our scouts and let them know which teams we were able to pick that other teams missed.
After competition, we summarize how our scouting data contributed to our success at the event.

We promote the mantra of “Robots win matches, scouting wins tournaments”.
All of our mentors and leadership students understand the value scouting has on our team, and are able to re-enforce that idea if they hear otherwise.
The structure of our team has a dedicated subteam (it’s one of our largest subteams) for the software development of our scouting system. By putting a lot of resources into our scouting system, team members recognize its importance to our team.
Our team also has a size-limited travel team, since we cover the entire cost of traveling to competition for each team member. The majority of our travel team is scouts, and we’re fortunate enough to have scouting be a privilege on our team.

In previous years, we’ve had a small prize (e.g. a hat) for the scout with the best SPR. In 2019, scouts could not view their SPR since SPR is only an estimate and works best for the 25% least accurate scouts, not for the more accurate scouts. At Champs this year, all of our scouts were extremely accurate, around 90-95% accuracy. At that range, SPR can’t accurately differentiate scouts.

In the years scouts have been able to see their SPR, we’ve seen some competitiveness that we don’t like to promote. Scouts with lower SPRs naturally compare themselves to the other scouts, even when their SPR is identical within margin of error.

We don’t encourage scouts to scout extra matches; every scout is required to take their breaks. Scouting is tedious and scouts need breaks.

We’ve found it to be a useful starting-point for our picklist, which saves us time since we don’t have to move as many teams around. After we initially sort teams, we don’t look at pick ability. We dedicate time before and during competition to adjust the weights that are used to calculate our pick abilities to get a better starting picklist. For example, after each competition, we look at our final picklist and adjust the weights to get as close as possible with our generated picklist.

6 Likes

Thank you for sharing. I noticed this section in your whitepaper:

• Scouting developers have struggled with their Git and GitHub practice due to a limited understanding of Git and GitHub and experience with a collaborative Git and GitHub workflow.
• To address this, we created specific training (including practice assignments) for Git and GitHub for new and veteran developers.

We too struggle teaching SCM and git concepts (to our software development students in general). We’ve not found any tutorials or training targeted at very novice developers. Is the git training your team developed something that you are able to share? With only two software mentors, we just don’t have the bandwidth to create something like that on our own.

Thanks again for a fascinating read.

Here are the resources we used last year:
Git(hub) Training.pdf
The Introduction Sequence, Ramping Up, and Moving Work Around lessons from https://learngitbranching.js.org/

We’re wrapping up our general software training in the next couple weeks, and we’ll look into releasing the resources we created for that (including improved Git & GitHub resources). We have a little bit of polishing and cleaning up that we’d like to do before releasing them. If we release them, we’ll be sure to make a post in this thread.

1 Like

Could you give some more information on how you are recording cycles prevented?

341 attempted a defensive metric this season that compared a scoring teams average cycles to the scoring teams cycles in a specific match and then assigned the difference as a defensive score to the defender. Is cycles prevented a similar metric?

The calculation for cycles prevented (which is part of points prevented) is detailed in Section 2.5.1.2.

For each offensive cycle, we record the match time and if it was defended or undefended. Additionally, we record the match time a robot plays defense.

By aggregating this data, we can determine which cycles a robot defended.

Cycles prevented looks at the offensive robot’s change in percent of cycles dropped and change in cycle time when the robot is under defense (and compares it to their overall undefended average across matches).

The formula for cycles prevented is in Section 2.5.1.2.

Please let us know if you have further questions!

1 Like

This is a great paper. Your scouting system is amazing. You state that you have 17 people working through the entire season to make it. This is more than many teams. If you only had one or two people working on it through the season, what would you focus on? Would you focus on all of the predictive systems (such as match score) or would you limit yourselves to just making a data collection system?

2 Likes

Love it! Keep doing great!

2 Likes

If we only had 1 or 2 people, we’d probably use a Google Form linked to a Google Spreadsheet. We’d keep data collection simple and focus our efforts on processing and visualization.

All processing would be done with spreadsheet formulas, which would allow us to do basic score predictions.

Our focus on visualization would be providing data for picklist creation, but we’d also try real-time visualization for match strategy, by viewing the spreadsheet on a mobile device (this isn’t ideal, but it’s possible to make a usable mobile display on a sheet). A stretch goal might be using Google App Script to create better mobile visualization for match strategy.

6 Likes

We’re in our 3rd year, the first two years we were paper scouting. This year we’re implementing scouting app. Our setup was heavily influenced by your 2018 whitepaper. I developed a bare bones proof of concept app to use at our off-season event last weekend. The scouts loved the ease of use and the real-time match previews. And I learned a lot about our system, but it was a huge success. The new system even got couple more students interested in joining our newly formed Strategy team.

You mentioned in the Future Improvements section, that you’re simplifying to supporting Android, Kotlin, and Python. Does this mean that Pit Scout App and the QR Scanner will both migrate to Android apps? Or do you intend to maintain an iOS QR Scanner?

I used Firestore, but my scouting app ran on people’s phones, which allowed me to not worry about disconnected scouting. I plan on moving to a QR first system like yours for 2020. Did you look at Firestore for 2019? I went with Firestore since it was were it seemed google was going for mobile apps. I had 2 months to learn android and develop a system solo, so didn’t have a time to try a ‘bake off’ of different options. Just looking for insight on Firestore vs Firebase.

Do you have a method to edit either raw or calculated data? For cases when a question box ruling might change the score but scouts already submitted their data.

With your map based UI, how/can scouts adjust a scoring error, such as scoring a cargo instead of a hatch and wanting to correct the error immediately after clicking ok?

We’re looking at running a 6 scout/ 4 super scout setup. Do you run both Scouts and Super Scout schedule on a 30 or 60 min shifts?

How are match replays handled for when a match is stopped and replayed, or when replaying a match that had already been finalized

Is there any way for notes to be added to match results, or do you use paper if there’s something important to note?

Brian

FYI, we’re supporting Android through the programming language Kotlin. The pit data collection will be in Android. We’re maintaining our existing QR scanner in Flutter (programming language Dart), which gives us a native Android and iOS app in one codebase. Most of our scouts use iOS phones, so we need to support a QR scanner for iOS. We aren’t planning on doing significant development on the QR scanner.

We looked at Firebase Cloud Firestore for the upcoming season. We also experimented with it a bit during the 2019 season. For the 2020 season, we’re using MongoDB, since we want a database that we can run locally and in the cloud. In our system, we have a laptop that runs all of our processing locally (so that it can be monitored). With an online database like Firebase Cloud Firestore or Firebase Realtime Database, we’ve had to create our own local caching solution.

If you’re looking between Firebase Cloud Firestore or Firebase Realtime Database, we’d recommend Firebase Cloud Firestore since it’s the newer version of Realtime Database, and it’s recommended by Firebase’s official documentation (https://firebase.google.com/docs/database/rtdb-vs-firestore).

We are able to edit raw data if we need to, but we usually avoid doing so, since our consolidation process can automatically fix errors from a single scout. It’s also inconvenient for us to edit raw data. For contested scores (e.g. does a climb count?) scouts wait to submit their data until the scores are posted on the audience display.

There is an undo button, but we generally recommend that scouts just continue scouting and let our consolidation process fix the error. We’ve seen scouts get caught up about an error, and then they mess up the rest of their data for the match by trying to fix it.

At Houston Champs 2019, we ran 45 minute shifts. We’ve done anywhere between 30 min to 60 min in the past. Scouts and Super Scouts have the same break schedule.

If a match is stopped before it ends, scouts don’t submit data. If a match is replayed after scouts have already submitted their data, we delete the previous data for that match.

We’ve had an electronic note feature in the past, but we found it was easier to use paper.