Okay, so I’ve been waiting and waiting and waiting to be able to tell the FIRST community about this, and I’m beyond excited to share tonight, and even though it’s not quite public yet, I’m getting this out here so people can start asking questions and making feature suggestions for this week.
I’m Sam Weaver, Lead Programmer for Team 4534: The Wired Wizards (archimedes division!) Early on in the season, our team’s leadership sat down and discussed scouting this year, and determined that we would not be able to dedicate a full time scouting staff at events this year. Programming was tasked to solve this, and we came up with an idea that we think redefines scouting for FIRST Stronghold.
Originating early on in the season under the codename “Project Magic Wand,” the name later changed to Project ORB, to better reflect what it does. What does it do, you may ask? Project Orb is an automatic predictive scouting system, that uses a combination of Neural Networks and statistical analysis that makes predictions about teams capabilities, specifically defense capabilities, high and low goal shooting abilities, climbing ability, and challenge consistency, and match predictions statistically more likely to be accurate than OPR. In essence, without entering in or providing any data, any user can view the capabilities of a robot to cross defenses, how many goals they can score in a match, and whether they can climb or not, completely automatically, for all 3144 active teams in the 2016 FRC Season. Cool, right?
How does it work?
Project ORB uses TheBlueAlliance’s API (thanks TBA!) to view and analyze all matcches a team has competed in over the entire season. This match data includes values about defense crossings, challenges and climbs, and high and low goal scores. Unfortunately, the data that the Field Management System at events provides to TheBlueAlliance is not spefici to which team completed which action, making the data along not very valuable. Here’s where our friends The Neural Networks come in! In our training process, we create a Feed Forward Neural Network for each and every**** team, and train it on their match data, with the goal to eliminate noise from their random alliance partnerships during qualifications. The results, are predictions on how many crossings of a defense a robot can make (ranging from zero to two.) This was the first step in Project Magic Wand, and we then adapted it to be more accurate, and adapted it for goal scoring and climbing and challenges.
How can I use it?
We did a beta test with all the teams at the 2016 NC District Championship (thanks guys o/ ) within our scouting app platform, and have since moved the data out to a standalone site, available at http://orb.scoutfrc.com/. Our programming team is still finishing up touches on the front end, so it is on our coming soon page at the moment, but we aim to have the site up and running on Thursday.
What are the significance of these numbers? How accurate are these results?
Inside the system, you will receive proficiency percentages for each defense on a team’s page. These percentages range from 0-100, and are typically mapped to the raw output of our networks, which range from zero to two. Unfortunately, due to how we gather the data, and trends in the game, some results are less accurate than others. Specifically, the Low Bar defense, for example, is almost never below a value of 1.5, due to the fact that there is a low bar robot in almost every match. As a result, the percentage for this defense will be tweaked to reflect a range that is more useful, such as 1.5-2. In contrast, you receive actual numbers for low and high goal shooting, which represent how many goals the system predicts they can score during a match.These numbers are broken up into auto and teleop values, which can allow you to predict a team’s auto routine. Strengths we’ve found in this system come in the ability to provide data in bulk, in supplement, or before your scouting team can get it. In addition, our system gives proficiency percentages of defenses, so while a team might report that they can complete both defenses in a defense group, our system can report which of the two they are weaker at, trumping what would typically be a boolean flag.
Please, please, ask questions. We want to get this system to the best possible state for this week’s competition. We rushed to put this post out tonight so that everyone could get the information as soon as possible, we can provide more data as we get more chances to figure out what people need to know, and as we keep tweaking our system.
Last but not least, I’d like to give special thanks to two of my programmers, Tom and Danny, for their extensive work on this project, and the rest of the programming team for their superb work this season. I wish all teams the best of luck, and we’ll see you at the competition!
Sam Weaver, Wired Wizards 4534